Despite the fact that generative models are extremely successful in practice, the theory underlying this phenomenon is only starting to catch up with practice. In this work we address the question of the universality of generative models: is it true that neural networks can approximate any data manifold arbitrarily well? We provide a positive answer to this question and show that under mild assumptions on the activation function one can always find a feedforward neural network that maps the latent space onto a set located within the specified Hausdorff distance from the desired data manifold. We also prove similar theorems for the case of multiclass generative models and cycle generative models, trained to map samples from one manifold to another and vice versa. Generative models such as Generative Adversarial Networks (GANs) are widely used for tasks such as image synthesis, semi-supervised learning, and domain adaptation (Brock et al., 2018; Radford et al., 2015; Zhang et al., 2017; . Such generative models are trained to perform a mapping from a latent space of a small dimension to some specified data manifold, typically represented by a dataset of natural images. Despite their success and excellent performance, the theory behind such models is not yet well understood. A recent survey of open questions about generative models (Odena, 2019) To answer these questions we adopt the following geometric approach, very amenable to precise mathematical analysis. Under the assumption of the Manifold Hypothesis (Goodfellow et al., 2016) , data comes from a certain data manifold. Then the goal of a generator network is to reproduce this data manifold as closely as possible by mapping the latent space into the ambient space of the data manifold. This intuitive understanding can be written more concretely as follows. Suppose that we are given the latent space M z , feedforward neural network f θ as a generator, and some target data manifold M. In order for the manifold M to be generated by f θ we require that the image of M z under f θ is sufficiently close to M, more specifically that the Hausdorff distance between f θ (M z ) and M is less than the given parameter ε. Hausdorff distance is a well-defined metric on the space of all compact subsets of Euclidean space and hence is equal to zero if and only if f θ (M z ) = Mthe case of precise replication of the data manifold. Thus, the question at hand can be formulated as follows: is it possible to approximate in the sense of the Hausdorff distance an arbitrary compact (connected) manifold using standard feedforward neural networks? By combining techniques from Riemannian geometry with well-known properties of neural networks we provide a positive answer to this question. We also show that the condition of being smooth is not necessary and the results are also valid for just topological manifolds. We further extend the discussed geometric approach for the theoretical analysis of many practical situations, for instance, to the case of data manifolds, which consist of multiple disjoint manifolds and correspond to multiclass datasets, and cycle generative models , which for two manifolds learn an approximately invertible mapping from one manifold to another. For the latter case we prove a somewhat surprising result that for any given pair of data manifolds of the same dimension, one can always train a pair of neural networks which are approximately inverses of one another, and map the first manifold almost onto the second one, and vice versa. In this work, we ignore specifics of the training algorithm (for instance, what loss function is used) and merely focus on understanding the generative capabilities of neural networks. In this work we have attempted to partially explain huge empirical success of generative models. Our results show only existence of neural networks approximating arbitrary manifolds, and do not specify how one can estimate the size of a network required for any given manifold. We hypothesize, however, that there might exist a connection between certain geometrical properties of a manifold (curvature, various topological properties), and the width/depth of a neural network required. One interesting direction of research left for a future work is analyzing this relation for datasets popular in computer vision, such as MNIST or CelebA, or toy datasets sampled from simple small dimensional manifolds (tori, circles), where one can easily vary the topological properties. A PROOFS Theorem 5.2 (Geometric Universality for Multiclass Manifolds). Let M = c i=1 M i be a "multiclass" data manifold, with each M i being a compact connected d-dimensional topological manifold (with or withour boundary). Then for every ε > 0 and δ > 0 and every universal nonlinearity σ there exists a fully connected neural network f θ (z) : I d → R n with the activation function σ such that the following properties hold. • There exists a collection Proof. Similar to the proof of Theorem 5.1 we will apply the universal approximation theorem to a certain function constructed with the help of Lemma 5.2. To construct such function let us select sets D i in the following way. We divide the interval [−1, 1] uniformly into c intervals, namely Intuition is very simple: we chop down the cube D on the first axis into smaller boxes, and remove some space between them. On each of the chunks D i we can now apply Lemma 5.2 for the corresponding manifold M i , obtaining a collection of maps . To construct a global continuous map f we can now simply linearly interpolate each of the maps f i from the right boundary of the neighboring one. By applying the universal approximation theorem to this function f , we finalize the proof. Lemma 6.3. Let f : M → N be an arbitrary smooth embedding. Let S ⊂ M be a smooth embedded submanifold. Then f | S is also a smooth embedding. Proof. The proof follows from the definition. Indeed, for every point x ∈ S ⊂ M we have T x S ⊂ T x M and restriction of the derivative of f onto this subspace is also injective. Note that f | S is also injective and open map.
We shot that a wide class of manifolds can be generated by ReLU and sigmoid networks with arbitrary precision. This paper provides certain basic guarantees on when manifolds can be written as the image of a map approximated by a neural net, and stitches together theorems from manifold geometry and standard universal approximation results. This paper theoretically shows that neural-network-based generative models can approximate data manifolds, and proves that under mild assumptions neural networks can map a latent space onto a set close to the given data manifold within a small Hausdorff distance.