Three-dimensional geometric data offer an excellent domain for studying representation learning and generative modeling. In this paper, we look at geometric data represented as point clouds. We introduce a deep autoencoder (AE) network with excellent reconstruction quality and generalization ability. The learned representations outperform the state of the art in 3D recognition tasks and enable basic shape editing applications via simple algebraic manipulations, such as semantic part editing, shape analogies and shape interpolation. We also perform a thorough study of different generative models including GANs operating on the raw point clouds, significantly improved GANs trained in the fixed latent space our AEs and, Gaussian mixture models (GMM). Interestingly, GMMs trained in the latent space of our AEs produce samples of the best fidelity and diversity.
 To perform our quantitative evaluation of generative models, we propose simple measures of fidelity and diversity based on optimally matching between sets point clouds. Three-dimensional (3D) representations of real-life objects are a core tool for vision, robotics, medicine, augmented and virtual reality applications. Recent encodings like view-based projections, volumetric grids and graphs, complement more traditional shape representations such as 3D meshes, level set functions, curve-based CAD models and constructive solid geometry BID1 . These encodings, while effective in their respective domains (e.g. acquisition or rendering), are often poor in semantics. For example, naïvely interpolating between two different cars in a view-based representation does not yield a representation of an "intermediate" car. Furthermore, these raw, high-dimensional representations are typically not well suited for the design of generative models via classic statistical methods. As such, editing and designing new objects with such representations frequently involves the construction and manipulation of complex, object-specific parametric models that link the semantics to the representation. This may require significant expertise and effort.Recent advances in deep learning bring the promise of a data-driven approach. In domains where data is plentiful, deep learning tools have eliminated the need for hand-crafting features and models. Deep learning architectures like autoencoders (AEs) BID22 Kingma & Welling, 2013) and Generative Adversarial Networks (GANs) BID10 BID20 Denton et al., 2015; BID6 are successful at learning complex data representations and generating realistic samples from complex underlying distributions. Recently, deep learning architectures for view-based projections BID30 BID14 , volumetric grids BID18 BID32 BID12 and graphs BID4 BID13 Defferrard et al., 2016; Yi et al., 2016b) have appeared in the 3D machine learning literature.In this paper we focus on point clouds, a relatively unexplored 3D modality. Point clouds provide a homogeneous, expressive and compact representation of surface geometry, easily amenable to geometric operations. These properties make them attractive from a learning point of view. In addition, they come up as the output of common range-scanning acquisition pipelines used in devices like the Kinect and iPhone's recent face identification feature. Only a handful of deep architectures for 3D point clouds exist in the literature: PointNet BID17 successfully tackled classification and segmentation tasks; BID14 used point-clouds as an intermediate step in their pipeline; Fan et al. (2016) used pointclouds as the underlying representation to extract 3D information from 2D images. We provide the first results that use deep architectures with the focus of learning representations and generative models for point clouds.Generative models have garnered increased attention recently in the deep learning community with the introduction of GANs BID10 ). An issue with GAN-based generative pipelines is that training them is notoriously hard and unstable BID24 . More importantly, there is no universally accepted way to evaluate generative models. In evaluating generative models one is interested in both fidelity, i.e. how much the generated points resemble the actual data, and coverage, i.e. what fraction of the data distribution a generated sample represents. The latter is especially important given the tendency of certain GANs to exhibit mode collapse. We provide simple methods to deal with both issues (training and evaluation) in our target domain. Our specific contributions are:• We design a new AE architecture-inspired by recent architectures used for classification BID17 -that is capable of learning compact representations of point clouds with excellent reconstruction quality even on unseen samples. The learned representations are (i) good for classification via simple methods (SVM), improving on the state of the art BID31 ; (ii) suitable for meaningful interpolations and semantic operations.• We create the first set of generative models which ( i) can generate point clouds measurably similar to the training data and held-out test data; (ii) provide good coverage of the training and test dataset. We argue that jointly learning the representation and training the GAN is unnecessary for our modality. We propose a workflow that first learns a representation by training an AE with a compact bottleneck layer, then trains a plain GAN in that fixed latent representation. Intuitively, training a GAN inside a compact, low-dimensional representation is easier. We point to theory BID0 that supports this idea, and verify it empirically. Latent GANs are much easier to train than monolithic (raw) GANs and achieve superior reconstruction with much better coverage. Somewhat surprisingly, GMMs trained in the latent space of fixed AEs achieve the best performance across the board.• We show that multi-class GANs work almost on par with dedicated GANs trained per-objectcategory, as long as they are trained in the latent space.• To support our qualitative evaluation, we perform a careful study of various old and new metrics, in terms of their applicability ( i) as objectives for learning good representations; (ii) for the evaluation of generated samples. We find that a commonly used point cloud metric, Chamfer distance, fails to discriminate certain pathological cases from good examples. We also propose fidelity and coverage metrics for our generative models, based on an optimal matching between two different samples, e.g. a set of point clouds generated by the model and a held-out test set.The rest of this paper is organized as follows: Section 2 outlines the necessary background and building blocks for our work and introduces our evaluation metrics. Section 3 introduces our models for latent representations and generation of point clouds. In Section 4, we evaluate all of our models both quantitatively and qualitatively, and analyze their behaviour. Further results and evaluation can be found in the appendix. The code for all our models is publicly available 1 .2 BACKGROUND DISPLAYFORM0 Autoencoders. Autoencoders (AE -inset) are deep architectures that aim to reproduce their input. They are especially useful, when they contain a narrow bottleneck layer between input and output. Upon successful training, the bottleneck layer corresponds to a lowdimensional representation, a code for the dataset. The Encoder (E) learns to compress a data point x into its latent representation, z. The Decoder (D) can then reproduce x from its encoded version z. DISPLAYFORM1 Generative Adversarial Networks. GANs are state-of-the-art generative models. The basic architecture (inset) is based on a adversarial game between a generator (G) and a discriminator (D). The generator aims to synthesize samples that look indistinguishable from real data (drawn from x ∼ p data ) by passing a randomly drawn sample z ∼ p z through the generator function G. The discriminator tries to tell synthesized from real samples. The most commonly used losses for the discriminator and generator networks are: DISPLAYFORM2 DISPLAYFORM3 where θ (D) , θ (G) are the parameters for the discriminator and the generator network respectively. In addition to the classical GAN formulation, we also use the improved Wasserstein GAN BID11 , which has shown improved stability during training.Challenges specific to point cloud geometry. Point clouds as an input modality present a unique set of challenges when building a network architecture. As an example, the convolution operator -now ubiquitous in image-processing pipelines -requires the signal (in our case, geometry) to be defined on top of an underlying grid-like structure. Such a structure is not available in raw point clouds, which renders them significantly more difficult to encode than e.g. images or voxel grids. Recent classification work on point clouds (PointNet -Qi et al. (2016a) ) bypasses this issue by circumventing 2D convolutions. Another issue with point clouds as a representation is that they are unordered -any permutation of a point set still describes the same shape. This complicates comparisons between two point sets, typically needed to define a loss function. This unorderedness of point clouds also creates the need for making the encoded feature permutation invariant.Point-set distances. Two permutation-invariant metrics for comparing unordered point sets have been proposed in the literature (Fan et al., 2016) . On the one hand, the Earth Mover's distance (EMD) BID21 is the solution of a transportation problem which attempts to transform one set to the other. For two equally sized subsets S 1 ⊆ R 3 , S 2 ⊆ R 3 , their EMD is defined by d EM D (S 1 , S 2 ) = min φ:S1→S2 x∈S1x − φ(x) 2 where φ is a bijection. Interpreted as a loss, EMD is differentiable almost everywhere. On the other hand, the Chamfer (pseudo)-distance (CD) measures the squared distance between each point in one set to its nearest neighbor in the other set: DISPLAYFORM4 It is still differentiable but more computationally efficient.Evaluation Metrics for representations and generative models. In the remainder of the paper, we will frequently need to compare a given set (distribution) of points clouds, whether reconstructed or synthesized, to its ground truth counterpart. For example, one might want to assess the quality of a representation model, in terms of how well it matches the training set or a held-out test set. Such a comparison might be done to evaluate the faithfulness and/or diversity of a generative model, and measure potential mode-collapse. To measure how well a point-cloud distribution A matches a ground truth distribution G, we use the following metrics:Coverage. For each point-cloud in A we find its closest neighbor in G; closeness can be computed using either CD or EMD, thus yielding two different metrics, COV-CD and COV-EMD. Coverage is measured as the fraction of the point-clouds in G that were matched to point-clouds in A. A high coverage score typically indicates that most of G is roughly represented within A. We presented a novel set of architectures for 3D point-cloud representation learning and generation. Our results show good generalization to unseen data and our representations encode meaningful semantics. In particular our generative models are able to produce faithful samples and cover most of the ground truth distribution without memorizing a few examples. Interestingly, we see that the best-performing generative model in our experiments is a GMM trained in the fixed latent space of an AE. While, this might not be a universal result, it suggests that simple classic tools should not be dismissed. A thorough investigation on the conditions under which simple latent GMMs are as powerful as adversarially trained models would be of significant interest.
Deep autoencoders to learn a good representation for geometric 3D point-cloud data; Generative models for point clouds. Approaches to learn GAN-type generative models using PointNet architecture and latent-space GAN.