A point cloud is an agile 3D representation, efficiently modeling an object's surface geometry. However, these surface-centric properties also pose challenges on designing tools to recognize and synthesize point clouds. This work presents a novel autoregressive model, PointGrow, which generates realistic point cloud samples from scratch or conditioned from given semantic contexts. Our model operates recurrently, with each point sampled according to a conditional distribution given its previously-generated points. Since point cloud object shapes are typically encoded by long-range interpoint dependencies, we augment our model with dedicated self-attention modules to capture these relations. Extensive evaluation demonstrates that PointGrow achieves satisfying performance on both unconditional and conditional point cloud generation tasks, with respect to fidelity, diversity and semantic preservation. Further, conditional PointGrow learns a smooth manifold of given images where 3D shape interpolation and arithmetic calculation can be performed inside. 3D visual understanding BID2 ; BID35 ) is at the core of next-generation vision systems. Specifically, point clouds, agile 3D representations, have emerged as indispensable sensory data in applications including indoor navigation BID6 ), immersive technology BID20 ; BID21 ) and autonomous driving ). There is growing interest in integrating deep learning into point cloud processing BID10 ; BID13 ; BID1 ; BID11 ; BID32 ; BID29 ). With the expressive power brought by modern deep models, unprecedented accuracy has been achieved on high-level point cloud related tasks including classification, detection and segmentation BID16 BID5 ; BID25 ; ; BID31 ). Yet, existing point cloud research focuses primarily on developing effective discriminative models BID29 ; BID19 ), rather than generative models. This paper investigates the synthesis and processing of point clouds, presenting a novel generative model called PointGrow. We propose an autoregressive architecture ; ) to accommodate the surface-centric nature of point clouds, generating every single point recurrently. Within each step, PointGrow estimates a conditional distribution of the point under consideration given all its preceding points, as illustrated in Figure 1 . This approach easily handles the irregularity of point clouds, and encodes diverse local structures relative to point distance-based methods BID7 ; BID1 ).However , to generate realistic point cloud samples, we also need long-range part configurations to be plausible. We therefore introduce two self-attention modules BID14 ; BID24 ; BID34 ) in the context of point cloud to capture these long-range relations. Each dedicated self-attention module learns to dynamically aggregate long-range information during the point generation process. In addition, our conditional PointGrow learns a smooth manifold of given images where interpolation and arithmetic calculation can be performed on image embeddings.Compared to prior art, PointGrow has appealing properties:• Unlike traditional 3D generative models that rely on local regularities on grids BID26 BID5 BID30 ; BID22 ), PointGrow builds upon DISPLAYFORM0 The point cloud generation process in PointGrow (best viewed in color). Given i − 1 generated points, our model first estimates a conditional distribution of z i , indicated as p(z i |s ≤i−1 ), and then samples a value (indicated as a red bar) according to it. The process is repeated to sample y i and x i with previously sampled coordinates as additional conditions. The i th point (red point in the last column) is obtained as DISPLAYFORM1 autoregressive architecture that is inherently suitable for modeling point clouds, which are irregular and surface-centric.• Our proposed self-attention module successfully captures the long-range dependencies between points, helping to generate plausible part configurations within 3D objects.• PointGrow, as a generative model , enables effective unsupervised feature learning, which is useful for recognition tasks, especially in the low-data regime.Extensive evaluations demonstrate that PointGrow can generate realistic and diverse point cloud samples with high resolution, on both unconditional and conditional point cloud generation tasks. In this work, we propose PointGrow, a new generative model that can synthesize realistic and diverse point cloud with high resolution. Unlike previous works that rely on local regularities to synthesize 3D shapes, our PointGrow builds upon autoregressive architecture to encode the diverse surface information of point cloud. To further capture the long-range dependencies between points, two dedicated self-attention modules are designed and carefully integrated into our framework. PointGrow as a generative model also enables effective unsupervised feature learning, which is extremely useful for low-data recognition tasks. Finally, we show that PointGrow learns a smooth image condition manifold where 3D shape interpolation and arithmetic calculation can be performed inside.
An autoregressive deep learning model for generating diverse point clouds. An approach for generating 3D shapes as point clouds which considers the lexicographic ordering of points according to coordinates and trains a model to predict points in order. The paper introduces a generative model for point clouds using a pixel RNN-like auto-regressive model and an attention model to handle longer-range interactions.