{"content":[{"text":"generative deep neural networks, often focusing volumet- \n ric representations for shapes [11] or entire scenes [31, 12]. \n These techniques show great promise since they can learn \n generalized patterns in a large variety of environments; \n however, existing data-driven scene completion methods \n rely on supervised training, requiring fully complete ground \n truth 3D models, thus depending on large-scale synthetic \n datasets such as ShapeNet [4] or SUNCG [31]. As a re- \n sult, although we have seen impressive results from these \n approaches on synthetic test sets, domain transfer and ap- \n plication to real-world 3D scans remains a major limitation. \n In order to address the shortcomings of supervised learn- \n ing techniques for scan completion, we propose a new self- \n supervised completion formulation that can be trained only \n on (partial) real-world data. Our main idea is to learn to gen- \n erate more complete 3D models from less complete data, \n while masking out any unknown regions; that is, from an \n existing RGB-D scan, we use the scan as the target and re- \n move frames to obtain a more incomplete input. In the loss \n function, we can now correlate the difference in partialness \n between the two scans, and constrain the network to predict \n the delta while masking out unobserved areas. Although \n there is no single training sample which contains a fully- \n complete 3D reconstruction, we show that our network can \n nonetheless generalize to predict high levels of complete- \n ness through a combined aggregation of patterns across the \n entire training set. This way, our approach can be trained \n without requiring any fully-complete ground truth counter- \n parts that would make generalization through a synthetic- \n to-real domain gap challenging. \n Furthermore, we propose a new sparse generative neural \n network architecture that can predict high-resolution geom- \n etry in a fully-convolutional fashion. For training, we pro- \n pose a progressively growing network architecture trained \n in coarse-to-ﬁne fashion; i.e., we ﬁrst predict the 3D scene \n at a low resolution, and then continue increasing the surface \n resolution through the training process. We show that our \n self-supervised, sparse generative approach can outperform \n state-of-the-art fully-supervised methods, despite their ac- \n cess to much larger quantities of synthetic 3D data. \n We present the following main contributions:","bbox":{"left":0.081882,"top":0.091866,"width":0.386035,"height":0.624764}},{"text":"2. Related Work \n RGB-D Reconstruction Scanning and reconstructing 3D \n surfaces has a long history across several research com- \n munities. With the increase in availability of commod- \n ity range sensors, capturing and reconstructing 3D scenes \n has become a vital area of research. One seminal tech- \n nique is the volumetric fusion approach of Curless and \n Levoy [7], operating on truncated signed distance ﬁelds \n to produce a surface reconstruction. It has been adopted \n by many state-of-the-art real-time reconstruction methods, \n from KinectFusion [22, 16] to VoxelHashing [23] and \n BundleFusion [10], as well as state-of-the-art ofﬂine recon- \n struction approaches [5]. \n These methods have produced impressive results in \n tracking and scalability of 3D reconstruction from com- \n modity range sensors. However, a signiﬁcant limitation that \n still remains is the partial nature of 3D scanning; i.e., a per- \n fect scan is usually not possible due to occlusions and un- \n observed regions and thus, the resulting 3D representation \n cannot reach the quality of manually created 3D assets.","bbox":{"left":0.504676,"top":0.088998,"width":0.386035,"height":0.31491800000000003}},{"text":"Deep Learning on 3D Scans With recent advances in \n deep learning and the improved availability of large-scale \n 3D scan datasets such as ScanNet [9] or Matterport [3], \n learned approaches on 3D data can be used for a variety \n of tasks like classiﬁcation, segmentation, or completion. \n Many current methods make use of convolutional op- \n erators that have been shown to work well on 2D data. \n When extended into 3D, they operate on regular grid rep- \n resentations such as distance ﬁelds [11] or occupancy grids \n [20]. Since dense volumetric grids can come with high \n computational and memory costs, several recent approaches \n have leveraged the sparsity of the 3D data for discrim- \n inative 3D tasks. PointNet [26, 27] introduced a deep \n network architecture for learning on point cloud data for \n semantic segmentation and classiﬁcation tasks. Octree- \n based approaches have also been developed [29, 33, 34] \n that have been shown to be very memory efﬁcient; how- \n ever, generative tasks involving large, varying-sized envi- \n ronments seems challenging and octree generation has only \n been shown for single ShapeNet-style objects [28, 32]. An- \n other option leveraging the sparsity of 3D geometric data \n is through sparse convolutions [14, 13, 6], which have seen \n success in discriminative tasks such as semantic segmenta- \n tion, but not in the context of generative 3D modeling tasks, \n where the overall structure of the scene is unknown.","bbox":{"left":0.504676,"top":0.42533,"width":0.386035,"height":0.378529}},{"text":"• A self-supervised approach for scene completion, en- \n abling training solely on incomplete, real-world scan \n data while predicting geometry more complete than \n any seen during training, by leveraging common pat- \n terns in the deltas of incompleteness.","bbox":{"left":0.098162,"top":0.73244,"width":0.369754,"height":0.075538}},{"text":"• A generative formulation for sparse convolutions to \n produce a sparse truncated signed distance function \n representation at high resolution: we formulate this \n hierarchically to progressively generate a 3D scene in \n end-to-end fashion","bbox":{"left":0.098162,"top":0.825725,"width":0.36975500000000006,"height":0.07553699999999997}},{"text":"Shape and Scene Completion Completing 3D scans has \n been well-studied in geometry processing. Traditional \n methods, such as Poisson Surface Reconstruction [17, 18], \n locally optimize for a surface to ﬁt to observed points, and \n work well for small missing regions. Recently, various","bbox":{"left":0.504676,"top":0.825273,"width":0.386035,"height":0.07598899999999997}}],"images_bbox":[],"images_bbox_not_text_overlap":[]}