This paper introduces the task of semantic instance completion: from an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry. This enables a semantically meaningful decomposition of a scanned scene into individual, complete 3D objects, including hidden and unobserved object parts. This will open up new possibilities for interactions with object in a scene, for instance for virtual or robotic agents. To address this task, we propose 3D-SIC, a new data-driven approach that jointly detects object instances and predicts their completed geometry. The core idea of 3D-SIC is a novel end-to-end 3D neural network architecture that leverages joint color and geometry feature learning. The fully-convolutional nature of our 3D network enables efficient inference of semantic instance completion for 3D scans at scale of large indoor environments in a single forward pass. In a series evaluation, we evaluate on both real and synthetic scan benchmark data, where we outperform state-of-the-art approaches by over 15 in mAP@0.5 on ScanNet, and over 18 in mAP@0.5 on SUNCG. Understanding 3D environments is fundamental to many tasks spanning computer vision, graphics, and robotics. In particular, in order to effectively navigate, and moreover interact with an environment, an understanding of the geometry of a scene and the objects it comprises is essential. This is in contrast to the partial nature of reconstructed RGB-D scans; e.g., due to sensor occlusions. For instance, for a robot exploring an environment, it needs to infer instance-level object segmentation and complete object geometry in order to perform tasks like grasping, or estimate spatial arrangements of individual objects. Additionally, for content creation or mixed reality applications, captured scenes must be decomposable into their complete object components, in order to enable applications such as scene editing or virtual-real object interactions; i.e., it might be insufficient to predict object instance masks only for observed regions. Thus, we aim to address this task of predicting object detection as well as instance-level completion for an input partial 3D scan of a scene; we refer to this task as semantic instance completion. Previous approaches have considered semantic scene segmentation jointly with scan completion , but lack the notion of individual objects. In contrast, our approach focuses on the instance level, as knowledge of instances is essential towards enabling interaction with the objects in an environment. In addition, the task of semantic instance completion is not only important towards enabling objectlevel understanding and interaction with 3D environments, but we also show that the prediction of complete object geometry informs the task of semantic instance segmentation. Thus, in order to address the task of semantic instance completion, we propose to consider instance detection and object completion in an end-to-end, fully differentiable fashion. From an input RGB-D scan of a scene, our new 3D semantic instance completion network first regresses bounding boxes for objects in the scene, and then performs object classification followed by a prediction of complete object geometry. Our approach leverages a unified backbone from which instance detection and object completion are predicted, enabling information to flow from completion to detection. We incorporate features from both color image and 3D geometry of a scanned scene, as well as a fully-convolutional design in order to effectively predict the complete object decomposition of varying-sized scenes. In summary, we present a fully-convolutional, end-to-end 3D CNN formulation to predict 3D instance completion that outperforms state-of-the-art, decoupled approaches to semantic instance completion by 15.8 in mAP@0.5 on real-world scan data, and 18.5 in mAP@0.5 on synthetic data: • We introduce the task of semantic instance completion for 3D scans; • we propose a novel, end-to-end 3D convolutional network which predicts 3D semantic instance completion as object bounding boxes, class labels, and complete object geometry, • and we show that semantic instance completion task can benefit semantic instance segmentation performance. In this paper, we introduced the new task of semantic instance completion along with 3D-SIC, a new 3D CNN-based approach for this task, which jointly detects objects and predicts their complete geometry. Our proposed 3D CNN learns from both color and geometry features to detect and classify objects, then predict the voxel occupancy for the complete geometry of the object in end-to-end fashion, which can be run on a full 3D scan in a single forward pass. On both real and synthetic scan data, we significantly outperform alternative approaches for semantic instance completion. We believe that our approach makes an important step towards higher-level scene understanding and helps to enable object-based interactions and understanding of scenes, which we hope will open up new research avenue. Table 6 : Anchor sizes (in voxels) used for SUNCG region proposal. Sizes are given in voxel units, with voxel resolution of ≈ 4.69cm Table 10 details the layers used in our backbone. 3D-RPN, classification head, and mask completion head are described in Table 11 . Additionally, we leverage the residual blocks in our backbone, which is listed in Table 9 . Note that both the backbone and mask completion head are fully-convolutional. For the classification head, we use several fully-connected layers; however, we leverage 3D RoIpooling on its input, we can run our method on large 3D scans of varying sizes in a single forward pass. We additionally list the anchors used for the region proposal for our model trained on our ScanNetbased semantic instance completion benchmark (Avetisyan et al., 2019; Dai et al., 2017a) and SUNCG datasets in Tables 5 and 6 , respectively. Anchors for each dataset are determined through k-means clustering of ground truth bounding boxes. The anchor sizes are given in voxels, where our voxel size is ≈ 4.69cm.
From an incomplete RGB-D scan of a scene, we aim to detect the individual object instances comprising the scene and infer their complete object geometry. Proposes an end-to-end 3D CNN structure which combines color features and 3D features to predict the missing 3D structure of a scene from RGB-D scans. The authors propose a novel end-to-end 3D convolutional network which predicts 3D semantic instance completion as object bounding boxes, class labels and complete object geometry.