# DORec: Decomposed Object Reconstruction Utilizing 2D Self-Supervised
Features

Jun Wu, Sicheng Li, Sihui Ji, Yue Wang, Rong Xiong, and Yiyi Liao†
†Corresponding author<EMAIL_ADDRESS>Wu, Sicheng Li, Sihui Ji, Yue
Wang, Rong Xiong and Yiyi Liao are with Zhejiang University, Hangzhou, China.

###### Abstract

Decomposing a target object from a complex background while reconstructing is
challenging. Most approaches acquire the perception for object instances
through the use of manual labels, but the annotation procedure is costly. The
recent advancements in 2D self-supervised learning have brought new prospects
to object-aware representation, yet it remains unclear how to leverage such
noisy 2D features for clean decomposition. In this paper, we propose a
Decomposed Object Reconstruction (DORec) network based on neural implicit
representations. Our key idea is to transfer 2D self-supervised features into
masks of two levels of granularity to supervise the decomposition, including a
binary mask to indicate the foreground regions and a K-cluster mask to
indicate the semantically similar regions. These two masks are complementary
to each other and lead to robust decomposition. Experimental results show the
superiority of DORec in segmenting and reconstructing the foreground object on
various datasets.

##  I INTRODUCTION

Reconstructing a 3D object from multi-view images seeks to obtain accurate and
compact geometry and photo-realistic textures. Decomposing the 3D object from
the background plays a crucial role in 3D scene understanding and can be
applied to various object-oriented applications, such as virtual reality,
autonomous driving, and robotic manipulation. Recently, neural implicit
representations [1, 2] significantly enhance the reconstruction quality, yet
it is still challenging to reconstruct a target foreground object while
decomposing it from the background without costly manual annotation.

In order to address the task of decomposed object reconstruction, [3, 4]
incorporate 2D masks as observations. The expensive manual annotation can be
avoided by obtaining the 2D masks from pre-trained segmentation models.
However, such coarse-grained masks are prone to errors, directly affecting the
precision of decomposed modeling (see Fig. LABEL:fig:teasesr, SemanticNeRF).
In contrast to leveraging the coarse-grained masks, another line of work
exploits fine-grained features, typically obtained from 2D self-supervised
models, for decomposed reconstruction. Several methods propose distilling such
features into the 3D space, allowing for decomposing the foreground in a post-
processing step [5], e.g., identifying a specific 3D region based on the
feature vector of a user query. While demonstrating promising performance,
such decomposition via post-processing makes them susceptible to noises and
false classifications (Fig. LABEL:fig:teasesr, DFF).

In this paper, we propose to utilize self-supervised 2D features of varying
granularity as guidance for decomposed object reconstruction. Our key idea is
to transfer self-supervised 2D features to two forms of masks for
decomposition, with a coarse-grained mask guiding the coarse decomposition and
a median-grained mask contributing to noise resistance. First, we observe that
self-supervised features with median granularity, e.g., a segmentation mask of
K-clusters, are less prone to erroneous classification than the binary mask,
as shown in Fig. LABEL:fig:teasesr. The median-grained clustering can be
easily applied to encourage decomposed reconstruction during training,
avoiding extracting certain regions via post-processing. Second, using the
median-grained features alone might lead the network to converge to
alternative feasible solutions due to the lack of clear foreground-background
guidance. We thus utilize the coarse-grained masks to provide rough yet
distinct boundary indications, which can offer a reliable convergence path for
the network in the early training stages. More specifically, we deploy a
compositional network, where the foreground model is a neural surface
representation for high-quality object geometries and the background model is
a neural radiance field for unbounded scenes. The network is supervised by the
RGB reconstruction loss and the segmentation losses using both the coarse-
grained and median-grained masks. We propose a training strategy that allows
the network effectively leverage both masks for decomposition. After training,
our model can render accurate foreground object masks as well as extract the
surface of the foreground object.

In summary, the major contributions of this work are: i) To address decomposed
object reconstruction problem, we propose a compositional neural implicit
network that utilizes 2D self-supervised features for decomposition, sparing
pixel-wise annotations. ii) We propose to utilize self-supervised features of
varying granularity, with the coarse-grained features for early guidance and
the median-grained features for noise resilience. iii) Our method obtains high
reconstruction quality and neat segmentation results, which is proved by
experiments on various object-centered and cluttered real datasets.

##  II Related Works

###  II-A Neural Implicit Representations

Traditional object reconstruction methods usually adopt discrete
representations, including voxel-based [6, 7], point-based [8, 9] and mesh-
based [10, 11]. Neural implicit representations [2, 12, 1] consider the 3D
space as a continuous function by mapping each 3D point to a signed distance
to the surface or an occupancy value, enabling high reconstruction accuracy
and thus gain popularity [13, 14, 15, 16, 17, 18]. However, these methods
require 3D supervision.

Neural Radiance Fields (NeRF) [19] proposes to map each 3D point to a density
value and a color taking viewing directions into consideration, enabling high-
fidelity novel view synthesis using multi-view 2D supervision [20, 21, 22,
23]. However, extracting surfaces from the density field of NeRF typically
yields unsatisfactory performance. A line of works hence replaces the density
field with a signed distance field (SDF) or an occupancy field to improve the
reconstruction quality [24, 25, 13, 26]. These methods focus on reconstructing
the full scene and are not designed for decomposing an arbitrary target
object.

###  II-B Decomposed Neural Implicit Representations

Decomposing neural implicit representations into different parts is useful for
scene editing [27, 28], object-level reconstruction [4], and generative models
[29]. Most of the decomposed neural implicit methods depend on 2D annotations
to guide the decomposition. SemanticNeRF [3] first proposes to add a semantic
head for predicting a semantic logit at every 3D point, enabling scene
classification while reconstructing. Similar ideas are further explored in
later works for semantic segmentation [30], panoptic segmentation [31], and
decomposed reconstruction [4]. With dense and accurate 2D annotations as
supervision, these methods achieve outstanding segmentation and reconstruction
performance. But the requirement of massive human labeling impedes their
broader practice. Other approaches utilize category-level knowledge [30, 32]
or object motion [33, 34] to help decomposition. In contrast, we focus on
reconstructing a prominent foreground object of arbitrary category in a
cluttered scene without assuming it is dynamic.

###  II-C Decomposed Neural Implicit Representations

Recently, with the development of transformer architectures [35, 36], self-
supervised learning methods have improved the generalization of image vision
models [37, 38]. Some methods explore how to decompose the neural implicit
representations without 2D annotations or prior knowledge [5] by utilizing 2D
features from pre-trained self-supervised networks. NeRF-SOS [39] adopts
unsupervised 2D features from visual transformer network DINO [37] to
constraint segmentation similarities and geometrical similarities between
patches. DFF [40] also exploits features from DINO, via supervising a proposed
feature field with DINO feature maps. We share similar ideas with them of
using the 2D features from pre-trained methods as foreground-sensitive
guidance, but argue that combining different types of features from DINO
yields a more stable result. We compare with them on real datasets to validate
our method. AutoRecon [41] aggregates DINO features to point clouds to perform
coarse foreground segmentation. They rely on the assumption of a planar
background for decomposition, thus hard to handle complex environments.

Figure 1: Method overview. The left part in blue shows our decomposed network
consisting of a background model $f_{\theta}$ and a foreground model
$f_{\phi}$. The background predictions $(\mathbf{c}_{b},\sigma_{b})$ and
foreground predictions $(\mathbf{c}_{f},\rho_{f})$ are blended together via
point-wise max composition. The right part in pink illustrates how we leverage
self-supervised 2D features of varying granularity to enable decomposed
reconstruction. The coarse-grained binary mask is noisy but provides a clear
indication of the foreground region. The median-grained mask is less prone to
error, leading to a more accurate decomposition.

##  III Preliminaries

###  III-A Surface-based Implicit Representation

We use neural surface-based implicit representation as our foreground model
for extracting the object’s surface.

Let $\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}$ be a camera ray emitted from the
focal principle $\mathbf{o}$ of a camera with a unit-norm viewing direction
$\mathbf{d}$, a set of 3D points $\mathbf{x}$ are sampled on the ray within a
bounded 3D volume. Given $(\mathbf{x},\mathbf{d})$ as inputs, we follow [42]
to project each input to a color value $\mathbf{c}(t)$ and an SDF value $s(t)$
with MLPs. Then, the opaque density $\rho(t)$ is computed as

$\rho(t)=\max\left(\frac{-\frac{d\Phi}{dt}(s(t))}{\Phi(s(t))},0\right).$ (1)

The sigmoid function $\Phi(s(t))=(1+e^{-\beta s(t)})^{-1}$ is the integral of
the logistic density distribution of the SDF value. To obtain pixel color
$\mathbf{C}(\mathbf{r})$ and accumulated weight $W(\mathbf{r})$, volume
rendering is adopted:

$\mathbf{C}(\mathbf{r})=\int_{n}^{f}T(t)\alpha(t)\mathbf{c}(t)dt,\quad
W(\mathbf{r})=\int_{n}^{f}T(t)\alpha(t)dt,$ (2)

where $n$ and $f$ refer to the near and far points for integration, $T(t)$ is
the accumulated transmittance defined as [19], and $\alpha(t)$ is the alpha
value derived from $\rho(t)$ [42].

###  III-B Self-supervised Visual Transformer

Vision Transformers (ViT) [35] split an input image into a sequence of fixed-
size patches with position embedding, and feed the sequence to a Transformer
encoder. To perform image classification, an extra learnable embedding
”classification token”, noted as [CLS], is added to the sequence. Perceiving
self-attention and cross-attention with the patches in the encoder, the output
of the [CLS] token goes through an MLP layer to obtain the final
classification probabilities. DINO [37] deploys the ViT in a self-supervised
training manner, providing foreground-sensitive features without a predefined
label set.

##  IV Methods

Given a set of N images $\left\\{\mathbf{I}_{1},...,\mathbf{I}_{N}\right\\}$
and the camera poses
$\left\\{\boldsymbol{\xi}_{1},...,\boldsymbol{\xi}_{N}\right\\}$, our goal is
to learn decomposed object reconstruction that separates the background from
the target object and vice versa.

In the following, we first introduce the whole pipeline and how to composite
the foreground and background models. Next, we show how we leverage 2D self-
supervised features for decomposition. Last, we present training details.

###  IV-A Decomposed Neural Networks

Pipeline. As shown in Fig. 1, our method takes a 3D point position
$\mathbf{x}$ and a viewing direction $\mathbf{d}$ as input, and then processes
them with a background model and a foreground model, respectively. We follow
[20] to build the background model, utilizing MLPs to obtain a density value
$\sigma_{b}$ and a color vector $\mathbf{c}_{b}$:

$f_{\theta}:(\mathbf{x},\mathbf{d})\mapsto(\mathbf{c}_{b},\sigma_{b}).$ (3)

In order to reconstruct the surface of the foreground object, we follow [42]
to predict an SDF value and a color vector $\mathbf{c}_{f}$, where the SDF
value is transferred to a point opaque density $\rho_{f}$:

$f_{\phi}:(\mathbf{x},\mathbf{d})\mapsto(\mathbf{c}_{f},\rho_{f}).$ (4)

A point-wise composition mechanism is then designed to blend the outputs of
two models to get unified outputs $\mathbf{c}$ and $\sigma$. Last, we use
volume rendering (Eq. 2) to obtain pixel-wise colors $\mathbf{C}$ and
accumulated weight $W$. Thanks to the independent branch design, we are able
to obtain the rendering results for the target object by rendering with the
outputs of the foreground model. Also, object surfaces could be extracted by
applying marching cubes to the outputs of the foreground model.

Figure 2: Point-wise max composition. Top: NeuS models regions in a unit
sphere using a foreground model and everything else using a background model.
Bottom: We model points inside the unit sphere using both the foreground and
background models. The values are combined via max composition, meaning the
lower density is ignored.

Point-wise max composition. Following [20], we adopt inverted sphere
parameterization to partition the scene into an inner unit sphere and an outer
volume. As illustrated in Fig. 2, the inner sphere is jointly modeled with the
foreground model $f_{\theta}$ and the background model $f_{\phi}$, whereas the
rest is represented using the background model alone. This allows us to
decompose the background region in the unit sphere from the foreground model.

For each point within the init sphere, we adopt point-wise composition to fuse
the foreground and the background models. In our preliminary experiments, we
experimentally observe that a simple max composition leads to better
decomposition:

$\displaystyle\sigma$ $\displaystyle=\max(\sigma_{b},\rho_{f})$ (5)
$\displaystyle\mathbf{c}$
$\displaystyle=\mathbf{c}_{b}\mathbbm{1}_{\sigma_{b}>\rho_{f}}+\mathbf{c}_{f}\mathbbm{1}_{\sigma_{b}<\rho_{f}}$
(6)

where $\mathbbm{1}_{\star}\rightarrow\\{0,1\\}$ refers to an indicator
function.

###  IV-B Leveraging 2D Self-Supervised Features

Decomposing the foreground and the background without manually labeling is
challenging. With the advancement of pre-trained segmentation models or self-
supervised models, existing methods utilize either coarse-grained binary masks
or fine-grained features to facilitate decomposition. We advocate the usage of
self-supervised 2D features for decomposition. In contrast to existing
methods, we propose to transfer self-supervised 2D features to masks of
different granularities, i.e., median-grained and coarse-grained.

Median-grained mask supervision. We extract median-grained masks from self-
supervised 2D features to facilitate decomposed reconstruction. As shown in
Fig. 3, we follow [38] to pair each input image with the reference image in
the session and use DINO with ViT-S/8 architecture to individually extract
their keys descriptors and attention maps of the [CLS] token of the last
transformer layer. Then, we apply the K-Means algorithm to get clusters
features of the key descriptors from both images, which represent the common
feature clusters that appear on both views. Note that this process is
conducted offline. Our key observation is that median-grained masks are less
prone to error, i.e., the same cluster often belongs to the same foreground or
background class. Thus, we design a cluster mask loss to enforce that the
pixels belonging to the same cluster should fall into the same class.

Given a set of M sampled rays, we retrieve their cluster values
$\left\\{l_{i}|l_{i}\in\mathcal{S}\right\\}_{i=1}^{M}$, where
$\mathcal{S}=\left\\{s_{j}\right\\}_{j=1}^{K}$ is the cluster index set. We
group sampled rays of the same cluster index and compute the average
foreground weight of each cluster index $s_{j}$:

$g_{s_{j}}=E(\\{W_{f_{i}}|l_{i}=s_{j}\\})$ (7)

where $E(\cdot)$ is an average operation and $W_{f}$ is the foreground weight
at one sampled ray. Next, we use the binary cross entropy loss to push the
average foreground weight of each cluster to either $0$ or $1$:

$\mathcal{L}_{cls_{s_{j}}}=-(g_{s_{j}}\log
g_{s_{j}}+(1-g_{s_{j}})\log(1-g_{s_{j}}))$ (8)

We sum up the losses of every cluster to get the final cluster mask loss

$\mathcal{L}_{cls}=\sum_{j\in K}\mathcal{L}_{cls_{s_{j}}}$ (9)

As illustrated in Fig. 1, $\mathcal{L}_{cls}$ is applied to each cluster.
Hence it is cluster-level supervision instead of pixel-level.

Coarse-grained mask supervision. By utilizing the cluster mask loss, rays
belonging to the same feature cluster are encouraged to be classified into the
same class, aiding in a clean decomposition. However, this per-cluster loss
alone does not specify which region should be in the foreground, resulting in
many possible solutions that can drive the loss close to zero, which is
unfavorable for the initial convergence of the network. Therefore, we propose
integrating coarse-grained binary masks to guide early-stage training of the
network.

Figure 3: Median-grained and coarse-grained masks obtained from self-
supervised 2D features.

As shown in Fig. 3, we follow [38] to utilize a simple voting procedure to
select clusters that are salient and common to both images as our binary mask,
indicating a separation between the foreground and the background. We deploy
the binary masks as supervision for the weights map rendered from the
foreground model by the binary cross entropy loss $\mathcal{L}_{seg}$. This
per-pixel loss provides a clear preference of which region should be
classified as foreground.

###  IV-C Training

Losses. Following [19], we minimize the reconstruction error between the
predicted color $\hat{\mathbf{C}}(\mathbf{r})$ and the ground truth color
$\mathbf{C}(\mathbf{r})$

$\mathcal{L}_{color}=\|\hat{\mathbf{C}}(\mathbf{r})-\mathbf{C}(\mathbf{r})\|_{2}^{2}$
(10)

Since we represent the foreground object as implicit surfaces, an Eikonal loss
$\mathcal{L}_{eik}$ [43] is utilized to regularize the predicted SDF values

To prevent the background model from representing all of the points inside the
sphere, we add a loss to punish the foreground model when its weighted alpha
sum approaches $0$

$\mathcal{L}_{fg}=-log(\frac{\beta}{B}\sum_{i}(\mathbbm{1}_{W_{f_{i}}>W_{th}}))$
(11)

where $W_{f}$ is the weighted alpha sum from the foreground model, $W_{th}$ is
a threshold value, $\beta$ is a scale hyper-parameter, and $B$ is the number
of rays.

Combing with the median-grained and coarse-grained mask supervision, our full
loss function is as follows:

$\mathcal{L}=\gamma_{1}\mathcal{L}_{color}+\gamma_{2}\mathcal{L}_{eik}+\gamma_{3}\mathcal{L}_{fg}+\gamma_{4}\mathcal{L}_{seg}+\gamma_{5}\mathcal{L}_{cls}$
(12)

Training strategy. We experimentally observe that the training strategy is
important for achieving stable decomposition. Specifically, we first train the
model from scratch using $\mathcal{L}_{color}$, $\mathcal{L}_{eik}$,
$\mathcal{L}_{fg}$ and $\mathcal{L}_{seg}$, as we observe that the coarse-
grained mask supervision $\mathcal{L}_{seg}$ provides a clear indicator of
foreground/background separation, leading to a reasonable initial
decomposition. Next, we fine-tune the full model with all losses, where
$\mathcal{L}_{cls}$ plays the main role in decomposing the foreground.

##  V Experiments

Dataset | BMVS | T&T | OnePose | YCB Video
---|---|---|---|---
DINO-Coseg | 0.765 | 0.890 | 0.840 | 0.667
SAM | 0.540 | 0.603 | 0.765 | 0.755
SNeRF+D | 0.747 | 0.916 | 0.826 | 0.699
DFF | / | / | 0.862 | 0.799
NeuS | 0.590 | 0.558 | 0.397 | 0.478
NeuS+D | 0.747 | 0.838 | 0.807 | 0.613
NeuS+S | 0.809 | 0.603 | 0.932 | 0.843
Ours | 0.876 | 0.946 | 0.961 | 0.938
TABLE I: Quantitative results of segmentation performance. We report mean mIoU
on each dataset to evaluate the decomposition performance.

The main purpose of our method is to build a decomposed neural implicit
representation for target objects in cluttered scenes. We evaluate our
approach to answer two questions. First, can our method reconstruct compact
object geometries and obtain neat segmentation results when synthesizing novel
views? Second, does our design in utilizing 2D unsupervised features enhance
the performance? To answer these questions, we quantitatively compare our
method with the state-of-the-art methods with the same setting in public
datasets.

Dataset | BlendedMVS | T&T | YCB Video
---|---|---|---
SceneID | 1 | 2 | 3 | Mean | 1 | 1 | 2 | 3 | 4 | 5 | Mean
NeuS | 0.073 | 0.161 | 1.075 | 0.436 | 0.105 | 0.120 | 0.098 | 0.059 | 0.326 | 0.085 | 0.138
SNeRF+D | 0.093 | 0.129 | 0.336 | 0.186 | 0.079 | 0.133 | 0.114 | 0.036 | 0.319 | 0.069 | 0.134
NeuS+D | 0.140 | 0.122 | 0.339 | 0.200 | 0.086 | 0.153 | 0.170 | 0.082 | 0.302 | 0.057 | 0.153
NeuS+S | 0.072 | 0.117 | 0.451 | 0.213 | 0.081 | 0.135 | 0.131 | 0.041 | 0.334 | 0.063 | 0.141
Ours | 0.068 | 0.090 | 0.170 | 0.109 | 0.054 | 0.111 | 0.086 | 0.037 | 0.298 | 0.064 | 0.119
TABLE II: Quantitative results of reconstruction performance. We report CD to
measure the reconstruction quality.

###  V-A Datasets and metrics

Datasets. We conduct experiments on 4 different real world datasets to
validate our method. BlendedMVS (BMVS) [44] and Tanks and Temples (T&T) [45]
are commonly used in 3D reconstruction. Since we aim at object-level
reconstruction, we use their object-centered scenes, specifically 3 scenes and
1 scene respectively. We manually label the test images and remove the
background in the provided meshes as ground truths. OnePose [46] is an object-
centered dataset, containing common domestic products placed on different
platforms and observed from top-down views. We use three objects from OnePose,
splitting each video into train and test sets, and segmenting the test sets
for evaluation. YCB Video [47] is a more challenging dataset, where objects
could be placed on a non-planar background. We follow [20] to place a sphere
to enclose the target object. In the ablation study, we demonstrate that our
method is robust to the radius and the center of the sphere. We use five
objects from YCB, evenly split the observations into train/test, and only use
the training RGBs and cameras to refine the provided camera poses by [48].

Metrics. We employ the following metrics for evaluation: mIoU (mean
Intersection-over-Union) to evaluate the decomposed segmentation results, and
CD (Chamfer Distance) [49] to measure the quality of reconstructed 3D
geometry.

Figure 4: Qualitative segmentation results in Onepose, YCB Video, and BMVS
datasets.

###  V-B Implementation details

For each object, the RGBs are paired with the first image to obtain DINO
features. For YCB Video scenes, we use the bounding rectangle of the projected
sphere on the 2D plane as a bounding box and extract DINO features from the
images within the box. For other scenes, since they are already object-
centered, we use the whole image for DINO. We train every object for 200k
iterations with color loss, eikonal loss, foreground regularization loss, and
binary mask loss from scratch, and finetune with cluster mask loss from 150k.
The implementation details of the network are presented in the AppendixDORec:
Decomposed Object Reconstruction Utilizing 2D Self-Supervised Features.

###  V-C Baselines

2D Self-supervised Baselines. We compare our method with state-of-the-art 2D
self-supervised methods to demonstrate that our approach can achieve accurate
segmentation accuracy, indicating the completeness and exclusiveness of our
reconstruction. DINO-coseg [38] is a co-segmenting visual transformer, which
provides features for our network, so we directly compare with their masks.
SAM [50] is a cutting-edge foundation model trained on large-scale data and
achieves impressive performance. We compare with it to show that naively
deploying masks for reconstruction presents inferior performance even with the
most promising methods. In object-centered datasets, we use the whole image as
a prompt to obtain a foreground and background mask. While in the non-object-
centered dataset YCB Video, we use the same 2D bounding boxes as we use in
DINO as prompts.

3D Reconstruction Baselines. We compare our method with other state-of-the-art
reconstruction methods to demonstrate that our approach can achieve more
complete and accurate models, thereby highlighting the effectiveness of our
application of self-supervised features. NeuS [42] models foreground and
background separately. We compare with NeuS to show that surface-based
representation brings compact geometry in reconstruction, but neatly
decomposing target objects from the background requires more guidance.
SemanticNeRF [3] is a radiance field framework that predicts 3D semantic
fields while reconstructing with mask supervision. An intuitive method to
adapt SemanticNeRF in our setting would be training with masks predicted by
[38]. We implement such a pipeline (SNeRF+D) by ourselves with the same masks
we use for fair comparisons. DFF [40] is a radiance field network that adopts
2D self-supervised features to supervise the learning of the proposed 3D
feature fields. Note that DFF requires a reference patch as input to extract
the interested region. We use a GT object mask at a single view to select the
target object.

###  V-D Comparison to the State of the Art

Foreground Segmentation.  We report the segmentation results in all datasets
in Tab. VIII. Our method outperforms other baselines in most of the scenes
with mIoU metrics on all the datasets. For the 2D self-supervised learning
methods (DINO-Coseg and SAM), they deploy 2D image feature attentions to
separate interested foregrounds from backgrounds, tending to falsely response
to textures or object shadows. As for the 3D reconstruction methods, the
performance of NeuS shows the results without neither background modeling
inside the sphere nor utilization of 2D self-supervised features, which could
be considered as a naive baseline of our method. SNeRF+D and NeuS+S rely on 2D
self-supervised masks as supervisions. They present similar results to their
2D counterparts due to a strong ability to fit the given masks. DFF learns a
feature field to aggregate self-supervised features from multi-frames, but
they neglect the similarities for the same cluster, and require a query
feature for post segmentation extracted by DINO from an input image and mask.
Due to the larger image sizes on the BMVS and TT datasets, we have to use
larger strides to extract dino features on our server. However, we observe
that training DFF with such coarse dino features yield less satisfactory
results. Thus, we do not include the results on these two datasets. Fig. 4
shows the qualitative results in OnePose, BlendedMVS, and YCB Video datasets.
Our method decomposes the target object from the background neatly.

Foreground Reconstruction. To validate the object-level reconstruction and
decomposition ability in different objects and environments, we compare our
method in 3 datasets with 3D reconstruction baselines. Since the OnePose
dataset doesn’t provide ground truth meshes, the comparison is spared. To
extract the mesh, we condition all the 3D points in the sphere with our
foreground network, and extract the surface with zero SDF value for marching
cubes [51]. The process is the same when extracting meshes from other NeuS-
based methods. While for the NeRF-based methods (SNeRF+D), we condition the
same set of 3D points in the neural radiance field to get density values for
marching cubes. To evaluate the reconstruction accuracy with the provided
ground truth mesh, we adopt a trajectory registration algorithm [52] to align
our reconstructed cameras with the ground truth cameras. The aligned
transformation is shared among all methods for fair comparisons.

As shown in Table. II, our method performs the best in most of the scenes,
which shows our reconstruction accuracy and completeness. Still, NeuS serves
as a naive baseline showing the undecomposed results, indicating the
difference between the object mesh and the background-contained mesh. As a
NeRF-based method, SNeRF+D shows inferior results in reconstruction. NeuS+S
takes advantage of the strong surface implicit representation and state-of-
the-art unsupervised segmentation masks, demonstrating even better results in
some of the cases. But their performance is unstable, showing their
sensitivity to the supervision masks. Fig. 5 displays the qualitative results
of our method and the baselines in Tanks and Temples, YCB Video, and
BlendedMVS datasets. Notice that the result of the scene in YCB Video (column
3) demonstrates our ability to decompose the target object from non-planar
backgrounds.

Figure 5: Qualitative reconstruction results in Tanks and Temples, YCB Video, and BlendedMVS datasets.  Figure 6: Training procedure. Segmentation results are improved by finetuning with our proposed cluster mask loss. | s0 | s1 | s2 | s3
---|---|---|---|---
binary mask loss (sc) |  | ✓ |  | ✓
cluster mask loss (sc) |  |  | ✓ |
cluster mask loss (ft) |  |  |  | ✓
scene 1 | 0.562 | 0.763 | 0.967 | 0.966
scene 2 | 0.642 | 0.705 | 0.319 | 0.958
TABLE III: Ablation study on training strategies. Here, mIoU is reported to indicate the decomposition performance.  offset | 0 | 0 | 0 |  5$\%$ |  10$\%$
---|---|---|---|---|---
scale | 1.0 | 0.9 | 1.2 | 1.0 | 1.0
mIoU | 0.920 | 0.924 | 0.920 | 0.918 | 0.922
TABLE IV: Ablation study on sphere parameters.

###  V-E Ablation studies

Effectiveness in training strategy. To validate the effectiveness of our
training strategy in utilizing two kinds of features, we conduct ablation
studies on two objects from each of the two datasets. Table. III demonstrates
the performance difference, in which (sc) indicates using the loss to train
from scratch, and (ft) refers to using it to finetune. As can be seen, when
neither binary mask loss nor cluster mask loss is deployed (s0), it’s
difficult for the network to decompose. When binary mask loss is provided
(s1), the decomposing results are better, but artifacts still exist due to the
noises in segments. When only cluster mask loss is utilized during training
(s2), although one of the objects displays outstanding performance in
decomposing, it fails on the other object. This demonstrates that as features
with higher granularity, they tend to guide the network towards alternative
yet unwanted solutions. Therefore, training with binary mask loss from scratch
and then finetuning with cluster mask loss (s3) achieves the most accurate and
stable results. We choose the s3 strategy to deploy all of the experiments in
this paper. Fig. 6 shows the improvements in segmentation results by
finetuning with cluster mask loss. Notice that the artifacts are gradually
eliminated by encouraging these pixels to follow others in the same cluster.

Sensitivities to sphere parameterization. We follow [42] to use the inverted
sphere parameterization, where the sphere determines the potential object
range. Therefore, we design ablation studies to investigate the influence of
sphere parameter perturbations on our results. We scale the sphere radius, and
also introduce offsets to the sphere center, where the magnitude of the offset
is given as a percentage of the sphere’s radius. As shown in Table. IV, the
variations in sphere parameters within certain limits do not affect our
results significantly, highlighting the robustness of our method’s
decomposition ability.

##  VI Conclusion

In this work, we propose a Decomposed Object Reconstruction method (DORec)
based on neural implicit representation, which learns to decompose the target
object from the background while reconstructing. Our method transfers 2D self-
supervised features into masks of two levels of granularity to supervise the
decomposition, including a binary mask to indicate the foreground regions and
a K-cluster mask to indicate the semantically similar regions. Extensive
experimental results on four real datasets have demonstrated the strong
ability of our method in both segmentation and object-level reconstruction.
One of the limitations of our method is the training process still takes
hours, and we plan to accelerate it with advanced network architectures or
training strategies in the future.

## References

  * [1] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
  * [2] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019.
  * [3] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [4] Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng.  Object-compositional neural implicit surfaces.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVII. Springer, 2022.
  * [5] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu.  Unsupervised discovery of object radiance fields.  arXiv preprint arXiv:2107.07905, 2021.
  * [6] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 628–644. Springer, 2016.
  * [7] Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang.  Surfacenet: An end-to-end 3d neural network for multiview stereopsis.  In Proceedings of the IEEE international conference on computer vision, 2017.
  * [8] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
  * [9] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
  * [10] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  A papier-mâché approach to learning 3d surface generation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
  * [11] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
  * [12] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16. Springer, 2020.
  * [13] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman.  Multiview neural surface reconstruction by disentangling geometry and appearance.  Advances in Neural Information Processing Systems, 33, 2020.
  * [14] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt.  Neural sparse voxel fields.  Advances in Neural Information Processing Systems, 33, 2020.
  * [15] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
  * [16] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  Advances in Neural Information Processing Systems, 32, 2019.
  * [17] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [18] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson.  Implicit surface representations as layers in neural networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [19] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [20] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  Nerf++: Analyzing and improving neural radiance fields.  arXiv preprint arXiv:2010.07492, 2020.
  * [21] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [22] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised nerf: Fewer views and faster training for free.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [23] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Transactions on Graphics (ToG), 41(4):1–15, 2022.
  * [24] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger.  Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction.  arXiv preprint arXiv:2206.00665, 2022.
  * [25] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34, 2021.
  * [26] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [27] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.  Learning object-compositional neural radiance field for editable scene rendering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
  * [28] Silvan Weder, Guillermo Garcia-Hernando, Aron Monszpart, Marc Pollefeys, Gabriel Brostow, Michael Firman, and Sara Vicente.  Removing objects from neural radiance fields.  arXiv preprint arXiv:2212.11966, 2022.
  * [29] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [30] Siddhant Ranade, Christoph Lassner, Kai Li, Christian Haene, Shen-Chi Chen, Jean-Charles Bazin, and Sofien Bouaziz.  Ssdnerf: Semantic soft decomposition of neural radiance fields.  arXiv preprint arXiv:2212.03406, 2022.
  * [31] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao.  Panopticnerf-360: Panoramic 3d-to-2d label transfer in urban scenes.  In International conference on 3D vision, 2023.
  * [32] Christopher Xie, Keunhong Park, Ricardo Martin-Brualla, and Matthew Brown.  Fig-nerf: Figure-ground neural radiance fields for 3d object category modelling.  In 2021 International Conference on 3D Vision (3DV). IEEE, 2021.
  * [33] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
  * [34] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli.  D2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video.  2022\.
  * [35] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [36] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
  * [37] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, 2021.
  * [38] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep vit features as dense visual descriptors, 2022.
  * [39] Zhiwen Fan, Peihao Wang, Yifan Jiang, Xinyu Gong, Dejia Xu, and Zhangyang Wang.  Nerf-sos: Any-view self-supervised object segmentation on complex scenes.  arXiv preprint arXiv:2209.08776, 2022.
  * [40] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing nerf for editing via feature field distillation.  arXiv preprint arXiv:2205.15585, 2022.
  * [41] Yuang Wang, Xingyi He, Sida Peng, Haotong Lin, Hujun Bao, and Xiaowei Zhou.  Autorecon: Automated 3d object discovery and reconstruction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [42] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  arXiv preprint arXiv:2106.10689, 2021.
  * [43] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.  Implicit geometric regularization for learning shapes.  arXiv preprint arXiv:2002.10099, 2020.
  * [44] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
  * [45] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: Benchmarking large-scale scene reconstruction.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [46] Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou.  Onepose: One-shot object pose estimation without cad models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [47] Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  arXiv preprint arXiv:1711.00199, 2017.
  * [48] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
  * [49] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, 2014.
  * [50] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [51] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  ACM siggraph computer graphics, 21(4):163–169, 1987.
  * [52] Shinji Umeyama.  Least-squares estimation of transformation parameters between two point patterns.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04):376–380, 1991.

Figure 7: Network Architecture of DORec.

###  -A Implementation Details of DORec

Network Architecture. Fig. 7 shows the detailed architecture of our DORec
model. We adopt the same network architecture in all experiments. We follow
[42] to build our foreground model and background model.

The foreground model takes as input the 3D location $\mathbf{x}$ to predict
the signed distance field value $\mathbf{s}_{f}$. Then it calculates the
point-wise gradients $\mathbf{n}=d\mathbf{s}/d\mathbf{x}$, and concatenates
them with the feature vector, the viewing direction $\mathbf{d}$, and the
location $\mathbf{x}$ to predict the color field value $\mathbf{c}_{f}$. The
background model takes as input the 3D location $\mathbf{x}$ to predict the
density field value $\sigma_{b}$. Then it combines the viewing direction
$\mathbf{d}$ to get the color field value $\mathbf{c}_{b}$.

Following NeRF [19], both $\mathbf{x}$ and $\mathbf{d}$ are mapped to a higher
dimensional space using a positional encoding (PE):

$\begin{split}\gamma(\mathbf{p})=(sin(2^{0}\pi\mathbf{p}),cos(2^{0}\pi\mathbf{p}),...,\\\
sin(2^{L-1}\pi\mathbf{p}),cos(2^{L-1}\pi\mathbf{p}))\end{split}$ (13)

In the foreground model, we set $L=6$ for $\gamma(\mathbf{x})$ and $L=4$ for
$\gamma(\mathbf{d})$. In the background model, we set $L=10$ for
$\gamma(\mathbf{x})$ and $L=4$ for $\gamma(\mathbf{d})$.

Sampling Strategy. We normalize the scene, placing the sphere at the center
with a radius of 1. We sample 128 points for the foreground model between the
range of [-1,1] with the hierarchical sampling strategy proposed by [42].
Specifically, we first uniformly sample 64 points along the ray, then
iteratively conduct importance sampling for 4 times. In each iteration, we
additionally sample 16 points. Then we sample extra 64 points outside the
normalized sphere and normalize them following [20]. We concatenate the
outside points (64) with the inside points (128) together to feed the
background model. When extracting mesh, we evenly sample $512\times 512\times
512$ points inside a cube with the radius of the sphere and use their
predicted SDF value for marching cubes [51].

Training Strategy. We randomly select one image from the trainset and sample
512 rays in each iteration. We use the Adam optimizer to train our networks.
The learning rate is first linearly warmed up from 0 to $5\times 10^{-4}$ in
the first 5k iterations, and then controlled by the cosine decay schedule to
the minimum learning rate of $2.5\times 10^{-5}$. We train each model for 150k
iterations with the losses $\mathcal{L}_{color}$, $\mathcal{L}_{eik}$,
$\mathcal{L}_{fg}$, and $\mathcal{L}_{seg}$ mentioned in the paper, then add
$\mathcal{L}_{clu}$ to finetune for another 50k iterations. The loss weights
${\gamma}_{i}$ are set to 1.0, 0.1, 0.01, 0.1, and 0.1, respectively. Every
model is trained on a single GTX 3090Ti GPU.

Applying DINO-Coseg. As described in the paper, we propose to transfer self-
supervised 2D features to masks of different granularities. We use the
official implementation of [38]. We adopt the ViT-S/8 network with a stride of
4 for the first convolutional layer and set the maximum cluster number as 16.
The feature-obtaining process is conducted before model training. It takes
about 2.5s to process one image in an NVIDIA 3090Ti GPU. We refer our readers
to [38] for more feature extraction network details.

###  -B Implementation Details of Baseline Methods

DINO-Coseg: For DINO-Coseg [38], we use the same coarse-grained binary masks
as our method use for fair comparison.

SAM: For SAM [50], we use the official implementation provided by the authors.
We adopt the public trained model and set the prompt point per side as 1 to
get a binary mask. In BlendedMVS, OnePose, and Tanks and Temples datasets, we
use the whole image as a prompt to obtain a foreground and background mask.
While in the non-object-centered dataset YCB Video, we use the same 2D
bounding boxes as we use as prompts.

NeuS: For NeuS [42], we use the official implementation provided by the
authors, except that we mask out the black edges in the images in OnePose
dataset when calculating color loss. For NeuS+D and NeuS+S, we use the DINO-
Coseg masks and SAM masks respectively.

SemanticNeRF: For SemanticNeRF [3], we also use the official implementation
provided by the authors. We use the binary we obtained from DINO-Coseg as
supervision in training SemanticNeRF. In extracting mesh, we also uniformly
sample $512\times 512\times 512$ points in the cubic encasing the sphere to
predict their densities and semantics, then use argmax to choose those points
belonging to foreground, and adopt marching cubes to generate mesh.

DFF: For DFF [40], we use the official implementation provided by the authors.
Since the code for DINO feature extraction is not presented, we follow the
introductions in their paper [40] to deploy [38] for feature extracting. Also,
DFF requires a reference patch as input to extract the interested region. We
use the GT object mask to choose the object region in the first input image,
then calculate the mean feature vector in the region as the query feature to
select the target object in every input image. We consider annotating a GT
object mask at one view of a similar cost of annotating a sphere, and thus we
do not apply a sphere projection to post-process the DFF results.

###  -C Datasets

OnePose Dataset: OnePose dataset [46] is firstly proposed for evaluating one-
shot object pose estimation methods. We choose this dataset to evaluate our
method because it contains various common products in different environments
with real lighting. We downsample the original image sequences with a ratio of
10 to get training images, and randomly sample from the rest to get testing
images. Tab. V shows the specific cases, sequences and numbers of images we
use for training and testing. We use the official implementation provided by
the authors to obtain the poses and a sparse pointcloud of each object. Since
no ground truth masks are provided in the dataset, we manually label all the
testing images.

Scene | Case | Sequence | Train# | Test#
---|---|---|---|---
1 | 0413-juliecookies-box | 3 | 63 | 17
2 | 0414-babydiapers-others | 1 | 73 | 18
3 | 0415-captaincup-others | 3 | 63 | 17
TABLE V: Sequences and images used in OnePose dataset. Scene | Object | Sequence | Train# | Test#
---|---|---|---|---
1 |  003$\\_$cracker$\\_$box | 0007 | 100 | 27
2 |  004$\\_$sugar$\\_$box | 0049 | 100 | 27
3 |  006$\\_$mustard$\\_$bottle | 0037 | 100 | 27
4 |  008$\\_$pudding$\\_$box | 0002 | 86 | 24
5 |  021$\\_$bleach$\\_$cleanser | 0036 | 100 | 27
TABLE VI: Sequences and images used in YCB Video dataset.

Tanks and Temples Dataset: For the Tanks and Temples dataset [45], we directly
use the data published by [20] for fair comparisons. We use the Truck scene to
validate our performance on object-centered unbounded scenes. We also manually
label all the testing images for no ground truth masks are provided. Since the
goal of our method is to decompose the object from backgrounds, we manually
remove the meshes of the ground surface in the ground truth mesh file, and use
this edited mesh when evaluating CD metric for all experiments.

BlendedMVS Dataset: For the BlendedMVS dataset [44], we use the data published
by [42] and [41] for fair comparisons. We use the Bear, Stone, and Gundam
scene to validate our performance on more object-centered scenes. We select
images with indices that are multiples of 10 from each sequence as the test
set, and the rest are used as the training set. We also manually label all the
testing images for no ground truth masks provided. Since the goal of our
method is to decompose the object from backgrounds, we manually remove the
meshes of the ground surface in the ground truth mesh file, and use this
edited mesh when evaluating CD metric for all experiments.

YCB Video Dataset Most of the datasets commonly used in the reconstruction
task are object-centered and non-cluttered, so we choose this dataset to
validate our performance in cluttered, non-object-centered scenes. The YCB
Video dataset [47] is widely used in object pose estimation tasks. It contains
92 image sequences, in which objects sampled from 21 different categories are
randomly placed or stacked on different tables. We use 5 objects from this
dataset, and choose 1 sequence for each object. We downsample the original
image sequences with a ratio of 20 to get training images, and randomly sample
from the rest to get testing images. Tab. VI shows the specific scenes,
objects, sequences, and number of images we use for training and testing. We
use the ground truth mask it provides for segmentation results evaluation.
Though the dataset provides poses of the objects with respect to cameras in
each view, we observe pose errors that lead to multi-view inconsistency.
Hence, we adopt COLMAP [48] to refine the poses and obtain the sparse
pointcloud for sphere annotation following [42]. After pose refinement, the
coordinates we use differ from the coordinate the ground truth mesh use. To
evaluate the reconstruction accuracy with the provided ground truth mesh, we
adopt a trajectory registration algorithm [52] to align our reconstructed
coordinate with the ground truth coordinates. The aligned transformation is
shared among all methods for fair comparisons.

seg. noise | 0 |  5$\%$ |  10$\%$ | 0 | 0
---|---|---|---|---|---
cls. noise | 0 | 0 | 0 |  5$\%$ |  10$\%$
mIoU | 0.947 | 0.946 | 0.942 | 0.951 | 0.954
TABLE VII: Additional ablation study on feature noises. Dataset | Scene ID | DINO-Coseg | SAM | SNeRF+D | DFF | NeuS | NeuS+D | NeuS+S | Ours
---|---|---|---|---|---|---|---|---|---
BlendedMVS | 1 | 0.595 | 0.614 | 0.643 | / | 0.772 | 0.643 | 0.875 | 0.784
| 2 | 0.830 | 0.570 | 0.720 | / | 0.523 | 0.720 | 0.869 | 0.879
| 3 | 0.843 | 0.437 | 0.879 | / | 0.475 | 0.879 | 0.683 | 0.964
| Mean | 0.765 | 0.540 | 0.747 | / | 0.590 | 0.747 | 0.809 | 0.876
Tanks Temples | 1 | 0.890 | 0.603 | 0.916 | / | 0.558 | 0.838 | 0.603 | 0.946
OnePose | 1 | 0.911 | 0.612 | 0.912 | 0.869 | 0.354 | 0.783 | 0.955 | 0.979
| 2 | 0.724 | 0.796 | 0.665 | 0.823 | 0.293 | 0.801 | 0.945 | 0.958
| 3 | 0.886 | 0.888 | 0.900 | 0.894 | 0.543 | 0.838 | 0.896 | 0.947
| Mean | 0.840 | 0.765 | 0.826 | 0.862 | 0.397 | 0.807 | 0.932 | 0.961
YCB Video | 1 | 0.605 | 0.743 | 0.662 | 0.707 | 0.353 | 0.591 | 0.952 | 0.955
| 2 | 0.702 | 0.843 | 0.746 | 0.938 | 0.551 | 0.451 | 0.489 | 0.926
| 3 | 0.741 | 0.821 | 0.749 | 0.774 | 0.500 | 0.567 | 0.911 | 0.920
| 4 | 0.525 | 0.769 | 0.533 | 0.682 | 0.576 | 0.596 | 0.927 | 0.920
| 5 | 0.763 | 0.599 | 0.806 | 0.895 | 0.408 | 0.858 | 0.935 | 0.930
| Mean | 0.667 | 0.755 | 0.699 | 0.799 | 0.478 | 0.613 | 0.843 | 0.938
TABLE VIII: Quantitative results of segmentation performance. We report mIoU
to evaluate the decomposition performance.

###  -D Additional Quantitative Results

Additional ablation study on feature noises. To examine the sensitivity of our
approach to these features, we conduct ablation studies on one objects from
the OnePose datasets. We add random noises to the segments and clusters to
explore their influence on decomposing results. As shown in Table. VII, when
the features are contaminated by random noise, our segmentation performance
remains strong, highlighting the robustness of our approach.

Additional quantitative results on comparison to the state-of-the-arts. In
Tab. VIII and Tab. II we report the full results of comparison to state-of-
the-art in foreground segmentation and reconstruction.

###  -E Additional Qualitative Results

Additional qualitative results on foreground segmentation. We show additional
qualitative results on foreground segmentation in Fig. 8 Fig. 9, and Fig. 10.

Figure 8: Additional qualitative results of novel view synthesis (top row) and
foreground segmentation (bottom row) on the Tanks and Temples dataset.

Additional qualitative results on foreground reconstruction. We show
additional qualitative results on foreground reconstruction in Fig. 11 and
Fig. 12.

Figure 9: Additional qualitative segmentation results on the OnePose dataset.
Figure 10: Additional qualitative segmentation results on the YCB Video
dataset. Figure 11: Additional qualitative segmentation results on the YCB
Video dataset. Figure 12: Additional qualitative segmentation results on the
BlendedMVS dataset.

###  -F Limitations

Currently, our method still holds two limitations. First, when part of the
background is encased by the target object and keeps consistent across views,
we are unable to separate it correctly, e.g. the red table circled by the
handle of the captain cup object in OnePose dataset. Second, the training
process still takes hours, and we plan to accelerate it with advanced network
architectures or training strategies in the future.