# DITTO: Dual and Integrated Latent Topologies for Implicit 3D Reconstruction

Jaehyeok Shim and Kyungdon Joo
Artificial Intelligence Graduate School, UNIST
<EMAIL_ADDRESS>
https://vision3d-lab.github.io/ditto Corresponding author.

###### Abstract

We propose a novel concept of dual and integrated latent topologies (DITTO in
short) for implicit 3D reconstruction from noisy and sparse point clouds. Most
existing methods predominantly focus on single latent type, such as point or
grid latents. In contrast, the proposed DITTO leverages both point and grid
latents (_i.e_., dual latent) to enhance their strengths, the stability of
grid latents and the detail-rich capability of point latents. Concretely,
DITTO consists of dual latent encoder and integrated implicit decoder. In the
dual latent encoder, a dual latent layer, which is the key module block
composing the encoder, refines both latents in parallel, maintaining their
distinct shapes and enabling recursive interaction. Notably, a newly proposed
dynamic sparse point transformer within the dual latent layer effectively
refines point latents. Then, the integrated implicit decoder systematically
combines these refined latents, achieving high-fidelity 3D reconstruction and
surpassing previous state-of-the-art methods on object- and scene-level
datasets, especially in thin and detailed structures. ††*Corresponding author.

(a) Input points (10K)

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 1:  Scene-level 3D reconstruction comparison on the Synthetic Rooms
dataset [30]. DITTO maximizes the benefits of both grid and point latents,
thereby improving 3D surface reconstruction performance. We particularly focus
on refining features based on point latents along with grid latents and
integrating them (namely, dual and integrated latent topologies). This
advancement enhances the ability to restore complex structures precisely, such
as thin and intricate geometries, which posed challenges for previous methods.

Figure 2:  Overview of DITTO. DITTO architecture consists of the proposed dual
latent encoder and integrated implicit decoder (IID) modules. In the encoder,
DITTO receives a point cloud $\mathcal{P}$ and generates point and grid
latents $\mathcal{C}$ and $\mathbf{T}$, respectively, using shallow FKAConv
layers [2]. These latents are refined in a U-shaped network composed with the
proposed DLL to produce refined point and grid latents, respectively
$\tilde{\mathcal{C}}$ and $\tilde{\mathbf{T}}$. Our IID estimates the
occupancy of given arbitrary query locations. The mesh can be obtained by
applying the Marching Cubes algorithm [23] to the occupancies estimated as a
form of regular grid.

##  1 Introduction

Implicit 3D reconstruction aims to determine surface boundaries by estimating
implicit values, such as occupancy and signed distance fields, based on given
query coordinates [25]. In particular, implicit 3D reconstruction has evolved
using geometric primitives like vectors [25, 6, 33, 28], grids [30, 38, 20,
31], and point clouds [1, 44] as intermediaries, namely, _latent
representations_. Prior studies have focused on selecting appropriate latent
representations for 3D reconstruction. Specifically, early methods [25, 33,
28, 6] use vectors as their latent representation because of their simplicity.
However, they fall short in handling large-scale scenes due to the absence of
a geometric prior (_i.e_., positional information). To alleviate this issue,
subsequent methods based on grid latent have emerged [7, 30, 20, 38]. Grid
latents have similar shapes with occupancy cube, the target domain of 3D
reconstruction. Thus, they offer high-fidelity reconstructions at the scene-
level but often lack detail because of resolution constraints. On the other
hand, point latent-based approaches [1, 44] enable detailed reconstruction
because they preserve the details of the input points without information loss
(_e.g_., quantization). However, they can produce unstable results due to
ambiguities, such as holes in thin structures, because they can be sensitive
to the noise inherent in the input point coordinates.

As an attempt to combine the strengths of each latent, Wang  _et al_. [45]
introduces a new alternating latent topology concept, so-called ALTO.
Concretely, ALTO simultaneously utilizes two latent representations by
alternatively projecting one latent into another. ALTO then decodes the
combined features in the form of a grid for 3D reconstruction. Such an
intuitive and alternative approach improves 3D reconstruction performance and
is meaningful as a first attempt. However, ALTO may overlook the advantages of
abundant features extractable from point latents and makes the implicit
decoder rely solely on a grid latent-based decoder for convenience.

In this work, we propose a novel concept of dual and integrated latent
topologies (DITTO) for implicit 3D reconstruction. The proposed DITTO aims to
systematically integrate the strengths of each latent while maintaining their
spatial structure of point and grid latents (_i.e_., dual latent).
Specifically, we seek to offset the inherent ambiguity of point latents
through the stability of grid latents and, conversely, complement the
resolution constraints of grid latents through the detailed representation by
point latents.

The proposed DITTO employs an encoder-decoder architecture for dual latent
(see Fig. 2). From a given point cloud, our encoder, called _dual latent
encoder_ , constructs point and grid latents and refines this dual latent
while preserving their original shapes. In particular, we propose a new
dynamic sparse point transformer (DSPT) for point features, which leverages
large receptive fields, enabling effective learning of point-based spatial
patterns. Based on DSPT, we design a dual latent layer (DLL) that iteratively
and separately updates dual latent with the correlation between two latents.
This DLL module allows us to implicitly learn challenging patterns, such as
thin objects, that cannot be handled by grid latent alone. Then, our decoder,
called _integrated implicit decoder (IID)_ , integrates enhanced dual latent
to estimate the implicit value. Unlike previous methods that utilize only a
subset of latents, our decoder considers dual latent; especially, we unify
grid-based and point-based implicit decoders together by introducing the
concept of integrated latent. IID helps to restore details by adjusting the
relationship between neighbor points and query, adapting to the surface
proximity. Finally, the proposed DITTO improves 3D surface reconstruction
performance, outperforming previous approaches and establishing a new state-
of-the-art (see Fig. 1).

The main contributions of DITTO are as follows:

  * •

DITTO is a new implicit 3D reconstruction method focusing on advanced feature
extraction and fusion of grid and point latents, enhancing 3D understanding
capabilities.

  * •

We design a new dual latent layer module that refines dual latent while
preserving their individual strengths. Particularly, we present a dynamic
sparse point transformer (DSPT) to emphasize point feature refinement.

  * •

We present a novel integrated implicit decoder that uniquely integrates two
latents, providing clear surface boundaries, and robust to thin and intricate
structures.

##  2 Related Work

3D reconstruction can be explicitly represented using a variety of geometric
primitives, such as point, voxel, and mesh, or it can be inherently
represented by leveraging such geometric primitives as latent representations
[25, 28, 6, 30, 20, 41, 38, 1, 45, 44]. We refer to the former as explicit 3D
reconstruction and the latter as implicit 3D reconstruction. The readers refer
to [45] for explicit representations. In this section, we discuss the
strengths and weaknesses of each latent for implicit 3D reconstructions.

Vector Latent Topologies. Early approaches, such as [25, 28, 6], employ an
encoder-decoder architecture in a similar manner. They encode a 3D shape into
vector latents by the encoder and then reconstruct the 3D shape by the
decoder. The decoder estimates the implicit values of a given query point at
arbitrary locations. However, since vector latents lack geometric priors, they
show decreased detailed reconstruction performance (_i.e_., vector latent-
based methods are vulnerable in large scenes with complex geometry).

Grid Latent Topologies. As an alternative to vector latents, grid latents that
encode geometric priors have been proposed [30, 41, 38, 31, 45]. Grid latent-
based methods quantize the input point cloud into a grid latent during
encoding. Then, they extract latent features at the query points using linear
interpolation of the adjacent grids during decoding.

According to geometric primitives, we can divide grid latent-based methods
into voxel-based and triplane-based methods. Voxel latents can densely store
latent features in the form of 3D grids [37, 46, 27, 13, 34]. ConvONet [30]
expands the previous ONet [25] by utilizing voxel latents instead of vector
latents. ConvONet and ALTO [45] take voxel latents as one of the base
representations, which reveal their effectiveness, especially for scene-level
reconstructions. However, voxels require cubic computation and inherently have
limited resolutions. On the other hand, triplanes use 2D planes, which have
less resolution constraints, allowing higher resolution than voxels. This
advantage leads to more effective restoration than voxels, particularly for
object-level tasks [5, 3, 14, 8, 35, 15, 43, 12], which has slightly lower
geometric complexity than scenes. For this reason, ConvONet and ALTO use
triplanes as the primary latent topologies in object-level reconstruction.

Even though grid latent-based approaches show effective reconstruction, but
still require point feature quantization. This process can result in a loss of
fine details of the 3D surface, which is a fundamental limitation of grid
latents.

Point Latent Topologies. There are a few methods [1, 47, 48, 44] that encode
features into point latents, typically offering benefits for the preservation
of spatial information. Namely, there is no need for quantization, preventing
the loss of details. POCO [1] encodes latent features into point latents and
further enhances each feature by leveraging the point latents of neighbor
points through point convolution [2] and attention mechanism [40]. ARO-Net
[44] improves point features by introducing methods like anchor points and
radial observations, enhancing the performance of the point-based implicit
decoder. However, these point latent-based methods may suffer instability from
preserving spatial information of the point cloud, including even noise
points. Concretely, point latents from noise points can affect neighbor
points. In addition, the query feature extraction process may introduce
ambiguity since different queries can share the same neighbor points. We
address these issues of point latents by leveraging the stability of grid
latents.

(a) Encoder architectures

(b) Decoder architectures

Figure 3:  Conceptual comparison of DITTO. We compare the concept of implicit
3D reconstruction methods in terms of latent representations: (a) encoders and
(b) decoders. In (b), the image of green chairs represents grid features.

Blended Latent Topologies. ALTO [45] introduces a new method that leverages
multiple types of feature representations, utilizing both grid and point
latents. This attempt is the first approach to combine the strengths of each
latent. Specifically, ALTO iteratively projects its features from grid to
point cloud and vice versa. By doing this, ALTO aims to preserve the details
inherent in the point cloud and enable feature sharing between planes, leading
to improved detailed surface reconstruction. However, ALTO primarily relies on
grid latents, with limited feature extraction from point latents, hindering
its capacity to fully exploit the potential of point latents. In particular,
its decoder exclusively employs grid latents, making it directly susceptible
to the resolution constraints inherent in grid latents.

To address these issues, we propose DITTO designed with advanced 3D geometry
understanding capabilities. DITTO comprises an enhanced point encoder, based
on FKAConv [2], and advanced module for extracting features from both point
and grid latents. Furthermore, we present an implicit decoder that leverages
the fusion of both grid and point latents for improved performance.

##  3 Dual and Integrated Latent Topologies

In this section, we propose a new topological concept, _dual and integrated
latent topologies_ (DITTO), for implicit 3D reconstruction from a given noisy
and sparse point cloud. DITTO employs dual latent (_i.e_., point and grid
latents) to leverage both the structural stability of grid latents and the
preciseness of point latents. DITTO, composed of encoder-decoder architecture
for dual latent, refines and integrates these two latents, overcoming
individual limitations and improving overall efficacy. This strategy leads to
high-fidelity surface reconstruction, even for thin, intricate structures.

###  3.1 Overview

Given a noisy and sparse point cloud
$\mathcal{P}=\\{\mathbf{p}_{i}\in\mathbb{R}^{3}\\}^{N}_{i=1}$ as input, the
goal of DITTO is to accurately reconstruct 3D surfaces in a form of occupancy
$\mathcal{O}=\\{o_{j}\in\\{0,1\\}\\}^{M}_{j=1}$ for query coordinates of
arbitrary location
$\mathcal{Q}=\\{\mathbf{q}_{j}\in\mathbb{R}^{3}\\}^{M}_{j=1}$, where $N$ and
$M$ are the number of input points and queries.

The proposed DITTO comprises two main parts: dual latent encoder and
integrated implicit decoder (see Fig. 2). In the dual latent encoder, we first
extract the point latents
$\mathcal{C}=\\{\mathbf{c}_{i}\in\mathbb{R}^{d}\\}^{N}_{i=1}$ for
$\mathcal{P}$ based on FKAConv layers [2], where $d$ is the dimension of the
point latents. We then project $\mathcal{C}$ to grid latents. Following the
convention [30, 45], we use triplanes $\mathbf{T}\in\mathbb{R}^{3\times
R\times R\times d}$ or voxels $\mathbf{V}\in\mathbb{R}^{R\times R\times
R\times d}$ as grid latents, where $R$ is the resolution of grids. In this
section, we explain the details of DITTO based on triplanes as grid latents,
but they can seamlessly be replaced with voxels. After extracting the initial
dual latent ($\mathcal{C}$ and $\mathbf{T}$), we refine them using a UNet
architecture [32], where each layer consists of the proposed dual latent layer
(DLL). This UNet estimates refined grid $\tilde{\mathbf{T}}$ and point
$\tilde{\mathcal{C}}$ latents. The detailed description of DLL is in Sec. 3.2.
Then, integrated implicit decoder (IID) estimates occupancy probability
$\tilde{\mathcal{O}}=\\{\tilde{o}_{j}\in\mathbb{R}\\}_{j=1}^{M}$ of given
query coordinates $\mathcal{Q}$ by integrating latents $\tilde{\mathbf{T}}$
and $\tilde{\mathcal{C}}$. IID effectively manipulates the distinct
characteristics of grid and point latents, facilitating the reconstruction of
detailed and thin structures. The detailed description of IID is in Sec. 3.3.
More detailed network architectures are provided in supplementary materials.

###  3.2 Dual Latent Layer

Figure 4:  Overview of our proposed DLL. The input consists of $\mathcal{C}$
and $\mathbf{T}$, while the output comprises $\mathcal{C}^{\prime}$ and
$\mathbf{T}^{\prime}$, representing the grid and point latents of the current
layer, respectively. The $\bar{\mathbf{T}}$ and $\bar{\mathbf{T}}^{\prime}$
represent the input and output grid latents, respectively, used for
establishing a dense skip-connection between the layers.

We present a dual latent layer (DLL) that adeptly combines the advantages of
both grid and point latents. Specifically, DLL individually enhances two
latents and infuses correlation between them to optimize their combined
performance while preserving their own strengths, instead of combining them
into a single latent (see Fig. 3(a)). In particular, we newly present a
dynamic sparse point transformer (DSPT) as a point feature extractor. DSPT
boosts point feature refinement by directly employing a transformer to point-
based dynamic windows.

Dynamic Sparse Point Transformer. The concept of DSPT is to apply a
transformer directly to points, stimulating local-global interactions among
point latents. Inspired by the concept of windowed attention [22], we create
point-based non-overlapping windows and apply self-attention to each window.
However, unlike images or voxels having regular grids, points have free form
and are unordered. To alleviate this issue, we adopt a sorting-based windowing
scheme, similar to DSVT [42] and CSwinTransformer [11]. Concretely, we sort
points for a certain axis and divide them into multiple windows so that each
window has an equal number of points. Subsequently, self-attention is applied
within each window. In a DSPT block, we repeat this procedure for each of the
x-, y- and z-axes (see 2D illustration of DSPT in Fig. 5).

Here, we describe the details of DSPT. For given point features $\mathcal{C}$
and their coordinates $\mathcal{P}$, we first sort $\mathcal{C}$ based on
$\mathcal{P}$ along a specific axis. We then divide the sorted point features
into windows $\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$ so that the number
of points belonging to each window is equal:

$\displaystyle\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}=\textit{split}(\textit{sort}_{\text{x}}(\mathcal{C},\mathcal{P}),L),$
(1) $\displaystyle\text{where\ \
}\mathcal{C}^{\text{wnd}}_{l}=\\{\mathbf{c}_{i_{\text{sorted}}}\in\mathbb{R}^{d}\\}_{i_{\text{sorted}}=1}^{{N/L}},$

$L$ is the number of windows, $\textit{sort}_{\text{x}}(\cdot,\cdot)$ denotes
a sort function, which sorts $\mathcal{C}$ _w.r.t_. their coordinates
$\mathcal{P}$ of x-axis, $\textit{split}(\cdot,\cdot)$ denotes a splitting
function, which divides the given $\mathcal{C}$ into $L$ windows, and
$i_{\text{sorted}}$ indicates the sorted index. For convenience, we use the
x-axis in Eq. (1), but we can also apply the y- and z-axes. Before applying
self-attention, a rotary positional embedding (RoPE) [36], modified one for
point clouds [19], is applied to the query and key of the inputs of self-
attention. Subsequently, a shared MLP is applied to each $\mathbf{c}$, and the
points are re-sorted in reverse to restore the original point order.

Figure 5:  Conceptual illustration of DSPT. We visualize DSPT in the 2D domain
for better understanding, but DSPT works in the 3D domain by sequentially
processing the x-, y-, and z-axes.

Overall Architecture of DLL for Dual Latent. The proposed DLL takes dual
latent, $\mathcal{C}$ and $\mathbf{T}$, from the previous DLL as input and
systematically refines each latent feature while maintaining their shape and
strengths (see Fig. 4). Within DLL, we first enhance grid latents $\mathbf{T}$
using a simple CNN-based architecture. Note that when the triplane is used as
grid latents, 3D-aware-conv [43] is additionally applied to induce feature
exchange among three planes. We call this enhanced grid latents as
intermediate grid features $\bar{\mathbf{T}}^{\prime}$, which is used to
refine $\mathcal{C}$ and form a dense skip-connection between consecutive DLL
modules, similar to [45].

In the point latent perspective, we combine $\mathcal{C}$ with the point
features projected from $\bar{\mathbf{T}}$, inducing information of stable
grid latents to point latents. We then refine these point latents using the
proposed DSPT. We denote these refined point latents as
$\mathcal{C}^{\prime}$. In the grid latent perspective, we project
$\mathcal{C}^{\prime}$ into grid domain and merge with intermediate latent
$\bar{\mathbf{T}}^{\prime}$, forming enhanced grid latents
$\mathbf{T}^{\prime}$, which creates a synergy of dual latent.

###  3.3 Integrated Implicit Decoder

We present an integrated implicit decoder (IID) that estimates implicit values
(_i.e_., occupancy) for a given query location using refined dual latent
$\tilde{\mathbf{T}}$ and $\tilde{\mathcal{C}}$. In particular, IID integrates
the grid and point latents while reinforcing the strengths and compensating
for shortcomings associated with each latent (see Fig. 3(b)). To this end, we
first analyze the pros and cons of each latent in terms of decoding
perspective and then introduce IID in detail.

Point Latents _vs._ Grid Latents in Decoding. Point-based implicit decoding
approaches like POCO [1] utilize K-nearest neighbors (KNN) to define neighbor
points for a given query point and use features of neighbor points to
calculate query feature. This point-based decoder is advantageous for detail
restoration. However, such point-based decoders can be fragile when handling
thin structures due to inherent ambiguity because different query points may
share the same neighbors, resulting in instability.

Grid-based decoding methods (_e.g_., ConvONet [30] and ALTO [45]) estimate
occupancy by interpolating features from adjacent grids for a given query to
determine the query feature. Additionally, ALTO employs an attention mechanism
to alleviate the resolution constraints inherent to grid latents, thereby
enhancing performance. Specifically, ALTO compares query feature and adjacent
grid features through subtraction-based cross-attention [49]. This comparison
allows ALTO to use not only the query feature but also the varying patterns of
adjacent grid features, mitigating the resolution constraints of grid latents.
However, despite this improvement, detailed reconstruction remains limited due
to the inherent limitations of relying solely on grid latents.

Figure 6:  Illustration of IID. IID makes query feature
$\mathbf{z}^{\mathbf{q}}_{0}$ and neighbor integrated features
$\\{\mathbf{z}_{1}^{\mathbf{q}},\dots,\mathbf{z}_{k}^{\mathbf{q}}\\}$. Then,
IID iteratively applies self-attention to a sequence of
$\\{\mathbf{z}_{0}^{\mathbf{q}},\dots,\mathbf{z}_{k}^{\mathbf{q}}\\}$ to
update query feature $\mathbf{z}_{0}^{\mathbf{q}}$. Finally, we estimate the
query feature $\mathbf{\hat{z}}_{0}^{\mathbf{q}}$ and apply linear layer to
estimate occupancy $\tilde{o}$.

Integrated Decoder for Dual Latent. The proposed IID selectively integrates
the advantages of each latent decoding method (_i.e_., a hybrid approach
between point and grid latent decoding). We basically adopt KNN-based
neighbors of point latents to determine neighbor point features since it can
recover the detailed reconstruction free from the resolution limit. Instead,
we handle the inherent limitation of point features by using grid-based
decoding. That is, we combine point features at neighbor locations with
adjacent grid features via interpolation. This integration enables IID to
effectively consider both latents. We call these combined features as neighbor
integrated features. Subsequently, we estimate implicit value through self-
attention between neighbor integrated features and query feature obtained from
grid latents. Note that since neighbor points are located around the surface,
the region of interest (_i.e_., size of the neighbor integrated features) is
adaptively defined _w.r.t_. query-neighbor distances. This adaptive mechanism
facilitates the reconstruction of clear surface boundaries of detailed and
intricate structures.

Table 1:  Object-level quantitative comparison on ShapeNet. From left to
right, the input point clouds become sparse (variations in density); they have
3K, 1K, and 300 points, respectively. The best scores are red in bold, and the
secondary ones are blue with underline.

Method | normal (3K points & noise level 0.005) | sparse (1K points & noise level 0.005) | sparse (300 points & noise level 0.005)
---|---|---|---
IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$ | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$ | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
ONet [25] | 0.761 | 0.87 | 0.891 | 0.785 | 0.772 | 0.81 | 0.894 | 0.801 | 0.778 | 0.80 | 0.895 | 0.806
ConvONet [30] | 0.884 | 0.44 | 0.938 | 0.942 | 0.859 | 0.50 | 0.929 | 0.918 | 0.821 | 0.59 | 0.907 | 0.883
POCO [1] | 0.926 | 0.30 | 0.950 | 0.984 | 0.884 | 0.40 | 0.928 | 0.950 | 0.808 | 0.61 | 0.892 | 0.869
ALTO [45] | 0.930 | 0.30 | 0.952 | 0.980 | 0.905 | 0.35 | 0.940 | 0.964 | 0.863 | 0.47 | 0.922 | 0.924
DITTO (ours) | 0.949 | 0.27 | 0.957 | 0.988 | 0.926 | 0.32 | 0.949 | 0.975 | 0.882 | 0.43 | 0.931 | 0.940

(a) GT Mesh

(b) Input points

(c) ConvONet [30]

(d) POCO [1]

(e) ALTO [45]

(f) DITTO (ours)

Figure 7:  Object-level 3D reconstruction comparison on ShapeNet [4] with 3K
input points. DITTO distinctively excels in reconstructing thin structures,
evidenced by the intricate details of lamps and benches. Uniquely, DITTO is
the only method that accurately captures the side mirror and the fine details
of car wheels.

The detailed IID is visualized in Fig. 6. First, we estimate query feature
$\mathbf{z}^{\mathbf{q}}_{0}$ by interpolating the grid features as in [30,
45] and applying linear layer so that the query feature has $2d$ dimensions:

$\mathbf{z}^{\mathbf{q}}_{0}=\textit{interpolate}(\tilde{\mathbf{T}},\mathbf{q})\mathbf{W},\quad\mathbf{z}^{\mathbf{q}}_{0}\in\mathbb{R}^{2d},\vspace{-1mm}$
(2)

where $\textit{interpolate}(\cdot,\cdot)$ computes a grid feature of a given
location using linear interpolation for adjacent grid latents, and
$\mathbf{W}\in\mathbb{R}^{d\times 2d}$ is a weight matrix of linear layer.
Subsequently, we find neighbor points $\mathcal{P}^{\mathbf{q}}$ and neighbor
point features $\tilde{\mathcal{C}}^{\mathbf{q}}$ of the query point
$\mathbf{q}$ by using KNN:

$(\mathcal{P}^{\mathbf{q}},\tilde{\mathcal{C}}^{\mathbf{q}})=\textit{KNN}(\mathbf{q},\mathcal{P},\tilde{\mathcal{C}},K),\vspace{-1mm}$
(3)

where $\textit{KNN}(\cdot,\cdot,\cdot,\cdot)$ returns given $K$ number of
neighbor point coordinates
$\mathcal{P}^{\mathbf{q}}=\\{\mathbf{p}^{\mathbf{q}}_{k}\in\mathbb{R}^{3}\\}_{k=1}^{K}$
and their features
$\tilde{\mathcal{C}}^{\mathbf{q}}=\\{\tilde{\mathbf{c}}^{\mathbf{q}}_{k}\in\mathbb{R}^{d}\\}_{k=1}^{K}$.
After that, we compute neighbor grid features $\mathbf{T}^{\mathbf{q}}$ by
interpolating grid latents for every $\mathbf{p}^{\mathbf{q}}_{k}$:

$\tilde{\mathbf{T}}^{\mathbf{q}}=\\{\textit{interpolate}(\tilde{\mathbf{T}},\mathbf{p}^{\mathbf{q}}_{k})\\}^{K}_{k=1}.\vspace{-1mm}$
(4)

Then, we construct the neighbor integrated features $\mathcal{Z}^{\mathbf{q}}$
by concatenating $\tilde{\mathcal{C}}^{\mathbf{q}}$ and
$\tilde{\mathbf{T}}^{\mathbf{q}}$ in channel direction:

$\mathcal{Z}^{\mathbf{q}}=\textit{concat}(\tilde{\mathcal{C}}^{\mathbf{q}},\tilde{\mathbf{T}}^{\mathbf{q}})=\\{\mathbf{z}^{\mathbf{q}}_{k}\in\mathbb{R}^{2d}\\}^{K}_{k=1},\vspace{-1mm}$
(5)

where $\textit{concat}(\cdot,\cdot)$ is a concatenation function. Then, we
refine $\mathbf{z}^{\mathbf{q}}_{0}$ by applying self-attention multiple times
on a sequence
$\\{\mathbf{z}^{\mathbf{q}}_{0},\dots,\mathbf{z}^{\mathbf{q}}_{K}\\}$
including both $\mathbf{z}^{\mathbf{q}}_{0}$ and $\mathcal{Z}^{\mathbf{q}}$.
We denote $\hat{\mathbf{z}}^{\mathbf{q}}_{0}$ as the refined query feature.
While applying self-attention, we use point-based RoPE [36] as mentioned in
Sec. 3.2. Note that during the self-attention, we update only
$\mathbf{z}^{\mathbf{q}}_{0}$ and the other elements of the sequence
$\\{\mathbf{z}^{\mathbf{q}}_{1},\dots,\mathbf{z}^{\mathbf{q}}_{K}\\}$ remain
unchanged. Finally, we estimate occupancy $\tilde{o}$ by applying linear layer
to $\hat{\mathbf{z}}^{\mathbf{q}}_{0}$:

$\tilde{o}=\hat{\mathbf{z}}^{\mathbf{q}}_{0}\mathbf{W}_{\text{out}},\vspace{-1mm}$
(6)

where $\mathbf{W}_{\text{out}}\in\mathbb{R}^{2d}$ is a weight matrix of linear
layer.

###  3.4 Training Objectives

We use binary cross entropy objective between $\tilde{\mathcal{O}}$ and
$\mathcal{O}$:

$\vspace{-2mm}\mathcal{L}(\tilde{\mathcal{O}},\mathcal{O}){=}{-}\frac{1}{M}\sum^{M}_{j=1}\left[o_{j}\log(\tilde{o}_{j}){+}(1{-}o_{j})\log(1{-}\tilde{o}_{j})\right].\vspace{-3mm}$
(7)

(a) Input Points (10K)

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 8:  Qualitative comparison of scene-level 3D surface reconstruction on
the Synthetic Rooms dataset [30].

##  4 Experiments

We evaluate DITTO against SoTA methods. Details of implementation are provided
in Sec. 4.1. Qualitative and quantitative comparisons for object-level and
scene-level are in Sec. 4.2 and Sec. 4.3, respectively. In addition, we
validate the generality in Sec. 4.4. Additional experiment results are
available in the supplementary materials.

###  4.1 Baselines, Datasets, Metrics

Implementation Details. We implement DITTO in PyTorch [29], utilizing xFormers
[18] and mixed-precision [26]. We train DITTO with Adam optimizer [17] and
cosine annealing learning rate scheduler [24]. For a fair comparison, we set
the resolution of triplanes as $R{=}64$ and $R{=}128$ for object- and scene-
level tasks, respectively. For voxels, we set $R{=}64$ in scene-level tasks.
In DSPT, we use $L{=}25$ for $\\{3\text{K},1\text{K},0.3\text{K}\\}$ input
points and $L{=}20$ for $10$K input points. More detailed hyperparameters are
described in the supplementary materials.

Comparison Methods. To assess the 3D reconstruction performance of DITTO, we
compare it with various baseline methods. These methods include a non-
learning-based method [16], as well as implicit methods that utilize diverse
latent topologies, such as vector [25], grid [30, 20], point [1] and blended
[45] latents. Our evaluation procedure primarily follows the previous SoTA
method, ALTO, including several additional experiments.

Datasets. For the evaluation of object-level surface reconstruction, we use
ShapeNet [4], which contains 13 categories of object watertight meshes. For
assessment of scene-level surface reconstruction, we use the Synthetic Rooms
dataset [30], which has 5K synthetically created rooms utilizing objects from
ShapeNet. We follow the same train/val/test splits in both datasets with
convention [30, 1, 45] for fair comparison. The points are randomly sampled,
and Gaussian noise is applied. In addition, we adopt ScanNet-v2 [10], which
contains 1,513 scene scans, for generality evaluation.

Table 2:  Scene-level quantitative comparison on the Synthetic Rooms dataset
[30]. We train each method for 10K input points with 0.005 noise level. The
triplane comparison involves assessing the impact of triplane in methods using
grid latents.

Method | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
---|---|---|---|---
ONet [25] | 0.475 | 2.03 | 0.783 | 0.541
SPSR [16] | - | 2.23 | 0.866 | 0.810
SPSR trimmed [16] | - | 0.69 | 0.890 | 0.892
ConvONet [30] | 0.849 | 0.42 | 0.915 | 0.964
DP-ConvONet [20] | 0.800 | 0.42 | 0.912 | 0.960
POCO [1] | 0.884 | 0.36 | 0.919 | 0.980
ALTO [45] | 0.914 | 0.35 | 0.921 | 0.981
Ours | 0.928 | 0.34 | 0.930 | 0.984
Triplane comparison
ConvONet [30] | 0.805 | 0.44 | 0.903 | 0.948
ALTO [45] | 0.895 | 0.37 | 0.910 | 0.974
Ours | 0.931 | 0.33 | 0.931 | 0.984

Evaluation Metrics. We measure the reconstruction performance using standard
quantitative metrics, such as IoU, Chamfer-$L_{1}$ distance, normal
consistency (NC), and F-score [39], following baseline methods. For
Chamfer-$L_{1}$ distance, we multiply 100 for convenience and use the
threshold value as 1% for F-score.

###  4.2 Object-Level 3D Surface Reconstruction

Quantitative Evaluation. In Table 1, we quantitatively compare the object-
level surface reconstruction performance on ShapeNet [4]. DITTO exhibits
superior performance across all metrics. Notably, DITTO demonstrates a
substantial lead in the IoU metric; a four times larger gap compared to the
previous SoTA [45]. In addition, evaluation on various input point densities
implies that DITTO shows robust scores regardless of the number of points,
even though we use point latents. This result demonstrates the effectiveness
and robustness of DITTO.

Qualitative Evaluation. Figure 7 shows the qualitative results. DITTO shows
clear surface boundaries, especially for thin and intricate structures. Note
that DITTO is the only method that successfully reconstructs the complex
structures, such as the intricate pedestal of the lamp (first row), back of
the bench (second row), rearview mirror and pattern on wheels of the car
(third row). In particular, reconstructing shapes with repeated thin
structures is a challenging problem. Grid latents struggle due to resolution
constraints, while point latents often fail to create clear boundaries due to
their inherent ambiguity. In this challenging case, DITTO successfully creates
a clear surface boundary.

Table 3:  Scene-level quantitative comparison on the Synthetic Rooms dataset
[30] with sparse and noisy input data. We train each method for sparse (3K
points with 0.005 noise level) and noisy (10K points with 0.025 noise level)
input point clouds.

Method | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
---|---|---|---|---
Sparse input points (3K input points)
ConvONet [30] | 0.818 | 0.46 | 0.906 | 0.943
POCO [1] | 0.801 | 0.57 | 0.904 | 0.812
ALTO [45] | 0.882 | 0.39 | 0.911 | 0.969
DITTO (ours) | 0.900 | 0.37 | 0.919 | 0.975
Noisy input points (0.025 noise level)
ConvONet [30] | 0.777 | 0.57 | 0.872 | 0.885
POCO [1] | 0.701 | 0.64 | 0.848 | 0.857
ALTO [45] | 0.804 | 0.55 | 0.877 | 0.898
DITTO (ours) | 0.811 | 0.55 | 0.875 | 0.898

###  4.3 Scene-Level 3D Surface Reconstruction

Quantitative Evaluation. We assess the scene-level reconstruction performance
on the Synthetic Rooms dataset [30]. The quantitative results are in Table 2.
DITTO surpasses previous methods in most of the metrics. Regarding grid latent
representation, DITTO marks a turning point. Most grid-based methods [30, 45]
with triplane representations show decreased performance in complex scene-
level reconstructions. In contrast, DITTO maintains consistent performance,
even with triplane representations. In addition, quantitative results focusing
on sparse and noisy inputs are in Table 3. DITTO shows outstanding performance
robust to sparse and noisy input point clouds. These results contrast the
method solely based on point latents, which is vulnerable to contamination of
the input point clouds.

(a) Input Points (10K)

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 9:  Qualitative comparison of ScanNet-v2 [30].

Qualitative Evaluation. We visualize the qualitative comparisons in Fig. 1 and
Fig. 8. The results of ConvONet [30], a grid-based method, are relatively
stable, with fewer holes in thin structure, but they lack detail (see lamps in
red boxes). On the other hand, POCO [1], a point-based method, shows better
detail but is less stable, often resulting in holes and artifacts (see lamps
and chairs in yellow boxes). ALTO [45], employing both latents, tends to offer
better detail compared to ConvONet and better stability than POCO. However,
ALTO displays less stability than ConvONet and less detail than POCO. Unlike
the previous approaches, DITTO validates both superior stability and detail.
This result demonstrates that DITTO achieves synergy from the integration of
two latents.

###  4.4 Ablation Study

Table 4:  Ablation study on the proposed modules. We train networks on the
Synthetic Rooms dataset by switching modules one by one from the baseline to
the suggested method.

Method | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
---|---|---|---|---
ALTO [45] (triplane; baseline) | 0.895 | 0.35 | 0.921 | 0.981
\+ DLL (DSPT backbone) | 0.921 | 0.35 | 0.925 | 0.981
\+ DLL (FKAConv [2] backbone) | 0.917 | 0.35 | 0.922 | 0.979
\+ DLL (PointTransformer [49] backbone) | 0.911 | 0.36 | 0.917 | 0.976
\+ IID | 0.907 | 0.35 | 0.913 | 0.976
\+ PointEncoder (FKAConv [2]) | 0.912 | 0.36 | 0.918 | 0.978
\+ DLL + IID | 0.925 | 0.34 | 0.927 | 0.983
\+ DLL + PointEncoder | 0.926 | 0.34 | 0.928 | 0.981
\+ IID + PointEncoder | 0.918 | 0.35 | 0.920 | 0.981
DITTO (ours) | 0.931 | 0.33 | 0.931 | 0.984

We evaluate the impact of each proposed module by incrementally incorporating
them into baseline ALTO. The results are presented in Table 4. DLL
demonstrates significant performance improvements regardless of the type of
point backbones. This result reveals that the point feature extraction module
that learns spatial patterns of point latents is more crucial than the simple
MLP of ALTO. Additionally, while the NC and F-score show a slight decrease
with IID alone, other metrics increase. However, combining DLL with IID leads
to substantial performance enhancement in every metric. We deduce this
variation in performance is due to the point latents from ALTO not being
optimized for learning spatial patterns.

###  4.5 Real-World 3D Surface Reconstruction

We conduct an additional experiment to assess generality (see Table 5 and Fig.
9). DITTO demonstrates superior performance than previous methods
qualitatively and quantitatively. DITTO successfully restores the sofas (see
red and yellow boxes), where other methods encounter difficulties. We also
evaluate the performance using triplanes as grid latents (see Triplane
comparison for grid latents). Unlike the scene-level results in Sec. 4.3,
triplanes generally show poorer performance on ScanNet-v2. We attribute this
performance issue to a lack of geometric inductive bias in triplanes. Since
each plane in a triplane misses information in a direction, triplane latents
are required to learn spatial rules that the voxels naturally have. This
limitation poses challenges for triplanes when reconstructing out-of-
distribution data, which have somewhat different geometrical properties.
Despite these challenges, DITTO shows better performance than other grid-based
methods with triplanes.

Table 5:  Quantitative comparison on ScanNet-v2 [10]. We test models pre-
trained with the Synthetic Rooms dataset [30] on ScanNet-v2, which is not used
during training. Since the dataset has no ground truth occupancy, we only
measure Chamfer distance and F-score, following [45].

Method | $N_{\text{train}},N_{\text{test}}=10\text{K},10\text{K}$ | $N_{\text{train}},N_{\text{test}}=10\text{K},3\text{K}$
---|---|---
Chamfer-$L_{1}$ | F-score | Chamfer-$L_{1}$ | F-score
ConvONet [30] | 1.02 | 0.694 | 1.01 | 0.719
POCO [1] | 0.87 | 0.757 | 0.93 | 0.737
ALTO [45] | 0.79 | 0.779 | 0.87 | 0.746
DITTO (ours) | 0.70 | 0.808 | 0.78 | 0.773
_Triplane comparison_
ConvONet [30] | 1.45 | 0.636 | 1.55 | 0.614
ALTO [45] | 1.43 | 0.640 | 1.44 | 0.601
DITTO (ours) | 1.26 | 0.677 | 1.26 | 0.660

##  5 Conclusion and Discussion

We have proposed DITTO, a novel concept of dual and integrated latent
topologies for implicit 3D reconstruction from noisy and sparse point clouds.
Specifically, we have studied the use of grid and point latents together as
dual latent to integrate their own strengths. To this end, we have proposed
the DLL architecture with the DSPT module for enhancing dual latent while
maintaining their original shape at the encoder level. Then, we explored how
to utilize both refined latents in the proposed integrated implicit decoder.
DITTO outperforms previous state-of-the-art implicit 3D reconstruction
methods, especially DITTO facilitates the reconstruction of thin structures
and intricate shape details.

Future Work. In this work, we have investigated the importance of point
latents along with grid latents. While grid latents mitigate the inherent
noise sensitivity of point latents, they do not fully resolve the coordinate
noise issue. As a future direction, we believe that enabling how to learn
point deformation would lead to further improvement.

## References

  * Boulch and Marlet [2022] Alexandre Boulch and Renaud Marlet.  Poco: Point convolution for surface reconstruction.  In _CVPR_ , 2022.
  * Boulch et al. [2020] Alexandre Boulch, Gilles Puy, and Renaud Marlet.  Fkaconv: Feature-kernel alignment for point cloud convolution.  In _ACCV_ , 2020.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _CVPR_ , 2022.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _ECCV_ , 2022.
  * Chen and Zhang [2019] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _CVPR_ , 2019.
  * Chibane et al. [2020] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.  Implicit functions in feature space for 3d shape reconstruction and completion.  In _CVPR_ , 2020.
  * Chou et al. [2023] Gene Chou, Yuval Bahat, and Felix Heide.  Diffusion-sdf: Conditional generative modeling of signed distance functions.  In _ICCV_ , 2023.
  * Çiçek et al. [2016] Özgün Çiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger.  3d u-net: learning dense volumetric segmentation from sparse annotation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19_ , 2016.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _CVPR_ , 2017.
  * Dong et al. [2022] Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo.  Cswin transformer: A general vision transformer backbone with cross-shaped windows.  In _CVPR_ , 2022.
  * Dong et al. [2023] Zijian Dong, Xu Chen, Jinlong Yang, Michael J Black, Otmar Hilliges, and Andreas Geiger.  Ag3d: Learning to generate 3d avatars from 2d image collections.  _arXiv preprint arXiv:2305.02312_ , 2023.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _CVPR_ , 2022.
  * Gao et al. [2022] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  _NeurIPS_ , 2022.
  * Gupta et al. [2023] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oğuz.  3dgen: Triplane latent diffusion for textured mesh generation.  _arXiv preprint arXiv:2303.05371_ , 2023.
  * Kazhdan and Hoppe [2013] Michael Kazhdan and Hugues Hoppe.  Screened poisson surface reconstruction.  _TOG_ , 2013.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Lefaudeux et al. [2022] Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza.  xformers: A modular and hackable transformer modelling library.  https://github.com/facebookresearch/xformers, 2022.
  * Li and Harada [2022] Yang Li and Tatsuya Harada.  Lepard: Learning partial point cloud matching in rigid and deformable scenes.  In _CVPR_ , 2022.
  * Lionar et al. [2021] Stefan Lionar, Daniil Emtsev, Dusan Svilarkovic, and Songyou Peng.  Dynamic plane convolutional occupancy networks.  In _WACV_ , 2021.
  * Liu et al. [2019] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han.  Point-voxel cnn for efficient 3d deep learning.  _NeurIPS_ , 2019.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Lorensen and Cline [1987] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  _SIGGRAPH_ , 1987.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In _ICLR_ , 2017.
  * Mescheder et al. [2019] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _CVPR_ , 2019.
  * Micikevicius et al. [2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu.  Mixed precision training.  In _ICLR_ , 2018.
  * Mittal et al. [2022] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  Autosdf: Shape priors for 3d completion, reconstruction and generation.  In _CVPR_ , 2022.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _CVPR_ , 2019.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _NeurIPS_ , 2019.
  * Peng et al. [2020] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger.  Convolutional occupancy networks.  In _ECCV_ , 2020.
  * Peng et al. [2021] Songyou Peng, Chiyu Jiang, Yiyi Liao, Michael Niemeyer, Marc Pollefeys, and Andreas Geiger.  Shape as points: A differentiable poisson solver.  _NeurIPS_ , 2021.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , 2015.
  * Saito et al. [2019] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In _ICCV_ , 2019.
  * Shim et al. [2023] Jaehyeok Shim, Changwoo Kang, and Kyungdon Joo.  Diffusion-based signed distance fields for 3d shape generation.  In _CVPR_ , 2023.
  * Shue et al. [2023] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.  3d neural field generation using triplane diffusion.  In _CVPR_ , 2023.
  * Su et al. [2021] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  _arXiv preprint arXiv:2104.09864_ , 2021.
  * Sun et al. [2022] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _CVPR_ , 2022.
  * Tang et al. [2021] Jiapeng Tang, Jiabao Lei, Dan Xu, Feiying Ma, Kui Jia, and Lei Zhang.  Sa-convonet: Sign-agnostic optimization of convolutional occupancy networks.  In _ICCV_ , 2021.
  * Tatarchenko et al. [2019] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In _CVPR_ , 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _NeurIPS_ , 2017.
  * Venkatesh et al. [2021] Rahul Venkatesh, Tejan Karmali, Sarthak Sharma, Aurobrata Ghosh, R Venkatesh Babu, László A Jeni, and Maneesh Singh.  Deep implicit surface point prediction networks.  In _ICCV_ , 2021.
  * Wang et al. [2023a] Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang.  Dsvt: Dynamic sparse voxel transformer with rotated sets.  In _CVPR_ , 2023a.
  * Wang et al. [2023b] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In _CVPR_ , 2023b.
  * Wang et al. [2023c] Yizhi Wang, Zeyu Huang, Ariel Shamir, Hui Huang, Hao Zhang, and Ruizhen Hu.  Aro-net: Learning implicit fields from anchored radial observations.  In _CVPR_ , 2023c.
  * Wang et al. [2023d] Zhen Wang, Shijie Zhou, Jeong Joon Park, Despoina Paschalidou, Suya You, Gordon Wetzstein, Leonidas Guibas, and Achuta Kadambi.  Alto: Alternating latent topologies for implicit 3d reconstruction.  In _CVPR_ , 2023d.
  * Yan et al. [2022] Xingguang Yan, Liqiang Lin, Niloy J Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Shapeformer: Transformer-based shape completion via sparse representation.  In _CVPR_ , 2022.
  * Zhang et al. [2022] Biao Zhang, Matthias Nießner, and Peter Wonka.  3dilg: Irregular latent grids for 3d generative modeling.  _NeurIPS_ , 2022.
  * Zhang et al. [2023] Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka.  3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.  _TOG_ , 2023.
  * Zhao et al. [2021] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun.  Point transformer.  In _ICCV_ , 2021.

Supplementary Material

## Overview

In this supplementary material, we provide detailed descriptions of DITTO that
could not be handled in the main paper due to space constraints. In Sec. 1, we
describe more details necessary for implementing our work, such as detailed
network architecture and hyperparameters. Additional ablation studies related
to DITTO are available in Sec. 2. We provide additional experiment results in
Sec. 3.

##  1 Implementation Details

Additional details of the DITTO network architecture are available in Sec.
1.1, and hyperparameters that are used for training DITTO are available in
Sec. 1.2.

###  1.1 Network Architecture Details

In this section, we provide additional network architecture details of our
work. DITTO mainly consists of the dual latent encoder and the integrated
implicit decoder (IID). The dual latent encoder can be subdivided into a point
encoder and an UNet with dual latent layers (DLLs). An illustration of the
dual latent encoder is available in Fig. 1. We describe details of each of the
modules below. In addition, we provide layer-level details in Table 4.

Figure 1:  Detailed illustration of our dual latent encoder.

Point Encoder. The point encoder receives input point cloud $\mathcal{P}$, and
generates point latents $\mathcal{C}$. This module then produces grid latents
$\mathbf{T}$ by projecting $\mathcal{C}$ onto either triplanes or voxels.
While extracting $\mathcal{C}$, we employ a stack of four FKAConv layers [2]
instead of the conventional architecture based on PointNet [21], called as
local pooled PointNet. Local pooled PointNet generates point features by
directly encoding the point coordinates, but this layer processes each point
independently, without considering the relationship between points. In
contrast, simply employing a FKAConv-based encoder similar to POCO [1] gives
additional performance gains. This improvement is further reinforced by DLL
and IID due to their emphasis on point latents.

Dual Latent Layer. Our DLL focuses on refining point latents. To analyze what
DLL learns, we visualize point features in Fig. 2. The features of DITTO
exhibit clear boundaries between different parts of the object. For instance,
the body of the airplane has a distinct color compared to its wings, and
similarly, the bottom and side parts of the chair and the gun have different
colors. In contrast, most of the point features of ALTO [45] have similar
colors. From this difference, we infer that DITTO appears to implicitly learn
semantic information, such as planes and their direction or curvature. We
expect that these point features can help provide a clear surface boundary
between two surfaces that are adjacent yet not in contact. We would like to
note that DITTO considers point-level geometry with a point feature extractor,
such as the proposed DSPT, while ALTO handles point latents using MLPs that
account for each point independently.

We provide a visual comparison of our DLL, along with the architecture design
of ALTO (see Fig. 3). Our design of DLL mainly differs in two aspects: the
DSPT layers for point features refinement and the skip-connection from point
latents to grid latents. ALTO outputs refined grid latents
$\mathbf{T}^{\prime}$ that are directly projected from the point latents.
These grid latents have many empty cells, hindering feature extraction of grid
latents. To address this problem, we find that simply creating a residual
connection between grid latents and those projected from point latents can
enhance performance (see our additional ablation study in Sec. 2.1).
Additionally, we incorporate a convolutional layer to reduce sparseness of
grid latents projected from point latents.

(a) Point features of DITTO

(b) Point features of ALTO [45]

Figure 2:  Visualization of point features. We visualize the of the refined
point features $\tilde{\mathcal{C}}$. These features are colored by reducing
feature dimension into three channels by using principal component analysis
(PCA).

(a) DITTO (ours)

(b) ALTO [45]

Figure 3:  Visual comparison of DLL and a layer of ALTO. Key differences are
highlighted with red outlines and red arrows.

Dynamic Sparse Transformer. IID receives point coordinates $\mathcal{P}$ and
their features $\mathcal{C}$, and enhances the point features. When dividing
the point features into windows
$\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$, we recycle the sorting indices
in each DSPT layer to reduce computation load. Concretely, since both the
number and the coordinates of points remain constant throughout all processes
of DITTO, we initially calculate the sorted indices of point coordinates and
reuse them in every DSPT layer. This method effectively reduces the
computation needed for recalculating sorted indices.

UNet with Dual Latent Layers. Our UNet architecture is similar to traditional
UNet [32, 9], but each layer is replaced with the proposed DLL module. Our
UNet has three down DLLs, a mid DLL, and three up DLLs (see Fig. 1). The down
DLL can optionally downsample the grid latents using MaxPooling, while the up
DLL can upsample them using transposed convolution. Specifically, within our
UNet, the second and third down DLLs downsample the grid latents, whereas the
first and second up DLLs upsample them.

Integrated Implicit Decoder. IID receives the refined latents and estimates
occupancy by comprehensive consideration of these latents. Note that, while
Table 4 describes IID with a single query point $\mathbf{q}$ for convenience,
IID actually processes multiple query points $\mathcal{Q}$ in parallel.

###  1.2 Hyperparameters

The detailed hyperparameters can be found in Table 1.

Table 1:  Detailed hyperparameters of DITTO.

Notation | Meaning | Object (3K) | Object (1K) | Object(0.3K) | Scene (10K) | Scene (3K)
---|---|---|---|---|---|---
Triplane | Triplane | Triplane | Triplane | Voxel | Triplane | Voxel
| Epoch | 1,000 | 1,000 | 1,000 | 2,500 | 2,500 | 2,500 | 2,500
| Learning rate | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4
| Batch size | 32 | 32 | 32 | 32 | 16 | 32 | 16
$R$ | Feature resolution | 64 | 64 | 64 | 128 | 64 | 128 | 64
$d$ | Channel size | 32 | 32 | 32 | 32 | 32 | 32 | 32
$L$ | # of windows in DSPT | 25 | 25 | 25 | 20 | 20 | 25 | 25
$K$ | # of neighbor points in IID | 32 | 32 | 32 | 32 | 32 | 32 | 32
$M$ | # of query points per training iteration | 2,048 | 2,048 | 2,048 | 2,048 | 2,048 | 2,048 | 2,048

##  2 Ablation Studies

We conduct ablation studies to demonstrate effectiveness of each module of
DITTO. Specifically, we perform ablation studies on the point encoder, a
residual connection, and the number of windows for DSPT in Sec. 2.1, and Sec.
2.2, respectively.

###  2.1 Ablation Study on Residual Connection

To demonstrate the impact of residual connections between grid latents and
those projected from point latents, we conduct an ablation study comparing
ALTO with and without this residual connection (see Table 2). The results
suggest that only a simple addition of residual connection can significantly
enhance performance. Moreover, a convolutional layer can improve performance
by reducing the sparsity of grid latents projected from point latents.

Table 2:  Ablation study on residual connection. Training and inference are
conducted on the Synthetic Rooms dataset with 10K input points and 0.005 noise
level.

# of windows | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
---|---|---|---|---
ALTO (triplane) | 0.895 | 0.37 | 0.910 | 0.974
ALTO + residual connection | 0.904 | 0.36 | 0.915 | 0.976
ALTO + residual connection + conv layer | 0.907 | 0.36 | 0.915 | 0.977

###  2.2 Ablation Study on DSPT

We conduct an ablation study to determine the appropriate number of windows
for DSPT. The results can be found in Table 3. The results demonstrate
robustness across various window numbers. However, the current window number
($L=20$) shows slightly improved performance.

Table 3:  Ablation study on the number of windows in our DSPT. The training
and inference are conducted on the Synthetic Rooms dataset with 10K input
points and 0.005 noise level.

# of windows | IoU $\uparrow$ | Chamfer-$L_{1}$ $\downarrow$ | NC $\uparrow$ | F-score $\uparrow$
---|---|---|---|---
40 | 0.929 | 0.34 | 0.930 | 0.984
25 | 0.930 | 0.34 | 0.930 | 0.983
20 (DITTO) | 0.931 | 0.33 | 0.931 | 0.984
10 | 0.929 | 0.34 | 0.930 | 0.984

##  3 Additional Experiment Results

In this section, we provide additional qualitative results in Sections 3.1,
3.2, 3.3. We also include a video of the qualitative results in
qualitative_video.mp4, which displays the results in turntable.

###  3.1 Additional Results on ShapeNet

Quantitative Results. We provide additional object-level 3D surface
reconstruction results on ShapeNet [4]. Detailed per-category quantitative
results are presented with different input point densities: 3K input points in
Table 5, 1K input points in Table 6, and 0.3K input points in Table 7, all at
a consistent noise level of 0.005. Each of these tables is a per-category
extension to Table 1 in the main paper. DITTO outperforms previous methods in
most categories. Note that, while ALTO [45] outperforms POCO [1] in most of
metrics, POCO shows higher F-score than ALTO when dealing with 3K input
points. In contrast, DITTO demonstrates superior performance in most metrics
and categories.

Qualitative Results. We present additional object-level 3D reconstruction
results on ShapeNet. The result meshes are visualized in Fig. 4 for 3K input
points at 0.005 noise level. DITTO shows high-fidelity reconstruction
especially thin and intricate structures such as the legs of the chairs and
the tables (second and third rows). Enhanced details are also notable, such as
the boundary interface between two parts of the chair (second row).

To demonstrate robustness for sparsity, we visualize qualitative results in
sparse case: 1K input points in Fig. 5, 0.3K input points in Fig. 6 with
consistent noise level. Even with sparse point clouds, DITTO shows superior
reconstruction quality in intricate shapes, such as the bookshelf (first row
in Fig. 5) and underside of the car (third row in Fig. 5). Moreover, the
results of DITTO generate more clear shape details, such as the chair (second
row in Fig. 5).

###  3.2 Additional Results on Synthetic Rooms

We provide additional qualitative results for scene-level 3D surface
reconstruction on the Synthetic Rooms dataset [30]. We visualize the results
in Fig. 7 for 10K input points with 0.005 noise level. Result meshes of DITTO
exhibit clear surface boundaries in intricate cases, such as the bookshelf
(box in the left scene), and the lamps (boxes in the middle and right scenes).
In addition, due to the precise detail reconstruction capability of DITTO, it
can successfully reconstruct fine details of chairs (boxes in the middle and
right scenes) and lamps (boxes in the middle scene).

To demonstrate the performance with sparse input point clouds, we visualize
the results for 3K input points in Fig. 8. Even with sparse point clouds,
DITTO outperforms previous methods. DITTO is the only method that successfully
reconstructs the chairs (boxes in the left and middle scenes). In addition,
DITTO is the most successful method in reconstructing the bookshelves (boxes
in the right scene).

###  3.3 Additional Results on ScanNet-V2

We present additional qualitative results on ScanNet-v2 [10] to demonstrate
generalization performance. Consistent with the main paper, we test on this
dataset using models pre-trained with the Synthetic Rooms dataset (see Fig.
9). DITTO successfully reconstructs the tables and sofas (boxes in the left
and middle scenes). For the right scene, our method successfully reconstructs
the tables and chairs, even though every previous method fails to generate the
scene accurately. These results demonstrate the robustness of DITTO in
handling complex geometries and details, even in the dataset that is not used
during the training phase.

Table 4:  Layer-level network module architecture designs of DITTO.

Layer Name | Input | Output
---|---|---
Point Encoder |  |
Input / Output | $\mathcal{P}\ (N\times 3)$ | $\mathcal{C}\ (N\times d),\mathbf{T}\ (3\times R\times R\times d)$
FKAConv layer | $\mathcal{P}\ (N\times 3)$ | $\mathcal{C}\ (N\times d)$
FKAConv layer | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times d)$
FKAConv layer | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times d)$
FKAConv layer | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times d)$
Quantization | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $\mathbf{T}\ (3\times R\times R\times d)$
DLL |  |
Input / Output | $\mathbf{T}\ (3\times R\times R\times d_{\text{in}}),\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d_{\text{in}})$ | $\mathbf{T}^{\prime}\ (3\times R/2\times R/2\times d_{\text{out}}),\mathcal{C}^{\prime}\ (N\times d_{\text{out}})$
ConvNet | $\mathbf{T}\ (3\times R\times R\times d_{\text{in}})$ | $\mathbf{T}\ (3\times R\times R\times d_{\text{out}})$
Conv2d | $\bar{\mathbf{T}}\ (3\times R\times R\times d_{\text{in}})$ | $\bar{\mathbf{T}}\ (3\times R\times R\times d_{\text{out}})$
Sum | $\mathbf{T}\ (3\times R\times R\times d_{\text{out}}),\bar{\mathbf{T}}\ (3\times R\times R\times d_{\text{out}})$ | $\bar{\mathbf{T}}\ (3\times R\times R\times d_{\text{out}})$
3D-Aware-Conv | $\bar{\mathbf{T}}\ (3\times R\times R\times d_{\text{out}})$ | $\bar{\mathbf{T}}^{\prime}\ (3\times R\times R\times d_{\text{out}})$
Linear | $\mathcal{C}\ (N\times d_{\text{in}})$ | $\mathcal{C}\ (N\times d_{\text{out}})$
Grid-to-Point | $\bar{\mathbf{T}}^{\prime}\ (3\times R\times R\times d_{\text{out}}),\mathcal{P}\ (N\times 3)$ | $(N\times d_{\text{out}})$
MLP | $(N\times d_{\text{out}})$ | $(N\times d_{\text{out}})$
Sum | $(N\times d_{\text{out}}),\mathcal{C}\ (N\times d_{\text{out}})$ | $\mathcal{C}\ (N\times d_{\text{out}})$
X-Axis DSPT | $\mathcal{P}(N\times 3),\mathcal{C}\ (N\times d_{\text{out}})$ | $\mathcal{C}\ (N\times d_{\text{out}})$
Y-Axis DSPT | $\mathcal{P}(N\times 3),\mathcal{C}\ (N\times d_{\text{out}})$ | $\mathcal{C}\ (N\times d_{\text{out}})$
Z-Axis DSPT | $\mathcal{P}(N\times 3),\mathcal{C}\ (N\times d_{\text{out}})$ | $\mathcal{C}^{\prime}\ (N\times d_{\text{out}})$
Point-to-Grid | $\mathcal{P}(N\times 3),\mathcal{C}^{\prime}\ (N\times d_{\text{out}})$ | $(3\times R\times R\times d_{\text{out}})$
Conv2d | $(3\times R\times R\times d_{\text{out}})$ | $(3\times R\times R\times d_{\text{out}})$
Sum | $(3\times R\times R\times d_{\text{out}}),\bar{\mathbf{T}}^{\prime}\ (3\times R\times R\times d_{\text{out}})$ | $\mathbf{T}^{\prime}\ (3\times R\times R\times d_{\text{out}})$
Pooling | $\mathbf{T}^{\prime}\ (3\times R\times R\times d_{\text{out}})$ | $\mathbf{T}^{\prime}\ (3\times R/2\times R/2\times d_{\text{out}})$
UNet with DLLs |  |
Input / Output | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d),\mathbf{T}\ (3\times R\times R\times d)$ | $\mathbf{T}^{\prime}\ (3\times R\times R\times d),\tilde{\mathcal{C}}\ (N\times d)\textbf{}$
DLL (down) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d),\mathbf{T}\ (3\times R\times R\times d)$ | $\mathcal{C}^{\prime}\ (N\times d),\mathbf{T}^{\prime}\ (3\times R\times R\times d)$
DLL (down) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d),\mathbf{T}\ (3\times R\times R\times d)$ | $\mathcal{C}^{\prime}\ (N\times 2d),\mathbf{T}^{\prime}\ (3\times R/2\times R/2\times 2d)$
DLL (down) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times 2d),\mathbf{T}\ (3\times R/2\times R/2\times 2d)$ | $\mathcal{C}^{\prime}\ (N\times 4d),\mathbf{T}^{\prime}\ (3\times R/4\times R/4\times 4d)$
DLL (mid) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times 4d),\mathbf{T}\ (3\times R/4\times R/4\times 4d)$ | $\mathcal{C}^{\prime}\ (N\times 8d),\mathbf{T}^{\prime}\ (3\times R/4\times R/4\times 8d)$
DLL (up) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times 8d),\mathbf{T}\ (3\times R/4\times R/4\times 8d)$ | $\mathcal{C}^{\prime}\ (N\times 4d),\mathbf{T}^{\prime}\ (3\times R/2\times R/2\times 4d)$
DLL (up) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times 4d),\mathbf{T}\ (3\times R/2\times R/2\times 4d)$ | $\mathcal{C}^{\prime}\ (N\times 2d),\mathbf{T}^{\prime}\ (3\times R\times R\times 2d)$
DLL (up) | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times 2d),\mathbf{T}\ (3\times R\times R\times 2d)$ | $\mathcal{C}^{\prime}\ (N\times d),\mathbf{T}^{\prime}\ (3\times R\times R\times d)$
DSPT |  |
Input / Output | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times d)$
sort | $\mathcal{P}\ (N\times 3),\mathcal{C}\ (N\times d)$ | $(N\times d)$
split | $(N\times d)$ | $\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$
Self-attention | $\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$ | $\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$
split-1 | $\\{\mathcal{C}^{\text{wnd}}_{l}\\}_{l=1}^{L}$ | $(N\times d)$
sort-1 | $(N\times d)$ | $\mathcal{C}\ (N\times d)$
Layer norm | $\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times d)$
Linear | $\mathcal{C}\ (N\times d)$ | $\mathcal{C}\ (N\times 4d)$
ReLU | $\mathcal{C}\ (N\times 4d)$ | $\mathcal{C}\ (N\times 4d)$
Linear | $\mathcal{C}\ (N\times 4d)$ | $\mathcal{C}\ (N\times d)$
IID |  |
Input / Output | $\tilde{\mathbf{T}}\ (3\times R\times R\times d),\mathcal{P}\ (N\times 3),\tilde{\mathcal{C}}\ (N\times d),\mathbf{q}\ (3)$ | $\tilde{o}\ (1)$
Interpolation | $\tilde{\mathbf{T}}\ (3\times R\times R\times d),\mathbf{q}\ (3)$ | $(d)$
Linear | $(d)$ | $\mathbf{z}_{0}\ (2d)$
KNN | $\mathbf{q}\ (3),\mathcal{P}\ (N\times 3),\tilde{\mathcal{C}}\ (N\times d)$ | $\mathcal{P}^{\mathbf{q}}\ (K\times 3),\tilde{\mathcal{C}}^{\mathbf{q}}\ (K\times d)$
interpolation | $\tilde{\mathbf{T}}\ (3\times R\times R\times d),\mathcal{P}^{\mathbf{q}}\ (K\times 3)$ | $\tilde{\mathbf{T}}^{\mathbf{q}}\ (K\times d)$
concat | $\tilde{\mathcal{C}}^{\mathbf{q}}\ (K\times d),\tilde{\mathbf{T}}^{\mathbf{q}}\ (K\times d)$ | $\mathcal{Z}^{\mathbf{q}}\ (K\times 2d)$
Self-attention | $\\{\mathbf{z}_{0}^{\mathbf{q}},\dots,\mathbf{z}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 2d),\\{\mathbf{q},\mathbf{p}_{1}^{\mathbf{q}},\dots,\mathbf{p}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 3)$ | $\mathbf{z}_{0}^{\mathbf{q}}\ (2d)$
Self-attention | $\\{\mathbf{z}_{0}^{\mathbf{q}},\dots,\mathbf{z}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 2d),\\{\mathbf{q},\mathbf{p}_{1}^{\mathbf{q}},\dots,\mathbf{p}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 3)$ | $\mathbf{z}_{0}^{\mathbf{q}}\ (2d)$
Self-attention | $\\{\mathbf{z}_{0}^{\mathbf{q}},\dots,\mathbf{z}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 2d),\\{\mathbf{q},\mathbf{p}_{1}^{\mathbf{q}},\dots,\mathbf{p}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 3)$ | $\mathbf{z}_{0}^{\mathbf{q}}\ (2d)$
Self-attention | $\\{\mathbf{z}_{0}^{\mathbf{q}},\dots,\mathbf{z}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 2d),\\{\mathbf{q},\mathbf{p}_{1}^{\mathbf{q}},\dots,\mathbf{p}_{K}^{\mathbf{q}}\\}\ ((K{+}1)\times 3)$ | $\hat{\mathbf{z}}_{0}^{\mathbf{q}}\ (2d)$
Linear | $\hat{\mathbf{z}}_{0}^{\mathbf{q}}\ (2d)$ | $\tilde{o}\ (1)$

Table 5:  Object-level quantitative comparison on ShapeNet with 3K input
points with noise level 0.005.

Method | IoU $\uparrow$ | Chamfer-$L_{1}\ \downarrow$
---|---|---
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.734 | 0.849 | 0.902 | 0.908 | 0.935 | 0.64 | 0.34 | 0.23 | 0.22 | 0.19
Bench | 0.682 | 0.830 | 0.865 | 0.890 | 0.919 | 0.67 | 0.35 | 0.28 | 0.26 | 0.23
Cabinet | 0.855 | 0.940 | 0.960 | 0.965 | 0.976 | 0.82 | 0.46 | 0.37 | 0.34 | 0.31
Car | 0.830 | 0.886 | 0.921 | 0.924 | 0.943 | 1.04 | 0.75 | 0.41 | 0.43 | 0.36
Chair | 0.720 | 0.871 | 0.919 | 0.925 | 0.948 | 0.95 | 0.46 | 0.33 | 0.32 | 0.29
Display | 0.799 | 0.927 | 0.956 | 0.962 | 0.973 | 0.82 | 0.36 | 0.28 | 0.27 | 0.25
Lamp | 0.546 | 0.785 | 0.877 | 0.868 | 0.914 | 1.59 | 0.59 | 0.33 | 0.34 | 0.28
Loudspeaker | 0.826 | 0.918 | 0.957 | 0.953 | 0.970 | 1.18 | 0.64 | 0.41 | 0.41 | 0.35
Rifle | 0.668 | 0.846 | 0.897 | 0.898 | 0.925 | 0.66 | 0.28 | 0.19 | 0.19 | 0.16
Sofa | 0.865 | 0.936 | 0.963 | 0.966 | 0.976 | 0.73 | 0.42 | 0.30 | 0.29 | 0.26
Table | 0.739 | 0.888 | 0.924 | 0.937 | 0.956 | 0.76 | 0.38 | 0.31 | 0.29 | 0.27
Telephone | 0.896 | 0.955 | 0.968 | 0.977 | 0.982 | 0.46 | 0.27 | 0.22 | 0.21 | 0.20
Vessel | 0.729 | 0.865 | 0.927 | 0.924 | 0.948 | 0.94 | 0.43 | 0.25 | 0.26 | 0.22
mean | 0.761 | 0.884 | 0.926 | 0.931 | 0.949 | 0.87 | 0.44 | 0.30 | 0.30 | 0.27
Method | NC $\uparrow$ | F-Score $\uparrow$
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.886 | 0.931 | 0.944 | 0.949 | 0.958 | 0.829 | 0.965 | 0.994 | 0.992 | 0.997
Bench | 0.871 | 0.921 | 0.928 | 0.941 | 0.950 | 0.827 | 0.964 | 0.988 | 0.991 | 0.996
Cabinet | 0.913 | 0.956 | 0.961 | 0.967 | 0.970 | 0.833 | 0.956 | 0.979 | 0.982 | 0.989
Car | 0.874 | 0.893 | 0.894 | 0.917 | 0.914 | 0.747 | 0.849 | 0.946 | 0.940 | 0.963
Chair | 0.886 | 0.943 | 0.956 | 0.959 | 0.968 | 0.730 | 0.939 | 0.985 | 0.985 | 0.994
Display | 0.926 | 0.968 | 0.975 | 0.976 | 0.981 | 0.795 | 0.971 | 0.994 | 0.993 | 0.997
Lamp | 0.809 | 0.900 | 0.929 | 0.924 | 0.942 | 0.581 | 0.892 | 0.975 | 0.962 | 0.984
Loudspeaker | 0.903 | 0.939 | 0.952 | 0.951 | 0.961 | 0.727 | 0.892 | 0.964 | 0.955 | 0.976
Rifle | 0.849 | 0.929 | 0.949 | 0.949 | 0.960 | 0.818 | 0.980 | 0.998 | 0.996 | 0.999
Sofa | 0.928 | 0.958 | 0.967 | 0.971 | 0.975 | 0.832 | 0.953 | 0.989 | 0.987 | 0.994
Table | 0.917 | 0.959 | 0.966 | 0.968 | 0.975 | 0.824 | 0.967 | 0.991 | 0.990 | 0.996
Telephone | 0.970 | 0.983 | 0.985 | 0.987 | 0.988 | 0.930 | 0.989 | 0.998 | 0.998 | 0.999
Vessel | 0.857 | 0.919 | 0.940 | 0.940 | 0.952 | 0.734 | 0.931 | 0.989 | 0.982 | 0.992
mean | 0.891 | 0.938 | 0.950 | 0.954 | 0.957 | 0.785 | 0.942 | 0.984 | 0.981 | 0.988

Table 6:  Object-level quantitative comparison on ShapeNet with 1K input
points with noise level 0.005.

Method | IoU $\uparrow$ | Chamfer-$L_{1}\ \downarrow$
---|---|---
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.748 | 0.825 | 0.850 | 0.872 | 0.908 | 0.59 | 0.39 | 0.32 | 0.29 | 0.23
Bench | 0.702 | 0.798 | 0.804 | 0.856 | 0.891 | 0.62 | 0.40 | 0.38 | 0.30 | 0.26
Cabinet | 0.862 | 0.926 | 0.936 | 0.953 | 0.964 | 0.76 | 0.50 | 0.46 | 0.37 | 0.35
Car | 0.837 | 0.867 | 0.878 | 0.901 | 0.921 | 0.99 | 0.83 | 0.60 | 0.50 | 0.45
Chair | 0.736 | 0.837 | 0.867 | 0.894 | 0.922 | 0.89 | 0.55 | 0.44 | 0.39 | 0.33
Display | 0.812 | 0.911 | 0.930 | 0.946 | 0.960 | 0.78 | 0.41 | 0.34 | 0.31 | 0.28
Lamp | 0.567 | 0.741 | 0.807 | 0.820 | 0.877 | 1.44 | 0.68 | 0.50 | 0.50 | 0.35
Loudspeaker | 0.831 | 0.899 | 0.923 | 0.933 | 0.951 | 1.14 | 0.72 | 0.54 | 0.48 | 0.42
Rifle | 0.680 | 0.801 | 0.850 | 0.862 | 0.892 | 0.63 | 0.36 | 0.27 | 0.25 | 0.20
Sofa | 0.873 | 0.921 | 0.937 | 0.952 | 0.964 | 0.69 | 0.47 | 0.38 | 0.33 | 0.30
Table | 0.757 | 0.858 | 0.880 | 0.913 | 0.937 | 0.70 | 0.44 | 0.38 | 0.33 | 0.30
Telephone | 0.897 | 0.946 | 0.953 | 0.968 | 0.975 | 0.46 | 0.29 | 0.26 | 0.23 | 0.21
Vessel | 0.736 | 0.840 | 0.880 | 0.893 | 0.923 | 0.91 | 0.51 | 0.37 | 0.33 | 0.27
mean | 0.772 | 0.859 | 0.884 | 0.905 | 0.926 | 0.82 | 0.50 | 0.40 | 0.35 | 0.32
Method | NC $\uparrow$ | F-Score $\uparrow$
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.894 | 0.922 | 0.920 | 0.933 | 0.949 | 0.850 | 0.946 | 0.970 | 0.976 | 0.990
Bench | 0.882 | 0.911 | 0.902 | 0.925 | 0.940 | 0.849 | 0.943 | 0.956 | 0.979 | 0.990
Cabinet | 0.925 | 0.949 | 0.945 | 0.957 | 0.964 | 0.852 | 0.939 | 0.951 | 0.972 | 0.978
Car | 0.904 | 0.885 | 0.867 | 0.889 | 0.904 | 0.763 | 0.819 | 0.868 | 0.912 | 0.934
Chair | 0.893 | 0.931 | 0.930 | 0.946 | 0.960 | 0.753 | 0.902 | 0.943 | 0.965 | 0.982
Display | 0.930 | 0.961 | 0.962 | 0.970 | 0.976 | 0.805 | 0.956 | 0.976 | 0.984 | 0.991
Lamp | 0.820 | 0.885 | 0.895 | 0.905 | 0.929 | 0.606 | 0.845 | 0.924 | 0.926 | 0.964
Loudspeaker | 0.914 | 0.929 | 0.928 | 0.936 | 0.950 | 0.740 | 0.863 | 0.908 | 0.926 | 0.951
Rifle | 0.859 | 0.916 | 0.928 | 0.936 | 0.949 | 0.828 | 0.957 | 0.984 | 0.987 | 0.994
Sofa | 0.937 | 0.950 | 0.950 | 0.960 | 0.969 | 0.846 | 0.932 | 0.961 | 0.974 | 0.985
Table | 0.918 | 0.950 | 0.949 | 0.961 | 0.970 | 0.842 | 0.947 | 0.964 | 0.979 | 0.989
Telephone | 0.972 | 0.980 | 0.979 | 0.984 | 0.986 | 0.940 | 0.983 | 0.990 | 0.994 | 0.996
Vessel | 0.866 | 0.906 | 0.913 | 0.923 | 0.940 | 0.740 | 0.899 | 0.952 | 0.961 | 0.979
mean | 0.901 | 0.929 | 0.928 | 0.940 | 0.949 | 0.801 | 0.918 | 0.950 | 0.964 | 0.975

Table 7:  Object-level quantitative comparison on ShapeNet with 0.3K input
points with noise level 0.005.

Method | IoU $\uparrow$ | Chamfer-$L_{1}\ \downarrow$
---|---|---
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.760 | 0.782 | 0.744 | 0.825 | 0.857 | 0.57 | 0.48 | 0.57 | 0.39 | 0.32
Bench | 0.716 | 0.743 | 0.707 | 0.801 | 0.835 | 0.60 | 0.50 | 0.56 | 0.39 | 0.34
Cabinet | 0.867 | 0.900 | 0.889 | 0.927 | 0.941 | 0.73 | 0.52 | 0.58 | 0.46 | 0.43
Car | 0.834 | 0.843 | 0.817 | 0.867 | 0.885 | 0.99 | 0.76 | 0.83 | 0.67 | 0.61
Chair | 0.736 | 0.787 | 0.776 | 0.840 | 0.871 | 0.89 | 0.67 | 0.71 | 0.52 | 0.45
Display | 0.817 | 0.885 | 0.878 | 0.917 | 0.931 | 0.76 | 0.47 | 0.49 | 0.38 | 0.35
Lamp | 0.567 | 0.663 | 0.681 | 0.747 | 0.808 | 1.38 | 1.02 | 0.93 | 0.76 | 0.61
Loudspeaker | 0.827 | 0.870 | 0.867 | 0.901 | 0.916 | 1.16 | 0.78 | 0.79 | 0.64 | 0.59
Rifle | 0.691 | 0.757 | 0.742 | 0.801 | 0.832 | 0.61 | 0.43 | 0.45 | 0.35 | 0.30
Sofa | 0.872 | 0.898 | 0.893 | 0.926 | 0.938 | 0.69 | 0.52 | 0.53 | 0.42 | 0.38
Table | 0.758 | 0.813 | 0.794 | 0.868 | 0.894 | 0.72 | 0.52 | 0.57 | 0.42 | 0.37
Telephone | 0.916 | 0.939 | 0.927 | 0.952 | 0.960 | 0.41 | 0.31 | 0.33 | 0.27 | 0.25
Vessel | 0.748 | 0.797 | 0.795 | 0.846 | 0.872 | 0.85 | 0.63 | 0.60 | 0.47 | 0.40
mean | 0.778 | 0.821 | 0.808 | 0.863 | 0.882 | 0.80 | 0.59 | 0.61 | 0.47 | 0.43
Method | NC $\uparrow$ | F-Score $\uparrow$
ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours) | ONet [25] | ConvONet [30] | POCO [1] | ALTO [45] |  DITTO (ours)
Airplane | 0.897 | 0.901 | 0.867 | 0.914 | 0.931 | 0.864 | 0.902 | 0.867 | 0.938 | 0.962
Bench | 0.878 | 0.886 | 0.864 | 0.906 | 0.920 | 0.860 | 0.912 | 0.882 | 0.947 | 0.966
Cabinet | 0.916 | 0.931 | 0.917 | 0.943 | 0.953 | 0.856 | 0.916 | 0.896 | 0.943 | 0.957
Car | 0.875 | 0.864 | 0.835 | 0.873 | 0.887 | 0.757 | 0.810 | 0.766 | 0.850 | 0.879
Chair | 0.889 | 0.905 | 0.885 | 0.923 | 0.940 | 0.754 | 0.850 | 0.833 | 0.910 | 0.941
Display | 0.926 | 0.947 | 0.938 | 0.956 | 0.964 | 0.813 | 0.926 | 0.916 | 0.957 | 0.967
Lamp | 0.813 | 0.853 | 0.834 | 0.875 | 0.902 | 0.618 | 0.771 | 0.781 | 0.857 | 0.908
Loudspeaker | 0.897 | 0.911 | 0.897 | 0.916 | 0.932 | 0.737 | 0.832 | 0.819 | 0.871 | 0.899
Rifle | 0.863 | 0.890 | 0.883 | 0.909 | 0.925 | 0.838 | 0.919 | 0.918 | 0.952 | 0.968
Sofa | 0.928 | 0.935 | 0.924 | 0.946 | 0.956 | 0.846 | 0.906 | 0.899 | 0.941 | 0.956
Table | 0.917 | 0.933 | 0.917 | 0.945 | 0.957 | 0.839 | 0.913 | 0.894 | 0.947 | 0.966
Telephone | 0.970 | 0.975 | 0.970 | 0.978 | 0.982 | 0.942 | 0.975 | 0.971 | 0.984 | 0.988
Vessel | 0.860 | 0.879 | 0.867 | 0.898 | 0.914 | 0.758 | 0.850 | 0.851 | 0.909 | 0.935
mean | 0.895 | 0.908 | 0.892 | 0.922 | 0.931 | 0.806 | 0.883 | 0.869 | 0.924 | 0.940

(a) GT Mesh

(b) Input points

(c) ConvONet [30]

(d) POCO [1]

(e) ALTO [45]

(f) DITTO (ours)

Figure 4:  Object-level 3D reconstruction comparison on ShapeNet [4] with 3K
input points and noise level 0.005.

(a) GT Mesh

(b) Input points

(c) ConvONet [30]

(d) POCO [1]

(e) ALTO [45]

(f) DITTO (ours)

Figure 5:  Object-level 3D reconstruction comparison on ShapeNet [4] with 1K
input points and noise level 0.005.

(a) GT Mesh

(b) Input points

(c) ConvONet [30]

(d) POCO [1]

(e) ALTO [45]

(f) DITTO (ours)

Figure 6:  Object-level 3D reconstruction comparison on ShapeNet [4] with 0.3K
input points and noise level 0.005.

(a) Input points

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 7:  Scene-level 3D reconstruction results on the Synthetic Rooms
dataset [30] with 10K input points and noise level 0.005.

(a) Input points

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 8:  Scene-level 3D reconstruction results on the Synthetic Rooms
dataset [30] with 3K input points and noise level 0.005.

(a) Input points

(b) ConvONet [30]

(c) POCO [1]

(d) ALTO [45]

(e) DITTO (ours)

Figure 9:  Scene-level 3D reconstruction results on the ScanNet-v2 dataset
[10].