# Performance Evalution of 3D Keypoint Detectors and Descriptors for Plants
Health Classification

Shiva Azimi
Department of Electrical Engineering
Indian Institute of Technology-Delhi
Hauz Khas  New Delhi 110016  India
<EMAIL_ADDRESS>
Brejesh Lall
Department of Electrical Engineering
Indian Institute of Technology-Delhi
Hauz Khas  New Delhi 110016  India
<EMAIL_ADDRESS>
Tapan K. Gandhi
Department of Electrical Engineering
Indian Institute of Technology-Delhi
Hauz Khas  New Delhi 110016  India
<EMAIL_ADDRESS>

## Abstract

Plant Phenomics based on imaging based techniques can be used to monitor the
health and the diseases of plants and crops. The use of 3D data for plant
phenomics is a recent phenomenon. However, since 3D point cloud contains more
information than plant images, in this paper, we compare the performance of
different keypoint detectors and local feature descriptors combinations for
the plant growth stage and it’s growth condition classification based on 3D
point clouds of the plants. We have also implemented a modified form of 3D
SIFT descriptor, that is invariant to rotation and is computationally less
intense than most of the 3D SIFT descriptors reported in the existing
literature. The performance is evaluated in terms of the classification
accuracy and the results are presented in terms of accuracy tables. We find
the ISS-SHOT and the SIFT-SIFT combinations consistently perform better and
Fisher Vector (FV) is a better encoder than Vector of Linearly Aggregated
(VLAD) for such applications. It can serve as a better modality.

##  1 INTRODUCTION

Increasing world population and the loss of arable land due to various
climatic and man made factors have necessitated the development of modern
tools and techniques for increasing crop yield [1]. Plant phenotyping is a
field of plant science dealing with the measurement of phenomes and how they
change in reaction to genetic and environmental changes.
In recent years, non-invasive image-based high-throughput plant phenotyping
has emerged as an important field of computer vision research [2].
Measurements of plant features such as biomass, size and inclination, and leaf
width, length, and area can be done using two dimensional (2D) [3] or three
dimensional (3D) models of the plant at different scales [4]. However,
approaches based on 2D models have many limitations since features such as the
angle, thickness, and orientation could not be represented using such models.
To overcome these issues associated with 2D models, 3D models could be used
for plant phenotyping.
3D modeling for plant phenotyping consists of two steps [2]. The first step
consists of localization and mapping. Here, we define the sensors’ poses and
compute the scene 3D model. The second step is the understanding step. Here,
we detect the plant, segment and classify its parts and perform the
measurements. To do this, we need to extract features of interest in the
plant. The features can be global or local, however local features have been
proven more successful in vision tasks like 3D object categorization and
recognition.
Extraction of local features has two main steps of detection of keypoints and
description of patches around these keypoints. In the keypoints detection
stage, points having rich information content identified. Detection of
keypoints (or interest points) is important for reducing the amount of
computation required in computer vision applications dealing with 3D point
clouds because of huge amount of data points present in such point clouds and
the high computational cost of descriptors. Once the keypoints are detected,
descriptors are extracted for these keypoints. A number of 3D keypoints
detectors and descriptors such as Harris [5], Scale-Invariant feature
transform (SIFT) [6], intrinsic shape signatures (ISS) [7], signatures of
histograms (SHOT) [8], etc., have been proposed in the literature.
This work is motivated by the need to quantitatively compare the performance
of different keypoint detectors and local feature descriptors for plant health
and growth monitoring. Although, there are studies in the existing literature
comparing the performance of 3D keypoint detectors and/or feature descriptors,
they all almost either compare either the keypoints detectors or the feature
descriptors in certain specific applications [9]. To the best of our
knowledge, there is no work in the existing literature that deals specifically
with the performance evaluation of the keypoints detectors and the local
feature descriptors as applied to 3D point cloud data in the context of plant
health classification and plant growth analysis. Working with 3D point cloud
data in the context of plant health classification and plant stress analysis
offers special challenges:

  * •

Plants are made up of very fine structures that makes it very difficult to
make a perfect scan. Thus the keypoint detectors local feature descriptors
have to be robust to noise and holes present in the point cloud data.

  * •

The lighting conditions in farming keep changing, offering different
illumination conditions, thus, making it very difficult to use the color
information for decision making.

In summary, in this paper, we consider three most common 3D keypoints
detectors, viz, Harris3D, 3D SIFT and ISS along with two prominent local
feature descriptors, viz, 3D SIFT and SHOT. We have also modified the 3D SIFT
descriptor, such that it is invariant to rotation. We provide a quantitative
comparison and analysis of these and investigate how they perform in
combinations.
The remainder of the paper is organized as follows. Section II gives a brief
overview of the related work. We give a description of the keypoints detectors
and the local feature descriptors that we have considered for evaluation in
Section III. performance evaluation is provided in Section IV. Finally, in
Section V, we provide some perspectives and conclude the paper.

##  2 Related Works

A number of methods have been proposed for creating 3D models for plants.
Early approaches for 3D digitization of plants used laser scanning using LiDAR
devices [10], Time-of-Flight cameras [11], Multi-View Stereo (MVS), etc. A 3D
reconstruction of plant using MVS and free moving cameras was proposed in [2]
for solving the occlusion problem.
After making suitable 3D reconstruction of plants with good resolution, we can
extract features of interest in the plants. Different traits like plant
volume, leaf area, and stem length can be estimated by using 3D plant model.
Wei et al. obtained some volumetric features after making 3D reconstruction of
plant by silhouette method [12]. Extracting features from surfaces meshes is
developed recently. Segmented mesh method for computing plant information such
as stem length, leaf length and width was proposed in [13].
However, there are not enough work in the existing literature about
descriptors and detectors for plant phenotyping. In addition, it is not clear
which of them suitable for which work and application. Most of the work
dealing with the performance of the keypoints detectors and local feature
descriptors, either evaluate the keypoints detectors or the descriptors, in a
particular context. Furthermore, it is not clear how different keypoints
detectors and descriptors work in combinations. F.Tombari evaluated 3D
keypoints detectors for object recognition in case of occlusion and clutter
[14]. In [15], the authors presented a comparison between 3D point detectors
and descriptors in the context of point cloud fusion (point cloud registration
or alignment). Y. Guo et al. [16] presented a comprehensive evaluation of 3D
local feature descriptors on eight different kind of datasets.

##  3 3D Keypoints Detectors and Descriptors

In this section, we briefly describe the 3D keypoints detectors and the 3D
local feature descriptors that we have considered for performance evaluation
in this paper.

###  3.1 3D Keypoints Detectors

####  3.1.1 Harris3D

The 3D-Harris detector is the extension of the 2D corner detection method of
Harris and Stephens [5], and works by taking the first order derivatives along
two orthogonal directions on the 3D surface. The derivatives are obtained by
fitting a quadratic surface to the neighborhood of a vertex. Unlike the 2D
detector, the image gradients in the covariance matrix are replaced by surface
normals. To find the keypoints, a Hessian matrix of the intensity is used
around each point. This matrix is then smoothed by an isotropic Gaussian
filter.

####  3.1.2 ISS

ISS keypoint detector [7] relies on region-wise quality measurements for 3D
object recognition which is designed to be stable, repeatable, informative,
and discriminative, ensuring highly accurate 3D shape matching and
recognition. This method uses the magnitude of the smallest eigenvalue and the
ratio between two successive eigenvalues.

####  3.1.3 SIFT3D

The SIFT keypoint detector finds local extrema in a Difference-of-Gaussians
(DoG) scale-space [6]. 3D SIFT is an extension of the 2D SIFT to 3D. First a
density function is approximated by sampling the data regularly in space. A
scale space is then built over the density function, and a search is done for
local maxima of the Hessian determinant. To generate scale space the input
cloud is convolved with a number of Gaussian filters whose standard deviations
differ by a fixed scale factor. The 3D SIFT keypoints are positioned at the
scale-space extrema of the DoG function. All the keypoints with low curvature
values are rejected to get stable results.

(a) Sorghum (control)

(b) Sorghum (heat)

(c) Sorghum (shade)

(d) Tobacco (control)

(e) Tobacco (heat)

(f) Tobacco (shade)

(g) Tomato (control)

(h) Tomato (heat)

(i) Tomato (shade)

Figure 1: Sample point clouds from the dataset for the three conditions:
control, heat and shade

###  3.2 3D Local Feature Descriptors

####  3.2.1 SIFT3D

3D SIFT involves the same functional steps as for 2D SIFT. In our case, we
have slightly modified the steps to make the descriptors invariant to rotation
and to speed up the description process. For example, we take the entire
$r-point$ neighborhood around each keypoint rather than dividing it into
$n\times n\times n$ subregions (with $n=4$, as in Lowe’s 2D SIFT). Using these
$r$-point a 2D-histogram is produced by grouping the angles $\theta$ and
$\phi$ into 10 degree angular bins. Similar to 2D SIFT, a regional Gaussian
weighting of
${e^{{{-2d}\mathord{\left/{\vphantom{{-2d}{{R_{\max}}}}}\right.\kern-1.2pt}{{R_{\max}}}}}}$
for the points that are at a distance $d$ is applied to the histogram. Here
$R_{max}$ represents the distance of the neighborhood point farthest from the
center. The prominent azimuth $\alpha$ and elevation $\beta$ for a keypoint
are given by the peaks of the 2D-histogram. Each keypoint $p$ is described by
its location
${\bf{p}}\buildrel\Delta\over{=}{\left[{{x_{p}},{y_{p}},{z_{p}}}\right]^{\,t}}$,
scale $\sigma_{p}$, and orientation angles $\alpha_{p}$ and $\beta_{p}$. To
ensure rotation invariance of the descriptor, the $r$-points $p_{i}$, $i\in
1,2,{\cdots},r$, with coordinates
${\bf{p}_{i}}\buildrel\Delta\over{=}{\left[{{x_{i}},{y_{i}},{z_{i}}}\right]^{\,t}}$
around the keypoint of interest $p$ are at first rotated in the dominant
orientation of $p$ using the transformation

$\displaystyle{\bf{p}}_{\bf{i}}^{\bf{{}^{\prime}}}=\left[{\begin{array}[]{*{20}{c}}{\cos{\alpha_{p}}\cos{\beta_{p}}}&{-\sin{\alpha_{p}}}&{-\cos{\alpha_{p}}\sin{\beta_{p}}}\\\
{\sin{\alpha_{p}}\cos{\beta_{p}}}&{\cos{\alpha_{p}}}&{-\sin{\alpha_{p}}\sin{\beta_{p}}}\\\
{\sin{\beta_{p}}}&0&{\cos{\beta_{p}}}\end{array}}\right]\cdot{{\bf{p}}_{\bf{i}}}.$
(4)

Then, the normal vector to the neighborhood $\bf{n}$ is calculated at the
current key point $\bf{p}$. For each of these rotated points
$\bf{p_{i}}^{\bf{{}^{\prime}}}$ in the $r$-points neighborhood of $\bf{p}$,
the vector ${{\bf{p}}^{\bf{{}^{\prime}}}}{\bf{p}}_{\bf{i}}^{\bf{{}^{\prime}}}$
is computed, where ${{\bf{p}}^{\bf{{}^{\prime}}}}$ is the keypoint $\bf{p}$
rotated according to the transformation (4). Then, we calculate the angle
$\delta$ as

$\halign to=0.0pt{\relax\global\advance\@IEEEeqncolcnt by
1\relax\begingroup\csname
@IEEEeqnarraycolPREc\endcsname#\relax\relax\relax\relax\relax\relax\relax\relax\relax\relax\csname
@IEEEeqnarraycolPOSTc\endcsname\relax\relax\relax\relax\relax\relax\relax\relax\relax\relax\endgroup&\global\advance\@IEEEeqncolcnt
by 1\relax\bgroup#\egroup&\global\advance\@IEEEeqncolcnt by 1\relax\hbox
to\z@\bgroup\hss#\egroup\cr\hfil$\displaystyle\delta= {\cos^{ - 1}}\left(
{\frac{{{{\bf{p}}^{\bf{'}}}{\bf{p}}_{\bf{i}}^{\bf{'}} \cdot{\bf{n}}}}{{\left|
{{{\bf{p}}^{\bf{'}}}{\bf{p}}_{\bf{i}}^{\bf{'}}} \right| \cdot\left| {\bf{n}}
\right|}}} \right). $\hfil&{\vrule
width=0.0pt,height=0.0pt,depth=0.0pt}&0.0pt{\hss(5)}\cr}$

Thus, every keypoint along with its neighborhood is represented by a 4-tuple
$\left(m,\theta,\phi,\delta\right)$. The $\left(\theta,\phi,\delta\right)$
space is then divided into $45$ deg bins, and these bins are populated by
adding up the corresponding values with a Gaussian weighting of
${e^{{{-2d}\mathord{\left/{\vphantom{{-2d}{{R_{\max}}}}}\right.\kern-1.2pt}{{R_{\max}}}}}}$.
The azimuth angle $\theta\in\left[0,360\right]$ deg is divided into 8 bins of
45 deg; the elevation angle $\phi\in\left[{{-90},90}\right]$ deg is divided
into 4 bins of 45 deg; and $\delta\in\left[0,180\right]$ deg which is also
split into 4 bins of 45 deg. Thus, the size of the 3D SIFT descriptors in our
case is $4\times 4\times 8=128$. Each 3D SIFT descriptor is normalized to
unity.

####  3.2.2 SHOT

Motivated by the SIFT descriptor, Signatures of Histograms of Orientations
(SHOT), was proposed to take the advantages of both signature as well as
histogram based methods simultaneously [8]. In SHOT, a unique and repeatable
local reference frame is computed for each keypoint using an eigenvalue
decomposition of the modified neighborhood covariance matrix $\bf{C}$. The
sample points $\bf{q_{i}}$ that lie in the support region of radius $\bf{r}$
are weighted based on their distances from the point $\bf{q}$ under
consideration, as shown below

$\halign to=0.0pt{\relax\global\advance\@IEEEeqncolcnt by
1\relax\begingroup\csname
@IEEEeqnarraycolPREc\endcsname#\relax\relax\relax\relax\relax\relax\relax\relax\relax\relax\csname
@IEEEeqnarraycolPOSTc\endcsname\relax\relax\relax\relax\relax\relax\relax\relax\relax\relax\endgroup&\global\advance\@IEEEeqncolcnt
by 1\relax\bgroup#\egroup&\global\advance\@IEEEeqncolcnt by 1\relax\hbox
to\z@\bgroup\hss#\egroup\cr\hfil$\displaystyle{\bf{C}} =
\frac{1}{{\sum\nolimits_{i:{d_{i}} \leq{\bf{r}}} {\left( {{\bf{r}} - {d_{i}}}
\right)} }}\sum\limits_{i:{d_{i}} \leq{\bf{r}}} {\left( {{\bf{r}} - {d_{i}}}
\right)} \left( {{{\bf{q}}_{\bf{i}}}{\bf{ - q}}} \right){\left(
{{{\bf{q}}_{\bf{i}}}{\bf{ - q}}} \right)^{t}},\hphantom{(6)} $\hfil&{\vrule
width=0.0pt,height=0.0pt,depth=0.0pt}&0.0pt{\hss(6)}\cr}$

where, ${d_{i}}={\left\|{{q_{i}}-q}\right\|_{2}}$. Using the reference frame,
a spherical grid is built around the input point. This grid divides supporting
points into grid cells. At each grid sector, a weighted cosine of the relative
normal angle is calculated, and the result is binned into a local histogram
for that cell. SHOT combines all local histograms into one descriptor of
length 352. In the last stage, the descriptor is normalized to unity.

Figure 2: Representation of the performance evaluation of 3D keypoint
detectors and descriptors

###  3.3 Fisher Vector (FV) and Vector of Linearly Aggregated Descriptors
(VLAD) Encoding

Once we have extracted 3D features from point cloud of the plants, the next
step is to classify them into different categories. Since, the number of
keypoints for different point cloud are different, we need to quantize the 3D
descriptors. In this paper, we have experimented with both FV [17] and VLAD
[18] for encoding the 3D features.
A FV is a statistics representing the distribution of a set of vectors, in our
case, a set of local surface descriptors. The FV encoding employs Gaussian
Mixture Model (GMM) to the local descriptors to construct a dictionary of $k$
words. Once the dictionary is created, the FV encodes the gradients of the
log-likelihood of the features under the GMM, with respect to the GMM
parameters and compare $N$ descriptors to $k$ visual words.
VLAD is another feature encoding and pooling method used to encode a set of
local feature descriptors using a dictionary built using k-means clustering,
unlike the FV, that used GMM.
The FV and VLAD representations offer many advantages compared with other
methods. They can be computed from smaller vocabularies, and therefore require
less computation. Furthermore, FV and VLAD perform well even with simple
linear classifiers as they don’t come up with curvy and complex boundaries
between classes. This is a significant advantage as linear classifiers are
very efficient to learn and easy to evaluate.
In this work, we directly quantize the features using FV and VLAD encoding.
Since the size of the generated codebook is small, we use Support Vector
Machine (SVM), a linear classifier, for training and classification.

###  3.4 SVM

A SVM is a supervised learning model, formally defined by a separating
hyperplane. Given the set of labeled training data (supervised learning), the
algorithm creates an optimal hyperplane that categorizes new test data, making
it a non-probabilistic binary linear classifier. There are two common
approaches for extending SVM from a binary classifier to a multi-class
classifier: one-versus-one and one-vs-all. In this paper, we have used the one
versus all approach.

##  4 Experimental Evaluation And Discussion

In this section we provide the quantitative as well as qualitative performance
of different keypoint detector and descriptor combinations in context of plant
health classification.

###  4.1 Dataset Used

In this work we used the dataset containing 3D scans of plant shoot
architectures provided by Salk Institute of Biological Science [19], as there
are not many publicly available dataset of 3D point clouds of plants prepared
to study the growth and health of plants under different environmental
conditions. This dataset contains $559$ 3D plant shoot architectures from 4
species (Arabidopsis, tomato, tobacco, and sorghum) scanned under multiple
conditions (ambient light (control), high-heat, high-light, shade, drought).
Each plant was scanned every 1 or 2 days through roughly 20 days of
development.
In our work we have used only sorghum, tobacco and tomato plants in three
different conditions of control, shade and heat because this data has the same
categories and the same number of replicates which makes it easier to compare
the results.

###  4.2 Experiments and Results

All our experiments were performed using MATLAB and/or the point cloud library
(PCL) (See Table 1). Each plant had three replicates in each category, viz,
control, heat and shade. All the point clouds across all the replicates for
each plant were divided in to categories based on the growth stage, and also
on the growth condition. For classification based on the growth condition, the
point clouds for each plant were grouped in to three categories: control, heat
and shade across all the replicates. For growth stage, we grouped the point
clouds for each plant in to three categories: stage 1, stage 2, and stage 3.
For this, the point clouds across all the growth conditions were grouped into
the three stages, with stage three representing the most mature plants. For
each point cloud in the dataset, we extract the keypoints using three most
common 3D keypoint detectors, viz, Harris 3D, ISS and 3D SIFT, giving us three
sets of keypoints. We then use SHOT and 3D SIFT descriptors to describe the
local surface around these sets of keypoints, giving us a total of six
detector-descriptor combinations for each point cloud. This is followed by the
quantization of these descriptors using FV and VLAD encoding. In this work, We
have performed the evaluation for the FV and VLAD encoded descriptors
separately. Once the descriptors have been extracted and quantized, SVMs are
trained to learn plant categories and to perform the plant classification for
both growth stage and growth condition. The involved steps are shown in Fig.
1. For the classification step, we use $80\%$ of the feature vectors for
training the SVM. Linear SVM classifier with one-versus-all approach is
applied for its efficiency and simplicity.

The performance evaluation of a detector - descriptor pair is done in terms of
classification accuracy. The accuracy is calculated as the percentage of
number of correct classifications to the total number of test inputs for
respective categories.

Table 1:  Implementation Platforms of the Used Techniques Technique | Platform
---|---
Harris3D Detector | PCL
ISS Detector | PCL
SIFT Detector | PCL
3D SIFT Descriptor | MATLAB
SHOT Descriptor | PCL
VLAD Encoder | MATLAB
Fisher Vector | MATLAB
Table 2: Classification Accuracy ($\%$) for Sorghum (both growth stage and growth conditions) Pair | Accuracy (FV) | Accuracy (VLAD)
---|---|---
| Conditon | Stage | Condition | Stage
Harris-SHOT | 62.96 | 59.26 | 66.67 | 66.67
ISS-SHOT | 85.18 | 77.78 | 85.18 | 81.48
SIFT-SHOT | 74.07 | 74.07 | 70.37 | 70.37
Harris-SIFT | 59.26 | 59.26 | 74.07 | 59.26
ISS-SIFT | 81.48 | 77.78 | 85.18 | 70.37
SIFT-SIFT | 85.18 | 62.96 | 88.89 | 62.96
Table 3: Classification Accuracy ($\%$) for Tobacco (both growth stage and growth conditions) Pair | Accuracy (FV) | Accuracy (VLAD)
---|---|---
| Conditon | Stage | Condition | Stage
Harris-SHOT | 55.56 | 59.26 | 62.96 | 55.56
ISS-SHOT | 81.48 | 85.18 | 81.48 | 77.78
SIFT-SHOT | 81.48 | 71.78 | 85.18 | 77.78
Harris-SIFT | 66.67 | 62.96 | 70.37 | 62.96
ISS-SIFT | 81.48 | 70.37 | 74.07 | 59.26
SIFT-SIFT | 85.18 | 62.96 | 77.78 | 62.96
Table 4: Classification Accuracy ($\%$) for Tomato (both growth stage and growth conditions) Pair | Accuracy (FV) | Accuracy (VLAD)
---|---|---
| Conditon | Stage | Condition | Stage
Harris-SHOT | 70.37 | 66.67 | 74.07 | 66.67
ISS-SHOT | 92.60 | 77.78 | 85.18 | 81.48
SIFT-SHOT | 88.89 | 81.48 | 88.89 | 77.78
Harris-SIFT | 81.48 | 74.07 | 70.37 | 66.67
ISS-SIFT | 96.30 | 70.37 | 88.89 | 70.37
SIFT-SIFT | 96.30 | 74.07 | 85.18 | 77.78
Table 5: Classification Accuracy ($\%$) Averaged Over All Three Plants (both growth stage and growth conditions) Pair | Accuracy (FV) | Accuracy (VLAD)
---|---|---
| Conditon | Stage | Condition | Stage
Harris-SHOT | 62.96 | 61.723 | 67.9 | 62.96
ISS-SHOT | 86.42 | 80.25 | 83.95 | 80.25
SIFT-SHOT | 81.48 | 75.78 | 81.48 | 75.30
Harris-SIFT | 69.14 | 65.43 | 71.60 | 62.96
ISS-SIFT | 86.42 | 72.84 | 82.71 | 66.67
SIFT-SIFT | 88.87 | 66.66 | 83.95 | 67.90

Tables 2, 3 and 4 show the classification accuracies of the six detector-
descriptor (in the same order) pairs with FV as well as VLAD encoding for both
condition-wise and growth-wise classification for sorghum, tobacco and tomato,
respectively. The detector-descriptor pair giving the worst and the best
results in each column is highlighted in red and blue, respectively. One
important observation regarding the physical structure of the plants is that
they vary significantly in the shape of the leaves and their curvature
variations. Sorghum has thin leaves and high curvature variations, tobacco has
broad leaves and low curvature variations, and tomato, also, has broad leaves
(smaller than tobacco) but has a relatively higher curvature variations
compared to tobacco. These differences in the shape affect the detection and
the description processes, and in turn affect the classification accuracies.
Table 2 shows that for sorghum, the Harris keypoints give poor results with
both SHOT and SIFT descriptors, while the ISS keypoints perform well with both
the descriptors. The SIFT descriptors perform very well with the SIFT
keypoints in case of FV as well as VLAD encoding. Similar trends are observed
for tobacco and tomato, as seen from Tables 3 and 4, with both the descriptors
performing well with the ISS keypoints while performing poorly with Harris
keypoints.
Table 5 shows the average classification accuracy across all the three plants.
Here ISS-SHOT and SIFT-SIFT outperform all other combinations. Also, one can
note that the classification accuracies are on an average better with FV
encoding as compared to VLAD encoding. One trend that is common across all the
tables is that the classification accuracies for growth stage classification
are lower than that for growth condition classification. This trend can be
attributed to the fact that the similarity between plants growing in the same
environmental condition observed over successive days will be usually higher
than the similarity between plants that are growing under different
environmental conditions.

##  5 CONCLUSIONS

In this paper, we compared the performance of different keypoint detector-
descriptors combinations in context of plant growth classification and it’s
growth condition using 3D point clouds of the plants. The performance was
evaluated in terms of the classification accuracy and the results were
presented in terms of accuracy tables. Experimental results show that the ISS
keypoints perform much better than the Harris keypoints with both SHOT and
SIFT descriptors. Furthermore, the ISS-SHOT combination gives better results
in the high curvature cases while the SIFT-SIFT combination performs well in
all the cases generally. Also, in general, FV performs better than VLAD
encoding. In future works, we intend to prepare our own dataset of 3D point
clouds of crops such as rice and wheat and will try to develop and analyze
novel methods and detector-descriptor combinations tailored for plant
phenomics application. We will also explore the use of neural networks and
other machine learning methods for applications related to plant phenomics.

## References

  * [1] A. J. Challinor, J. Watson, D. Lobell, S. Howden, D. Smith, and N. Chhetri, “A meta-analysis of crop yield under climate change and adaptation,” _Nature Climate Change_ , vol. 4, no. 4, p. 287, 2014.
  * [2] T. T. Santos, L. V. Koenigkan, J. G. A. Barbedo, and G. C. Rodrigues, “3d plant modeling: localization, mapping and segmentation for plant phenotyping using a single hand-held camera,” in _European Conference on Computer Vision_. Springer, 2014, pp. 247–263.
  * [3] R. Panwar, K. Goyal, N. Pandey, and N. Khanna, “Imaging system for classification of local flora of uttarakhand region,” in _2014 International Conference on Power, Control and Embedded Systems (ICPCES)_. IEEE, 2014, pp. 1–6.
  * [4] S. Srivastava, S. Bhugra, B. Lall, and S. Chaudhury, “Drought stress classification using 3d plant models,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2046–2054.
  * [5] C. Harris and M. Stephens, “A combined corner and edge detector in alvey vision conference,” _Manchester, UK_ , 1988.
  * [6] D. G. Lowe, “Object recognition from local scale-invariant features,” in _Computer vision, 1999. The proceedings of the seventh IEEE international conference on_ , vol. 2. Ieee, 1999, pp. 1150–1157.
  * [7] D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung, “On visual similarity based 3d model retrieval,” in _Computer graphics forum_ , vol. 22, no. 3. Wiley Online Library, 2003, pp. 223–232.
  * [8] S. Salti, F. Tombari, and L. Di Stefano, “Shot: Unique signatures of histograms for surface and texture description,” _Computer Vision and Image Understanding_ , vol. 125, pp. 251–264, 2014.
  * [9] S. Srivastava and B. Lall, “Deeppoint3d: Learning discriminative local descriptors using deep metric learning on 3d point clouds,” _Pattern Recognition Letters_ , 2019.
  * [10] S. Delagrange and P. Rochon, “Reconstruction and analysis of a deciduous sapling using digital photographs or terrestrial-lidar technology,” _Annals of botany_ , vol. 108, no. 6, pp. 991–1000, 2011.
  * [11] G. Alenyà, B. Dellen, and C. Torras, “3d modelling of leaves from color and tof data for robotized plant measuring,” in _Robotics and Automation (ICRA), 2011 IEEE International Conference on_. IEEE, 2011, pp. 3408–3414.
  * [12] W. Fang, H. Feng, W. Yang, L. Duan, G. Chen, L. Xiong, and Q. Liu, “High-throughput volumetric reconstruction for 3d wheat plant architecture studies,” _Journal of Innovative Optical Health Sciences_ , vol. 9, no. 05, p. 1650037, 2016.
  * [13] A. Paproki, J. Fripp, O. Salvado, X. Sirault, S. Berry, and R. Furbank, “Automated 3d segmentation and analysis of cotton plants,” in _Digital Image Computing Techniques and Applications (DICTA), 2011 International Conference on_. IEEE, 2011, pp. 555–560.
  * [14] F. Tombari, S. Salti, and L. Di Stefano, “Performance evaluation of 3d keypoint detectors,” _International Journal of Computer Vision_ , vol. 102, no. 1-3, pp. 198–220, 2013.
  * [15] R. Hänsch, T. Weber, and O. Hellwich, “Comparison of 3d interest point detectors and descriptors for point cloud fusion,” _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , vol. 2, no. 3, p. 57, 2014.
  * [16] Y. Guo, M. Bennamoun, F. Sohel, M. Lu, J. Wan, and N. M. Kwok, “A comprehensive performance evaluation of 3d local feature descriptors,” _International Journal of Computer Vision_ , vol. 116, no. 1, pp. 66–89, 2016\.
  * [17] F. Perronnin and C. Dance, “Fisher kernels on visual vocabularies for image categorization,” in _Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on_. IEEE, 2007, pp. 1–8.
  * [18] H. Jégou, M. Douze, C. Schmid, and P. Pérez, “Aggregating local descriptors into a compact image representation,” in _Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on_. IEEE, 2010, pp. 3304–3311.
  * [19] S. Navlakha, “3d scans of plant shoot architectures,” Jul 2017. [Online]. Available: https://data.mendeley.com/datasets/9k7zctdyhs/1


# From Points to Multi-Object 3D Reconstruction

Francis Engelmann1,∗ Konstantinos Rematas2 Bastian Leibe1 Vittorio Ferrari2

1 RWTH Aachen University 2 Google Research

###### Abstract

We propose a method to detect and reconstruct multiple 3D objects from a
single RGB image. The key idea is to optimize for detection, alignment and
shape jointly over all objects in the RGB image, while focusing on realistic
and physically plausible reconstructions. To this end, we propose a keypoint
detector that localizes objects as center points and directly predicts all
object properties, including 9-DoF bounding boxes and 3D shapes – all in a
single forward pass. The proposed method formulates 3D shape reconstruction as
a shape selection problem, it selects among exemplar shapes from a given
database. This makes it agnostic to shape representations, which enables a
lightweight reconstruction of realistic and visually-pleasing shapes based on
CAD-models, while the training objective is formulated around point clouds and
voxel representations. A collision-loss promotes non-intersecting objects,
further increasing the reconstruction realism. Given the RGB image, the
presented approach performs lightweight reconstruction in a single-stage, it
is real-time capable, fully differentiable and end-to-end trainable. Our
experiments compare multiple approaches for 9-DoF bounding box estimation,
evaluate the novel shape-selection mechanism and compare to recent methods in
terms of 3D bounding box estimation and 3D shape reconstruction quality.

Figure 1:  We propose a single-stage model for realistic multi-object 3D
reconstruction from a single RGB image. The model detects object center-points
and performs reconstruction by jointly estimating 9-DoF bounding boxes and
representation-agnostic 3D shape exemplars.  ††footnotetext: ∗ Work performed
during internship at Google Research, Zurich. Figure 2: Overview of the
proposed approach. Given a single RGB image, our model detects object centers
as keypoints in a heatmap $Y$. The network directly predicts shape exemplars
$\mathbf{z}$ and 9-DoF bounding boxes jointly for all objects in the scene.
The collision loss $\mathcal{L}_{coll}$ favors non-intersecting
reconstructions. Our method predicts lightweight, realistic and physically
plausible reconstructions in a single pass.

##  1 Introduction

Extracting 3D information from a single image has multiple applications in
computer vision, robotics and specifically on mobile AR/VR devices. Indeed,
the field has gained great momentum in the computer vision community [9, 21,
29, 34, 43]. 3D information can come in many forms: 3D bounding boxes, point
clouds, meshes, voxels or distance fields. The choice of the representation
often depends on the task at hand. In this paper, we aim to extract all the
above information in an efficient and scalable way, all from just a single
view and in a single pass.

Recent methods [9, 21] perform multi-object reconstruction by independently
processing detections from state-of-the-art object detectors [13, 22] or
jointly predict multiple objects in a dense voxel grid [34], which can be
computationally expensive due to scalability issues. Instead, inspired by
CenterNet [47], a framework for accurate and efficient 2D object detection, we
propose to use a keypoint detector to localize objects as sparse center-points
and directly predict 9-DoF bounding boxes and shapes jointly for all objects
in the scene. The CenterNet architecture is modular and can easily be extended
to solve varying tasks such as 2D detection, 3D detection, human body pose
estimation and tracking [44, 46]. In this paper, we argue for a complete and
coherent 3D reconstruction of multiple objects using CenterNet where each
pixel votes for a class label, a 3D bounding box, and a 3D shape exemplary to
place objects into the world coordinate frame.

Another key aspect relates to the question of the best shape representation.
While numerous representations have been proposed, Signed Distance Functions
(SDF) [31], meshes [9, 11], voxel grids [34], point clouds [7, 17], and even
hybrid approaches [37], they all have task-dependent advantages and
disadvantages. In this work, we propose a representation-independent shape
selection mechanism. That is, shape exemplars are selected from a given shape
database that can implement different (or multiple) representations. The most
convenient representation is chosen depending on the specific task, be it for
defining objective functions or for visualization purposes Fig. 1.

Additionally, we take extra provisions for a realistic and physically
plausible reconstruction. In particular, objects should be properly placed in
the world frame and should not intersect with each other. Inspired by recent
methods on human body pose estimation in 3D scenes [12, 18, 45], we add a
collision loss that supports plausible reconstructions such that reconstructed
objects do not intersect. To summarize, given a RGB image, our single-stage
method performs lightweight reconstruction, it is real-time capable, fully
differentiable and end-to-end trainable. In our experiments, we compare
different 9-DoF bounding box formulations, we evaluate our shape selection
mechanism using soft labels and compare with the current state-of-the-art
CoReNet [34].

#### Contributions.

Our key contributions are:

  * •

We propose a method for multi-object 3D reconstruction that extends the
CenterNet [47] framework to perform fully holistic 3D scene reconstruction in
a single-stage network and from a single RGB image.

  * •

We present a shape-selection mechanism to perform 3D object reconstruction,
where we reformulate the 1-of-K classification task using soft target labels
based on geometric similarities between exemplar 3D shapes: this significantly
improves over hard-labels as used in previous baselines [39].

  * •

We obtain physically plausible reconstructions by leveraging a collision loss
that encourages non-intersecting reconstructions. Further, the use of CAD
models guarantees valid and realistic shapes.

  * •

Our approach is agnostic to different shape representations. Since we
formulate the shape reconstruction problem as selecting a shape exemplar (,
index in a precomputed database of shapes), we can choose from any
representation given the estimated shape exemplar.

##  2 Related Work

#### 3D from a single image.

Single image 3D reconstruction has seen tremendous progress over the last
years, with various shape representations being examined. Works like [6, 8,
14, 36, 42, 43] operate on voxel grids, a representation that fits very well
with convolutional neural networks. Other methods output point clouds [7, 17],
taking advantage of their compactness. Another line of work [4, 11, 19, 26,
41] outputs meshes, a powerful representation that provides neighborhood
structure to the 3D shape. Recently, implicit representations [5, 27, 30, 32]
have gained popularity for their ability to represent fine details at
arbitrary resolutions. An alternative to the 3D shape regression is the work
of [39] that poses the 3D reconstruction as a classification/retrieval
problem. However, all of these methods focus on the single object case: the
image contains a single object to be reconstructed often on a white
background. By having every pixel predict a 3D bounding box, a shape index
similar to [39], and the 9-DoF, we are able to handle arbitrary number of
objects in the scene in a single forward pass.

#### Multi-object 3D reconstruction.

Recently, there has been progress also in the case of multi-object 3D
reconstruction. Im2CAD [16] performs object detection and room layout
estimation in an input image, and then retrieves 3D shapes from a database and
aligns them to match the detections. However, it involves a secondary non-
differentiable optimization step, that renders and matches the estimations
with the input image. 3D-RCNN [20] estimates the 3D shape of each object
instance in an image through a render-and-compare learning approach, where the
shape is represented as a linear basis from a dataset of 3D models. This shape
representation though is mostly accurate for classes with low intra-class
variability such as cars and humans. Given an image and a set of object
proposals, Tulsiani et al.. [40] decompose the underlying 3D scene into a room
layout, a set of voxels grids for every object, together with their
rotation/translation/scaling parameters. In the work of [15], the 3D scene is
represented as a graph that is being optimized so the configuration of objects
and room layout matches the semantic and geometric properties of the input
image. Mesh R-CNN [9] can be seen as an extension of Mask-RCNN [13] to
estimate 3D meshes for every object instance in an image, but without
reasoning about their scale/depth ambiguity. More recently, Total 3D
Understanding [29] proposes a system that estimates the room layout, 3D object
bounding boxes and a mesh for every object as well. However, the 3D estimates
depend on the initial 2D bounding boxes. Our work is similar to the recent
Mask2CAD [21] which also predicts object centers and retrieves CAD models for
reconstruction. However, they estimate the 3D placement of the retrieved
shapes individually for each object and do not model collision avoidance,
while our approach performs a joint estimation of all objects and avoids
intersections between reconstructed objects.

One common element of the aforementioned works is that they are based on two-
stage architectures where the objects are first detected and then a second
stage individually estimates their shapes. In contrast, we propose a single-
stage method which scales well with the number of objects and does not involve
further post-processing mechanisms.

CoReNet [34] performs dense shape prediction in a fixed $128^{3}$ voxel grid,
which does not scale with the size of the reconstructed world. In our case, we
can allocate $128^{3}$ voxels to each individual object, thus the total voxel
count scales with the number of objects in a scene. Moreover, CoReNet bakes
all scene information into one model during training (number of objects, class
combinations). Instead, our approach is more modular and can detect and
reconstruct a variable number of objects, as well as new combinations of
classes not seen during training. Our approach predicts both a 9-DoF oriented
bounding box and shape. Additionally, our shape representation is independent
of the actual representation: we can predict signed distance functions, point
clouds, occupancy grids and meshes, which naturally leads to realistic scene
reconstructions, while CoReNet reconstructions often show detached object
parts or holes in the reconstructed shapes, especially in multi-object scenes.

Figure 3: Object detection as keypoint detection. _Left:_ Predicted heatmaps
$\hat{Y}$ visualizing the per-pixel probability for being an object center.
The heatmap $\hat{Y}_{c}$ of each class $c$ is shown in a different color. The
peaks of the distributions are shown as white circles $\circ$, they correspond
to the detected object centers $\hat{\mathbf{p}}$ from which the object
properties are predicted. _Right:_ Predicted object properties. We show the
estimated 9-DoF bounding boxes and the 3D shapes using the point cloud
representation.

##  3 Method Overview

We now present each module and the corresponding losses of our full model as
shown in Fig. 2. We formulate object detection as a keypoint detection problem
similar to CenterNet [47], where each object is represented by its center
point in the 2D image (Sec. 3.1). From the detected center points, we directly
estimate realistic shapes (Sec. 3.2) and oriented 3D object bounding boxes
(Sec. 3.3) To further promote reconstructions that are physically plausible,
we propose a collision loss that penalizes the intersection of multiple
objects (Sec. 3.4).

###  3.1 Object Detection as Keypoint Detection

The first part of our method is a keypoint detector that follows the setup of
CenterNet [47]. Given a single RGB image $I\in\mathbb{R}^{W\times H\times 3}$,
the detector localizes keypoints (here: object centers) by predicting class-
specific heatmaps $\hat{Y}\in[0,1]^{\frac{W}{R}\times\frac{H}{R}\times C}$
(Fig. 3, _left_) where $C$ is the number of object classes and $R$ = $4$ is a
down-sampling factor. The detected center points
$\\{\hat{\mathbf{p}}_{i}\in\mathbb{R}^{2}\\}$ (shown as $\circ$ in Fig. 3)
correspond to the local maxima in the predicted heatmaps $\hat{Y}$. They are
obtained using non-maximum-suppression, which is implemented as a $3\times 3$
max pooling. We associate a confidence score
$s_{i}=\hat{Y}_{\hat{\mathbf{p}}_{i}}$ to each detected keypoint
$\hat{\mathbf{p}}$. The feature backbone – which takes the input image $I$ and
generates the output heatmaps $\hat{Y}$ – is implemented as a stacked
hourglass model [28].

During training, we follow [23, 47] and generate the target heatmaps $Y$ by
splatting the ground truth center points $\mathbf{p}_{i}$ using Gaussian
kernels $\mathcal{N}(\mathbf{p}_{i},\sigma_{i})$ with $\sigma_{i}$ depending
on the projected size of the object $i$. Training the keypoint detector relies
on the focal loss [25] and is computed over all pixels $(x,y)$ and classes
$c\in\\{1,\ldots,C\\}$ in the heatmaps:

$\mathcal{L}_{key}\text{\,=\,}\frac{-1}{N}\sum_{xyc}\begin{cases}(1\text{\,-\,}\hat{Y}_{xyc})^{\alpha}\cdot\text{log}(\hat{Y}_{xyc})&\text{if
$Y_{xyc}$\,=\,1}\\\
(1\text{\,-\,}Y_{xyc})^{\beta}\cdot(\hat{Y}_{xyc})^{\alpha}\cdot\text{log}(1\text{\,-\,}\hat{Y}_{xyc})&\text{else}\\\
\end{cases}$ (1)

where $N$ is the number of ground truth objects, $\alpha$ = 2 and $\beta$ = 4
are the hyper-parameters of the focal loss. After detecting the object
instances as center points, the network jointly selects 3D shapes (Sec. 3.2)
and estimates 3D bounding boxes (Sec. 3.3) for each object in the scene.

Figure 4: Shape selection. We compare one-hot encoding (hard labels, _left_)
for supervising the shape selection problem with soft labels (_right_) which
allow for multiple shape predictions at the same time and are based on
geometric similarity, specifically the Euclidean distance between SDF shape
representations.

###  3.2 Shape Selection

Instead of directly reconstructing shape representations such as meshes, voxel
grids or point clouds [7, 11, 34], our method operates indirectly, by
selecting shape exemplars. More precisely, the network is trained to select
for each detection one shape exemplar $z$ among a set of $K$ shape exemplars
from a given shape database. This choice is motivated by our goal to
reconstruct realistic scenes, since it guarantees valid shapes from the object
database unlike recent reconstruction methods which can produce incomplete,
noisy or over-smoothed reconstructions [2]. Similarly, the recent work of
Tatarchenko et al. [39] concludes that current methods for single-view 3D
reconstruction primarily work because of recognizing the type of shape
depicted in the image, rather than truly recovering the geometric details
unique to that particular instance.

To reiterate, in this work, the shape estimation problem is formulated as a
shape selection problem which chooses one shape exemplar $\hat{z}$ from a
given shape database $\mathcal{Z}$ of $K$ shape exemplars. After predicting an
exemplar $\hat{z}$, an explicit shape representation $X$ (voxel-grid, point
cloud, CAD model ) can be chosen freely from the precomputed databases
$\mathcal{Z}_{X}$ (described next) depending on the task or loss function at
hand. As such, the presented model is agnostic towards any particular shape
representation.

#### Building the shape database $\mathcal{Z}$.

The presented shape database is a set of representative shape exemplars
selected from a given set of CAD models. Once our shape database is built, the
full set of the original CAD models is no longer required. We now describe how
those exemplary shapes are selected. First, the CAD models are transformed
into a canonical orientation, position and scale. Specifically, all models are
facing down the negative Z-axis, the centroids are translated to the origin,
and we apply anisotropic scaling such that the models fit into the unit cube.
Then, for each object $i$, we compute the signed distance function (SDF)
representation $\phi^{i}$ of the corresponding CAD model. After
discretization, downsampling to $32^{3}$ grids and flattening to vectors, we
cluster the objects using k-Means++ [1] with $k$ = $50$, for each object class
separately. The total number $K$ of shape exemplars in the database
$\mathcal{Z}$ is $K=k\cdot C$ where $C$ is the number of object types (chairs,
bottle, ). The objects appearing in the training images are already annotated
by their corresponding CAD model. Hence, we can re-label each object with
their nearest shape exemplar $z^{k}$. Additionally, the shape database can be
extended to store explicit shape representations such as SDFs
$\mathcal{Z}_{\phi}$ = $\\{\phi^{k}\\}_{k=1}^{K}$, point clouds
$\mathcal{Z}_{\mathcal{P}}$ = $\\{\mathcal{P}^{k}\\}_{k=1}^{K}$ or CAD models
$\mathcal{Z}_{\text{CAD}}$ = $\\{\text{CAD}^{k}\\}_{k=1}^{K}$. In each case,
the stored representation corresponds to the model that is closest to the
cluster center under the clustering metric ($L_{2}$ distance over $\phi$).

#### Training the shape selection network module.

One straightforward approach consists in training a 1-of-$K$ classifier.
Specifically, for each object $i$ in the input image, the network predicts a
vector $\hat{{\mathbf{z}}}^{i}\in\mathbb{R}^{K}$ scoring it against each of
the $K$ exemplar shapes in the shape database $\mathcal{Z}$. We can then place
a cross-entropy loss $CE(\cdot,\cdot)$ on this output and supervise it with
the ground truth one-hot encoding of the target shape
$\mathbf{z}^{i}\in\\{0,1\\}^{K}$ (Fig. 4, _left_):

$\displaystyle\mathcal{L}^{\prime}_{\mathbf{z}}$
$\displaystyle=\frac{1}{M}\sum_{i=1}^{M}CE\big{(}\mathbf{z}^{i},\sigma{(\hat{\mathbf{z}}^{i})}\big{)}$
(2)
$\displaystyle=-\frac{1}{M}\sum_{i=1}^{M}\sum_{k=1}^{K}{z}^{i}_{k}\cdot\text{log}\big{(}\sigma(\hat{{\mathbf{z}}}^{i})_{k}\big{)}$
(3)

where $M$ is the number of detections in the image, $\sigma$ is the softmax
function (next paragraph, where we use sigmoid $S$ instead), and $z^{i}_{k}$
is the $k$-th entry in vector $\mathbf{z}^{i}$. At test time, the predicted
shape exemplary $\hat{z}^{i}$ is computed as
$\hat{z}^{i}=\text{argmax}_{k}(\hat{\mathbf{z}}^{i})$. This approach
corresponds to the clustering baseline presented by Tatarchenko et al.in [39].

The issue with this approach is that two objects $\\{i,j\\}$ that are
geometrically similar ($\phi^{i}\approx\phi^{j}$) can have disagreeing
supervision signals $\\{\mathbf{z}^{i},\mathbf{z}^{j}\\}$. This can have a
negative impact on the network training, as the network is asked to
simultaneously predict a high value for one of the $K$ database shapes, while
also predicting a low value for another, very similar shape. Instead, we
propose as alternative formulation a soft relaxation of the binary target
labels $\mathbf{z}\in\\{0,1\\}^{K}$ which takes the geometric similarity of
shapes into account. Specifically, we allow to predict multiple shape
exemplars simultaneously and they are no longer mutually exclusive.

Formally, we redefine the target labels $\mathbf{z}$ using a shape similarity
function $d(\cdot,\cdot)$ (Fig. 4, _right_) such that:

$\mathcal{L}_{\mathbf{z}}=-\frac{1}{M}\sum_{i=1}^{M}\sum_{k=1}^{K}d(i,k)\cdot\text{log}\big{(}S(\hat{z}_{k})\big{)}$
(4)

where $S$ is the sigmoid function and

$d(i,k)=[1-\lVert\phi^{i}-\phi^{k}\rVert_{2}]_{+}$ (5)

where $[\,\cdot\,]_{+}=\text{max}(\,\cdot\,,\,0)$ and $\lVert\cdot\rVert_{2}$
is the Euclidean distance between the shape exemplars’ SDFs $\phi^{k}$ in the
shape database $\mathcal{Z_{\phi}}$, and $\phi^{i}$ is the ground truth SDF of
object $i$.

In the following, we will refer to these labels as _soft_ -labels, and when
using the one-hot encoding as _hard_ -labels. In Sec. 4, we show that our
alternative soft formulation is key to improve shape selection. At test time,
we simply select the shape exemplar with the highest output value by the
network. Next, we describe our approach to estimate the 3D bounding boxes,
which are subsequently used to transform the estimated object shapes from
their canonical database pose into the scene coordinate frame.

###  3.3 3D Bounding Box Estimation (9-DoF Poses)

Along with the realistic shape representation, we aim at finding a 9-DoF
bounding box for each object in the input image $I$. We describe now the
estimation of the 9-DoF bounding box parameters, capturing the object pose in
the scene. They include a 3D rotation $\hat{\mathbf{R}}\in SO(3)$, a 3D
translation $\hat{\mathbf{t}}\in\mathbb{R}^{3}$ and a 3D scale
$\hat{\mathbf{s}}\in\mathbb{R}^{3}$. These parameters are used to transform
the estimated object shape from its canonical database pose to the scene
coordinate frame.

In _CenterNet_ , Zhou et al. [47] formulate the rotation estimation as a
combination of classification over quantized bins followed by regression to a
continuous offset. That formulation requires the definition of multiple loss
functions along with carefully tuned loss weights. Instead, we directly
parameterize the object rotation as a 3D rotation matrix $\hat{\mathbf{R}}\in
SO(3)$. Specifically, our network predicts a 9-dimensional output interpreted
as a $3\times 3$ rotation matrix $\mathbf{M}$ with (differentiable) SVD
decomposition [10] $\mathbf{M}$ = $\mathbf{U\Sigma V}^{\top}$. The
corresponding symmetric orthogonal rotation matrix $\hat{\mathbf{R}}$ is then
obtained by projecting $\mathbf{M}$ into $SO(3)$ [24]:

$\hat{\mathbf{R}}=\mathbf{U}\mathbf{\Sigma}^{\prime}\mathbf{V}^{\top},\;\text{where
}\mathbf{\Sigma}^{\prime}=\text{diag}\big{(}[1,1,\text{det}(\mathbf{U}\mathbf{V}^{\top})]\big{)}$
(6)

This formulation is straightforward and can directly be optimized using, , the
Frobenius norm [10]: $\lVert\mathbf{R}-\hat{\mathbf{R}}\rVert_{F}$. The
translation $\hat{\mathbf{t}}\in\mathbb{R}^{3}$ is defined as the vector from
the scene origin to the 3D bounding box centroid, and can be optimized, , with
the _Huber_ loss (smooth-$L_{1}$):
$\lVert\mathbf{t}-\hat{\mathbf{t}}\rVert_{H}$. Instead, we propose to jointly
optimize both the rotation $\hat{\mathbf{R}}$ and the translation
$\hat{\mathbf{t}}$ using the concatenated transformation
$\mathbf{T}=[\mathbf{R}\,|\,\mathbf{t}]$. Specifically, we minimize the
squared Euclidean distance between the point cloud $\mathcal{P}^{i}$ of the
object under the estimated $\hat{\mathbf{T}}$ and ground truth transformation
$\mathbf{T}$. Formally, we have:

$\mathcal{L}_{\mathbf{Rt}}=\sum_{i=1}^{M}\sum_{\mathbf{x}\in\mathcal{P}^{i}}\lVert\mathbf{T}^{i}\,\mathbf{x}-\hat{\mathbf{T}}^{i}\,\mathbf{x}\rVert_{2}^{2}$
(7)

where $M$ is the number of objects in the image, $\mathbf{x}\in\mathbb{R}^{3}$
is a point in the point cloud $\mathcal{P}^{i}$ sampled from the surface of
the ground truth object $i$ in the input image.

Finally, the scale loss $\mathcal{L}_{\mathbf{s}}$ is implemented as the
$\text{L}_{1}$ distance between predicted and ground truth 3D scale averaged
over all objects in the input image. Similar to [47], the neural network
branch that predicts the bounding box parameters is class-agnostic (the same
for all classes $c$) and only receives supervision at the ground truth center
locations. In summary, the loss for the 9-DoF bounding box estimation consists
of two terms: $\lambda_{Rt}\mathcal{L}_{Rt}+\lambda_{s}\mathcal{L}_{s}$.

l0.2cmt0.6cmCADjl3.0cmt3.0cmCADil0.6cmt0.6cm$\widetilde{\phi}^{j}$l3.0cmt2.8cm$\mathcal{P}^{i}$

Figure 5: Visualization of the collision loss. The collision loss penalizes
colliding objects, contributing to an improved realism of the reconstructed
scene. _Left:_ Physically implausible reconstruction of two colliding objects.
_Right:_ The colors represent the SDF values sampled at the point positions
$\mathcal{P}^{i}$ of the cup in the SDF $\widetilde{\phi}^{j}$ of the bottle.
Outside the object the sampled values are zero (blue) and increase with the
distance to the surface (from blue to red).

###  3.4 Collision Loss

Towards our goal of realistic multi-object reconstruction, it is not only
important that the individual objects exhibit realistic shapes, but also that
their poses form a physically plausible spatial configuration in the scene.
One specific concern is that reconstructed objects should not intersect or
collide with each other. However, the model we just presented in practice
often predicts colliding shapes, especially for nearby objects.

As a remedy, we propose to add a collision loss that inflicts a penalty
whenever two or more reconstructed objects collide. In particular, we rely on
the convenient property of our model that it can choose from multiple shape
representations, and use the SDF representation $\phi^{j}$ of an object $j$
and the point cloud $\mathcal{P}^{i}$ of another object $i$ to compute the
point-to-surface distance. Specifically, the SDF reveals $\phi^{j}$ the
distance of a point to the nearest surface of object $j$. It is negative
inside the object and positive outside. Therefore we define
$\widetilde{\phi}=\text{min}(-\phi,0)$ such that the values are positive
inside the object and zero outside. Formally, the collision loss for one
object $i$ with all other objects $j$ is:

$\mathcal{L}_{coll}^{i}=\sum\limits_{\begin{subarray}{c}j=1\\\ i\neq
j\end{subarray}}^{M}\sum_{\mathbf{x}\in\mathcal{P}^{i}}\widetilde{\phi}^{j}(\mathbf{T}^{ij}\mathbf{x})$
(8)

where $M$ is the total number of detections in the scene, $\mathbf{T}^{ij}$ is
the transformation matrix placing the point cloud $\mathcal{P}^{i}$ of object
$i$ into the local coordinate system of object $j$. As we store the SDFs
values as discrete voxel grids, we perform differentiable trilinear
interpolation when sampling $\widetilde{\phi}^{j}$ at the continuous point
positions $\mathbf{T}^{ij}\mathcal{P}^{i}$. Fig. 5 provides a visual
interpretation of the loss. Inside object $j$, the SDF $\widetilde{\phi}^{j}$
is positive and zero outside. Note that the SDF $\phi$ and point clouds
$\mathcal{P}$ can be pre-computed, as the shape reconstruction task is
formulated as an exemplar selection problem in our model, so all possible
output shapes are known beforehand.

The collision loss over all objects in a scene is:

$\mathcal{L}_{coll}=\sum_{i=1}^{M}\rho(\mathcal{L}_{coll}^{i})$ (9)

where $\rho(x)$ = $\frac{x^{2}/2}{1+x^{2}}$ is the robust Geman-McClure loss
[10] compensating for varying point densities among objects.

###  3.5 Training Details

The full model is optimized by minimizing the multi-task loss $\mathcal{L}$
defined using the previously introduced losses:

$\mathcal{L}=\mathcal{L}_{key}+\lambda_{\mathbf{Rt}}\mathcal{L}_{\mathbf{Rt}}+\lambda_{\mathbf{s}}\mathcal{L}_{\mathbf{s}}+\lambda_{\mathbf{z}}\mathcal{L}_{\mathbf{z}}+\lambda_{coll}\mathcal{L}_{coll}$
(10)

where $\lambda$ are weighting coefficients with associated values
$\\{10,\,10,\,0.1,\,1.0\\}$ respectively. One important observation is that
the collision loss can contradict the pose losses
$\mathcal{L}_{Rt},\mathcal{L}_{s}$, especially in the beginning of the
training process when the initial object pose estimates are still quite far
away from the ground truth. Penalizing colliding objects at this stage is not
helpful and even has a negative impact on convergence speed. Therefore, we
enable the collision loss only after 100 epochs; before that we set its weight
$\lambda_{coll}=0$. We train the entire network from scratch and end-to-end
using the Adam optimizer, and a batch size of 32 for 300 epochs on four P100
GPUs. Training the model to convergence takes about 48 hours. After 5 epochs
of warm-up, we use a constant learning rate of $10^{-3}$ and perform cosine-
decay after 200 epochs. We implemented our model in TensorFlow 2. We found
strong data augmentation to be critical for training stability. Specifically,
we perform HSV-color augmentation and random horizontal image flipping (Fig.
6).

Figure 6: Data augmentation examples. Strong data augmentation is essential.
We perform HSV-color augmentation and random horizontal flipping. For
comparison, the top-left image shows an example that is not augmented.

$\CIRCLE$ $\ocircle$ Table $\CIRCLE$ $\ocircle$ Chair $\CIRCLE$ $\ocircle$
Sofa $\CIRCLE$ $\ocircle$ Cup $\CIRCLE$ $\ocircle$ Bowl $\CIRCLE$ $\ocircle$
Bottle

Figure 7: Qualitative results. _Top rows:_ Single RGB input images. _Bottom rows:_ Outputs of the presented method. We show the 9-DoF object bounding boxes and the selected shapes exemplars visualized using the CAD model database $\mathcal{Z}_{\text{CAD}}$. | Abs. 3D IoU per Object Class |  | Abs. 3D IoU |  | Rel. 3D IoU
---|---|---|---|---|---
| bottle | bowl | chair | mug | sofa | table |  | mean | global |  | mean | global
\footnotesize{$1$}⃝ CoReNet $m_{8}$ [34] | 61.0 | 32.2 | 30.2 | 46.8 | 54.4 | 32.4 |  | 43.0 | 49.1 |  | 43.0 | 49.1
\footnotesize{$2$}⃝ CoReNet $m_{9}$ [34] | 61.8 | 36.2 | 30.1 | 48.0 | 52.9 | 34.8 |  | 43.9 | 49.8 |  | 43.9 | 49.8
\footnotesize{$3$}⃝ Points2Objects (Ours) | 63.54 | 30.19 | 18.90 | 41.47 | 44.5 | 19.83 |  | 36.41 | 44.68 |  | 59.49 | 73.01
\footnotesize{$4$}⃝ Points2Objects (Ours, aligned) | 78.23 | 39.86 | 30.61 | 47.26 | 54.92 | 38.69 |  | 48.26 | 51.97 |  | 78.86 | 84.92
\footnotesize{$5$}⃝ Oracle | 86.03 | 56.50 | 42.08 | 66.11 | 66.33 | 50.18 |  | 61.20 | 61.20 |  | 100.0 | 100.0
Table 1: Comparison with CoReNet [34]. Per-class and mean IoU over all classes
and class-agnostic global IoU on $128^{3}$ voxel grid. We show absolute
reconstruction scores (Abs. 3D IoU) and relative scores (Rel. 3D IoU), that
is, relative to the maximum possible scores. For our model, the maximum
possible score is indicated by the ground truth oracle \footnotesize{$5$}⃝.

##  4 Experiments

The quantitative evaluation consists of three parts and each one addresses a
core contribution of this paper: We compare multiple 9-DoF bounding box
estimation mechanisms and report improved scores over the one used in
CenterNet [47], the collision loss reduces the number of collisions which
increases the realism and physical plausibility of the reconstructions, and we
show that our shape selection mechanism using soft-labels improves over hard-
labels as used by Tatarchenko et al. [39]. Finally, we compare our method to
the current state-of-the-art for multi-object reconstruction CoReNet [34].
Qualitative results on synthetic data are shown in Fig. 7 and on real data in
the appendix.

#### Datasets.

We evaluate the reconstruction performance following _CoReNet_ [34], using two
datasets _ShapeNet-pairs_ and _ShapeNet-triplets_. They contain $256$ $\times$
$256$ px2 photorealistic renderings of either pairs or triplets of ShapeNet
[3] models placed on a ground plane with full global illumination using the
PBRT [33] renderer against an environment map background. The scenes are
rendered from a random camera view point (yaw and pitch). Objects are placed
at random locations on the ground plane, with random scale and rotation,
making sure they do not overlap in 3D. This setup is especially well suited to
evaluate the physical plausibility of our multi-object reconstruction
approach.

To build the shape database $\mathcal{Z}$ described in Sec. 3.2, we use
ShapeNet [3] as the correspondences between its CAD models and the objects
rendered in the images are readily available in the CoReNet datasets. We set
$k$ = $50$, with the number of object types $C$ = $6$ (ShapeNet-triplets) we
obtain $K$ = $300$ exemplary shapes in the shape database $\mathcal{Z}$.

#### How should the 3D bounding box be optimized?

In Sec. 3.3, we presented different approaches to estimate the rotation and
translation of 3D bounding boxes which we compare here. Specifically, we
compare the combined loss $\mathcal{L}$ from Eq. 7 with the individual losses
$\mathcal{L}_{\textbf{R}}$ and $\mathcal{L}_{\textbf{t}}$ defined using the
Frobenious norm and the Huber loss, as described in Sec. 3.3.
$\mathcal{L}_{\textbf{M}}$ is equal to $\mathcal{L}_{\textbf{R}}$ but does not
perform the projection into $SO(3)$, such that the resulting matrix is not
guaranteed to be a valid rotation matrix [24]. Finally, we compare to the
rotation parameterization from [47], , the combination of a classification
loss $\mathcal{L}_{\text{bin}\mathbf{R}}$ over quantized bins followed by a
regression loss $\mathcal{L}_{\text{off}\textbf{R}}$ to continuous offsets. We
use the mean average precision (mAP) as 3D object detection metric [35] with
3D IoU threshold 0.25 and 0.5, as originally proposed in [38]. The results of
this experiment are shown in Tab. 2. In summary, the best option is to
directly predict the rotation matrix $\mathbf{R}$ using SVD and optimize it
together with the translation $\mathbf{t}$ using our
$\mathcal{L}_{\mathbf{Rt}}$.

9-DoF Bounding Box 3D mAP: | @ 0.5 | @ 0.25
---|---|---
$\mathcal{L}_{\text{bin}\mathbf{R}}+\mathcal{L}_{\text{off}\mathbf{R}}+\mathcal{L}_{\mathbf{t}}$ (as in [47]) | 43.29 | 74.97
$\mathcal{L}_{\mathbf{M}}+\mathcal{L}_{\mathbf{t}}$ | 44.75 | 76.96
$\mathcal{L}_{\mathbf{R}}+\mathcal{L}_{\mathbf{t}}$ | 46.81 | 77.19
$\mathcal{L}_{\mathbf{Rt}}$ (Eq. 7) | 48.59 | 77.21
Table 2: 3D Bounding Box Optimization. We compare different representations to
estimate the rotation and translation of 3D bounding boxes as described in
Sec. 3.3. The metric is mean average precision (mAP) with IoU thresholds 0.5
and 0.25.

#### How effective is the collision loss?

An important aspect of multiple object reconstruction is physical
plausibility, , reconstructed objects should not intersect. To evaluate the
effectiveness of the collision loss, we measure collisions as the mean
intersecting volume (mIV) between colliding objects and the overall number of
collisions. We report both scores in Tab. 3 on the validation split of
ShapeNet-triplets. The collision loss substantially decreases the intersecting
volume and reduces the number of collisions by 60%.

| mIV | # Collisions
---|---|---
$\mathcal{L}^{\prime}$ | $1168.8$ | $4116$ |  $\curvearrowright$ $-60.5\%$
$\mathcal{L}^{\prime}+\mathcal{L}_{coll}$ | $\mathbf{794}$ | $\mathbf{1627}$
Table 3: Effect of the collision loss. We report the mean intersection volume
(mIV) over all objects and scenes, and the total number of collisions for our
model with and without collision loss.

#### How do soft- and hard-labels affect shape estimation?

In Sec. 3.2, we present two approaches to select shape exemplars from the
shape database $\mathcal{Z}$. The first one optimizes
$\mathcal{L}^{\prime}_{\textbf{z}}$ (Eq. 3) using hard-labels, , one-hot
encoding of target labels $z$ as presented in [39]. The second approach
$\mathcal{L}_{\textbf{z}}$ (Eq. 4) relies on soft-labels taking into
consideration the geometric similarity of object shapes, therefore allowing to
predict multiple plausible shapes instead of forcing the network to make a
hard decision on one particular shape. Using the evaluation methodology from
[34], we measure the shape reconstruction as intersection-over-union (IoU) on
a $128^{3}$ voxel grid (Tab. 4). We report both mean IoU over all classes and
class-agnostic global IoU. Our shape-selection mechanism using soft-labels
significantly improves shape prediction by $+4.1$ mIoU over the hard-labels
baseline [39].

Shape Estimation Abs. 3D IoU: | mean | global
---|---|---
$\mathcal{L}^{\prime}_{\mathbf{z}}$ (Eq. 3) Hard-Labels (as in [39]) | $32.23$ | $40.28$
$\mathcal{L}_{\mathbf{z}}$ (Eq. 4) Soft-Labels (ours) | $\mathbf{36.41}$ | $\mathbf{44.68}$
Table 4: Soft vs. hard labels. Shape reconstruction quality in terms of
intersection-over-union (IoU) on a $128^{3}$ voxel grid.

#### How does the presented method compare to CoReNet?

First, we compare the reconstruction performance of our method with CoReNet
[34]. Given an image, [34] predicts a dense $128^{3}$ voxel grid. Each voxel
is either empty or assigned to an object-class, trained with the focal loss
($m_{8}$) \footnotesize{$1$}⃝ or the IoU loss ($m_{9}$) \footnotesize{$2$}⃝,
see Tab. 1. Our method reaches a higher relative 3D IoU (59.5 43.9) but does
not quite match CoReNet’s absolute 3D IoU (36.4 43.9). The relative score
takes the maximum possible score into account, , as our model is supervised
with clustered shapes (from the shape database $\mathcal{Z}$) it can only be
as good as this supervision. The oracle \footnotesize{$5$}⃝ indicates this
best possible score for our model, using the ground truth 9-DoF bounding box
and the ground truth shapes from $\mathcal{Z}$ used to supervise our model. We
also perform Procrustes alignment\footnotesize{$4$}⃝ to the ground truth to
abstract from 9-DoF estimation errors (48% 36%).

Second, we analyze the generalization capabilities of both models under
varying number of objects and class-type combinations (Tab. 5). We train on
ShapeNet-pairs and evaluate on ShapeNet-triplets, and vice-versa. Our model
generalizes well when trained on triplets and evaluating on pairs (36.41
36.21). Both CoReNet and ours experience performance drops when trained on
pairs and evaluating on triplets, but ours looses less than CoReNet (-10%
-22%).

Method | Train | Test | 3D mIoU
---|---|---|---
CoReNet [34] | triplets | pairs | $-$ |
CoReNet [34] | triplets | triplets | $43.9$ |  $\curvearrowright$ $-22.3\%$
CoReNet [34] | pairs | triplets | $34.1$
Points2Objects (Ours) | triplets | pairs | $36.2$ |
Points2Objects (Ours) | triplets | triplets | $36.4$ |  $\curvearrowright$ $\mathbf{-10.1\%}$
Points2Objects (Ours) | pairs | triplets | $32.7$
Table 5: Generalization to varying object types and cardinality.

##  5 Conclusion

We have presented an end-to-end trainable model for realistic and joint 3D
multi object reconstruction from a single input RGB image. Specifically, we
extend the CenterNet paradigm to coherently predict multiple 3D objects. That
is, objects are first detected as points, then reconstructed by jointly
estimating 9-DoF object bounding boxes and corresponding 3D shape exemplars
from a given shape database. The presented model is agnostic to shape
representations and flexible towards changing the representations in the shape
database. We further aim towards realistic and physically plausible
reconstructed scenes. To that end, the model encourages collision-free
reconstructions and uses CAD models as shape representations to guarantee
valid and realistic object shapes. On a technical level, we propose a
mechanism to perform shape selection using soft labels, and propose to combine
SVD-based rotation and translation estimation as opposed to individual
optimizations.

#### Acknowledgments:

We thank Sergi Caelles, Stefan Popov and Kevis-Kokitsi Maninis for helpful
discussions, the CoReNet models and support with the datasets. Further, we
also thank Jonas Schult and Theodora Kontogianni for helpful feedback on the
paper and contributions to the supplementary video. This work is supported, in
parts, by the ERC Consolidator Grant DeeViSe (ERC-2017-COG-773161).

## Supplementary Material

In this supplementary material, we provide additional details on the inference
phase of our model, as well as runtime information. Further, we show
additional qualitative results of our approach and of CoReNet applied to real-
world images recorded using a consumer mobile device. Finally, we discuss
differences and limitations of both models.

##  Appendix A Details on inference phase

During training, the model heatmaps $\hat{Y}$ are solely supervised at ground
truth pixel locations $\mathbf{p}$, all other pixels are ignored. The
predicted heatmap values $\hat{Y}_{\mathbf{\hat{p}}_{i}}$ can directly be
interpreted as confidence scores $s_{i}$. The details of the training process
are described in Sec. 3.1 of the paper and are analogous to [47]. At test
time, each pixel in the predicted heatmaps
$\hat{Y}\in[0,1]^{\frac{W}{R}\times\frac{W}{R}\times C}$ corresponds to a
potential object prediction. In our experiments, the heatmaps have a
resolution of $64\times 64$. However, we only consider detections with a
confidence score $s_{i}$ above a fixed threshold $\tau$, $s_{i}>\tau$. For the
experiments on the synthetic CoReNet [34] datasets, we set $\tau=10^{-2}$ and
for the inference on the real-world images (discussed below), we set
$\tau=10^{-1}$ to compensate for the larger uncertainty originating from the
generalization gap since we evaluate on real data while training on synthetic
data.

##  Appendix B Inference speed

One full pass of the presented method from the input image of size $256$
$\times$ $256$ px2 to the multi-object 3D reconstruction takes about $29$ ms.
This corresponds to 34 FPS, our method is real-time capable running an a P100
GPU. Unlike [34], our method does not require additional post-processing steps
(, marching cubes) to extract the final object surface. Instead, we place the
detected shape exemplar from the shape database into the scene and align it to
the predicted 9-DoF bounding box.

##  Appendix C Additional qualitative results on real scenes

We show additional qualitative result on real pictures taken with a mobile
phone. The images in this real dataset show comparable object classes and
imitate the general placement of objects of the synthetic dataset on which it
is trained. Besides cropping and resizing the input images to $256$ $\times$
$256$ px2, no additional pre-processing steps are performed. As described in
the main paper, our model is trained on the synthetic ShapeNet-triplet dataset
presented in [34] and now evaluated on real images. We also show qualitative
results from the CoReNet $m_{9}$ model, trained an the same synthetic dataset
and evaluated on the newly recorded real images. Qualitative results on both
models are shown below.

##  Appendix D Discussion and comparison to CoReNet

In general, the method presented in this work generalizes notably better to
real images than CoReNet, in terms of qualitative reconstructions. This is
mostly due to the fact that we represent objects as points and formulate
reconstruction as a shape selection problem. In particular, this means that
our reconstructions are always valid shapes without holes or detached
components. Our strong data augmentation is likely to contribute as well to
the improved generalization from synthetic training data. Further, our method
performs reconstruction on an object level, such that different instances of
the same semantic class are represented individually along with a 9-DoF
bounding box each.

Additionally, our method performs reconstruction per object instance. In
CoReNet, different instances of the same object class are not easily obtained
since individual objects of the same semantic class share a dense voxel-grid.
Separating different instances could be achieved via connected components but
might fail in case of wrongly disconnected parts. On a technical side, CoReNet
predicts a dense $128^{3}$ voxel grid aligned to the image. This leads to
scalability issues when increasing the scene size (cubic growth). Further,
CoReNet cannot reconstruct objects that are truncated in the input image and
therefor outside of the dense $128^{3}$ voxel grid. Finally, it reconstructs
objects in the camera coordinate system while our method predicts objects in
the world-coordinate system. Our approach has the advantage that objects can
directly be rendered on a ground plane. However, both methods perform the
reconstruction only up to scale due to missing camera extrinsics at test time.

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

Input Image | CoReNet | Ours
---|---|---

## References

  * [1] David Arthur and Sergei Vassilvitskii.  K-Means++: The Advantages of Careful Seeding.  In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms, 2007.
  * [2] Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, and Matthias Nießner.  SceneCAD: Predicting Object Alignments and Layouts in RGB-D Scans.  In European Conference on Computer Vision (ECCV), 2020.
  * [3] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository, 2015.
  * [4] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaako Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer.  In Neural Information Processing Systems (NeurIPS), 2019.
  * [5] Zhiqin Chen and Hao Zhang.  Learning Implicit Fields for Generative Shape Modeling.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [6] Christopher B Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction.  In European Conference on Computer Vision (ECCV), 2016.
  * [7] Haoqiang Fan, Hao Su, and Leonidas J. Guibas.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [8] R. Girdhar, D.F. Fouhey, M. Rodriguez, and A. Gupta.  Learning a Predictable and Generative Vector Representation for Objects.  In European Conference on Computer Vision (ECCV), 2016.
  * [9] Georgia Gkioxari, Jitendra Malik, and Justin J Johnson.  Mesh R-CNN.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [10] Gene H. Golub and Charles F. Van Loan.  Matrix Computations (3rd Ed.).  Johns Hopkins University Press, USA, 1996.
  * [11] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry.  AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [12] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black.  Resolving 3D Human Pose Ambiguities with 3D Scene Constraints.  In International Conference on Computer Vision (ICCV), 2019.
  * [13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In International Conference on Computer Vision (ICCV), 2017.
  * [14] Philipp Henzler, Niloy J Mitra, , and Tobias Ritschel.  Escaping Plato’s Cave: 3D Shape From Adversarial Rendering.  In International Conference on Computer Vision (ICCV), 2019.
  * [15] Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu.  Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image.  In European Conference on Computer Vision (ECCV), 2018.
  * [16] Hamid Izadinia, Qi Shan, and Steven M Seitz.  IM2CAD.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [17] L. Jiang, Shaoshuai Shi, Xiaojuan Qi, and J. Jia.  GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction.  In European Conference on Computer Vision (ECCV), 2018.
  * [18] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent Reconstruction of Multiple Humans from A Single Image.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [19] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3D Mesh Renderer.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [20] Abhijit Kundu, Yin Li, and James M. Rehg.  3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [21] Weicheng Kuo, Anelia Angelova, Tsung-yi Lin, and Angela Dai.  Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve.  In European Conference on Computer Vision (ECCV), 2020.
  * [22] Weicheng Kuo, Anelia Angelova, Jitendra Malik, and Tsung-Yi Lin.  ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors.  In abs/1904.03239, 2019.
  * [23] H. Law and Deng J.  Cornernet: Detecting Objects as Paired Keypoints.  In European Conference on Computer Vision (ECCV), 2018.
  * [24] J. Levinson, Carlos Esteves, Kefan Chen, Noah Snavely, A. Kanazawa, Afshin Rostamizadeh, and A. Makadia.  An Analysis of SVD for Deep Rotation Estimation.  In Neural Information Processing Systems (NeurIPS), 2020.
  * [25] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal Loss for Dense Object Detection.  In International Conference on Computer Vision (ICCV), 2017.
  * [26] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li.  Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning.  In International Conference on Computer Vision (ICCV), 2019.
  * [27] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy Networks: Learning 3D Reconstruction in Function Space.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [28] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked Hourglass Networks for Human Pose Estimation.  In European Conference on Computer Vision (ECCV), 2016.
  * [29] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang.  Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [30] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [33] Matt Pharr, Wenzel Jakob, and Greg Humphreys.  Physically Based Rendering: From Theory to Implementation.  Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 3rd edition, 2016\.
  * [34] Stefan Popov, Pablo Bauszat, and Vittorio Ferrari.  CoReNet: Coherent 3D scene reconstruction from a single RGB image.  In European Conference on Computer Vision (ECCV), 2020.
  * [35] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep Hough Voting for 3D Object Detection in Point Clouds.  In International Conference on Computer Vision (ICCV), 2019.
  * [36] Stephan Richter and Stefan Roth.  Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [37] Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, and Richard Newcombe.  FroDO: From Detections to 3D Objects.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [38] Shuran Song, Samuel Lichtenberg, and Jianxiong Xiao.  SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [39] Maxim Tatarchenko*, Stephan R. Richter*, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What Do Single-view 3D Reconstruction Networks Learn?  In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [40] Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A. Efros, and Jitendra Malik.  Factoring shape, pose, and layout from the 2d image of a 3d scene.  In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [41] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images.  In European Conference on Computer Vision (ECCV), 2018.
  * [42] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, and Joshua B Tenenbaum.  MarrNet: 3D Shape Reconstruction via 2.5D Sketches.  In Neural Information Processing Systems (NeurIPS), 2017.
  * [43] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.  Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling.  In Neural Information Processing Systems (NeurIPS), 2016.
  * [44] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3D Object Detection and Tracking.  arXiv:2006.11275, 2020.
  * [45] Siwei Zhang, Yan Zhang, Qianli Ma, Michael J. Black, and Siyu Tang.  PLACE: Proximity Learning of Articulation and Contact in 3D Environments.  In International Conference on 3D Vision (3DV), 2020.
  * [46] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl.  Tracking Objects as Points.  European Conference on Computer Vision (ECCV), 2020.
  * [47] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as Points.  In arXiv preprint arXiv:1904.07850, 2019.


&\note{d}{=}h\Big(\phase{\abs{\mbf{x}_{2,2}}+\mbf{z}_{2,2}}\!\modsub\!2\phase{\abs{\mbf{x}_{2,1}}+\mbf{z}_{2,1}}\!\modplus\!\phase{\abs{\mbf{x}_{2,0}}+\mbf{z}_{2,0}}\bgiven \abs{\rvec{x}_2},\brace{|\mbf{y}_{2,i}|}_{i=0}^{2}\Big)\nonumber\\
&\note{e}{=}h\fparen{\phi(2,\abs{\rvec{x}_2})\given \abs{\rvec{x}_2},\brace{|\mbf{y}_{2,i}|}_{i=0}^{2}},
\label{eq:app_lb_h(y2|x2)_a}
\end{align}

where (a) follows directly from (<ref>) and that $\phase{\rvec{x}}_2$ is irrelevant for $\phase{\mbf{y}}'_{2,2}$, (b) follows as we added or subtracted given phases which does not change the entropy, and (c) holds as conditioning reduces the entropy. Moreover, in (d) we used that according to (<ref>) for $m=2$ we have $\thetab_{2,2} = 2\thetab_{2,1}\modsub\thetab_{2,0}$; finally, in (e) the $\phi(\cdot)$ function is defined in (<ref>).

For the case $m>2$, we follow similar footsteps as in (<ref>) and write

\begin{align}
h&\fparen{\phase{\mbf{y}}_{2,m}\given \rvec{x}_2,\brace{\phase{\mbf{y}}_{2,i}}_{i=0}^{m-1},\brace{|\mbf{y}_{2,i}|}_{i=0}^{m}} \nonumber\\
&= h\fparen{\phase{\mbf{y}}'_{2,m}\given \abs{\rvec{x}_2},\brace{\phase{\mbf{y}}'_{2,i}}_{i=0}^{m-1},\brace{|\mbf{y}_{2,i}|}_{i=0}^{m}} \nonumber\\
&=h\Big( \phase{\mbf{y}}'_{2,m}\modsub\phase{\mbf{y}}'_{2,m-1}\modsub\phase{\mbf{y}}'_{2,m-2}\modplus\phase{\mbf{y}}'_{2,m-3}\bgiven \abs{\rvec{x}_2},\brace{\phase{\mbf{y}}'_{2,i}}_{i=0}^{m-1},\brace{|\mbf{y}'_{2,i}|}_{i=0}^{m}\Big)\nonumber\\
&\note{a}{\le }h\fparen{\phi(m,\abs{\rvec{x}_2})\given \abs{\rvec{x}_2},\brace{|\mbf{y}_{2,i}|}_{i=0}^{m}},
\label{eq:app_lb_h(y2|x2)_b}
\end{align}

where in the last inequality, we used that the conditioning reduces entropy and that $\thetab_{2,m}= \thetab_{2,m-1}\modplus\thetab_{2,m-2}\modsub\thetab_{2,{m-3}}$, for $m>2$. Note that we could possibly use $\thetab_{2,m} = 2\thetab_{2,m-1}\modsub\thetab_{2,m-2}$ as in (<ref>) but it would lead to a looser bound. Finally, combining (<ref>) and (<ref>) we obtain for $m\ge2$

\begin{equation}
h\fparen{\phase{\mbf{y}}_{2,m}\given \rvec{x}_2,\brace{\phase{\mbf{y}}_{2,i}}_{i=0}^{m-1},\brace{\abs{\mbf{y}_{2,i}}}_{i=0}^{m}} \le h\fparen{\phi(m,\abs{\rvec{x}_2})\given \abs{\rvec{x}_2},\brace{|\mbf{y}_{2,i}|}_{i=0}^{m}}.\label{eq:app_lb_h(y2|x2)_ab_final}
\end{equation}

For the second entropy term on the RHS of (<ref>),

\begin{align}
h\fparen{\phase{\mbf{y}}_{2,m}\given \rvec{x}_2,\vec{\thetab}_2,\big\{\phase{\mbf{y}}_{2,i}\big\}_{i=0}^{m-1},\big\{\abs{\mbf{y}_{2,i}}\big\}_{i=0}^{m}}
&= h\fparen{\phase{\mbf{y}}_{2,m}-\phase{\mbf{x}}_{2,m}\given \rvec{x}_2,\vec{\thetab}_2,\big\{\phase{\mbf{y}}_{2,i}\big\}_{i=0}^{m-1},\big\{\abs{\mbf{y}_{2,i}}\big\}_{i=0}^{m}} \nonumber\\
&\note{a}{=}  h\fparen{\thetab_{2,m}\modplus\phase{\abs{\mbf{x}_{2,m}}+\mbf{z}_{2,m}}\bgiven \abs{\mbf{x}_{2,m}},\thetab_{2,m},\abs{\mbf{y}_{2,m}}}\nonumber\\
&\note{b}{=} h\paren{\phase{\abs{\mbf{x}_{2,m}}+\mbf{z}_{2,m}}\bgiven \abs{\mbf{x}_{2,m}},\abs{\mbf{y}_{2,m}}},\label{eq:app_lb_h(y2|x2)_c}
\end{align}

where (a) follows as $\brace{\phase{\mbf{y}}_{2,i}}_{i=0}^{m-1}$ are irrelevant given ${\thetab}_{2,m}$ and (b) follows since $\thetab_{2,m}$ is independent of $\phase{\abs{\mbf{x}_{2,m}}+\mbf{z}_{2,m}}$.

Substituting (<ref>) and (<ref>) into (<ref>) gives

\begin{align}
h&\fparen{\brace{\mbf{y}_{2,m}}_{m=2}^{M-1}\bgiven \rvec{x}_2,\mbf{y}_{2,0},\mbf{y}_{2,1}}\nonumber\\
&\le (M-2)\log(\pi e)+\sum_{m=2}^{M-1} \bigg(h\fparen{\phi(m,\abs{\rvec{x}_2})\bgiven \abs{\rvec{x}_2},\brace{|\mbf{y}_{2,i}|}_{i=0}^{m}} -h\fparen{\phase{\abs{\mbf{x}_{2,m}}+\mbf{z}_{2,m}}\bgiven \abs{\mbf{x}_{2,m}},|\mbf{y}_{2,m}|}\bigg)\nonumber\\
&=(M-2)\log(\pi e)+\sum_{m=2}^{M-1}\E[g(m,\abs{\rvec{x}_2})],\label{eq:lb_h(y2|x2)_b_final}
\end{align}

where $g(\cdot)$ is defined in (<ref>). This concludes the proof.

§.§ Proof of Lemma <ref>

Following similar steps as in <cit.>, we can write

\begin{align}
I\paren{\rvec{x}_2;\vec{\thetab}_1\given \rvec{x}_{1},\rvec{y}_{1},\rvec{y}_2}
&= h\paren{\vec{\thetab}_1\given \rvec{x}_{1},\rvec{y}_{1},\rvec{y}_2}-h\paren{\vec{\thetab}_1\given \rvec{x}_{1},\rvec{x}_{2},\rvec{y}_{1},\rvec{y}_2}\nonumber\\
&\note{a}{\le} h\paren{\vec{\thetab}_1\given\rvec{x}_{1},\rvec{y}_{1}}-h\paren{\vec{\thetab}_1\given \rvec{x}_{1},\rvec{x}_{2},\rvec{y}_{1},\rvec{y}_2,\vec{\thetab}_2}\nonumber\\
&\note{b}{=} h\paren{\vec{\thetab}_1\given\rvec{x}_{1}, \rvec{y}_{1}}-h\paren{\vec{\thetab}_1\given \rvec{x}_{1},\rvec{y}_{1},\vec{\thetab}_2}\nonumber\\
&= I\paren{\vec{\thetab}_2;\vec{\thetab}_1\bgiven\rvec{x}_{1},\rvec{y}_{1}}\nonumber\\
&\note{c}{=} I\paren{\{\thetab_{2,m}\}_{0}^{1};\vec{\thetab}_1\bgiven\rvec{x}_{1},\rvec{y}_{1}}\nonumber\\
&= h\paren{\{\thetab_{2,m}\}_{0}^{1}\bgiven\rvec{x}_{1},\rvec{y}_{1}}-h\paren{\{\thetab_{2,m}\}_{0}^{1}\bgiven\rvec{x}_{1},\rvec{y}_{1},\vec{\thetab}_1}\nonumber\\
&\note{d}{=} h\paren{\{\thetab_{2,m}\}_{0}^{1}\bgiven\rvec{x}_{1},\rvec{y}_{1}}-h\paren{\{\thetab_{2,m}\}_{0}^{1}\bgiven \vec{\thetab}_1},
\label{eq:epsilon_proof}
\end{align}

where (a) holds because conditioning reduces the entropy; (b) follows since given $\thetacb{2}$, $\thetacb{1}$ is independent of the pair $(\rvec{y}_2,\rvec{x}_2)$; (c) follows as $\{\thetab_{2,m}\}_{0}^{1}$ is sufficient statistics for $\vec{\thetab}_2$; (d) holds since $\rvec{\thetab}_2$ and $(\rvec{y}_1,\rvec{x}_1)$ are independent given $\rvec{\thetab}_1$.

The first term on the RHS of (<ref>) can be upper-bounded as

\begin{align}
h\paren{\brace{\thetab_{2,m}}_{0}^{1}\bgiven \rvec{x}_{1},\rvec{y}_{1}}
&=h\fparen{\brace{\thetab_{2,m}}_{0}^{1}\bgiven \abs{\rvec{x}_{1}},\phase{\rvec{x}}_{1}, |\rvec{y}_{1}|, \phase{\rvec{y}}_{1}}\nonumber\\
&=h\fparen{\brace{\thetab_{2,m}\modsub\phase{\mbf{y}}_{1,m} \modplus \phase{\mbf{x}}_{1,m}}_{0}^{1}\bgiven \abs{\rvec{x}_{1}},\phase{\rvec{x}}_{1}, |\rvec{y}_{1}|, \phase{\rvec{y}}_{1}}\nonumber\\
&\note{a}\le h\paren{\brace{\thetab_{2,m}\modsub\phase{\mbf{y}}_{1,m}\modplus \phase{\mbf{x}}_{1,m}}_{0}^{1}\bgiven \abs{\rvec{x}_1},\brace{|\mbf{y}_{1,m}|}_{0}^{1}}\nonumber\\
&\note{b}{=}h\big(\brace{\thetab_{2,m}\modsub\thetab_{1,m} \modsub\phase{\abs{\mbf{x}_{1,m}}+\mbf{z}_{1,m}}}_{0}^{1}\bgiven|\rvec{x}_1|,\brace{|\mbf{y}_{1,m}|}_{0}^{1}\big)\nonumber\\
&\note{c}{=}h\big(\brace{\Deltacb{2}\modplus m\Deltarb{2}\modsub\phase{\abs{\mbf{x}_{1,m}}+\mbf{z}_{1,m}}}_{0}^{1}\bgiven|\rvec{x}_1|,\brace{|\mbf{y}_{1,m}|}_{0}^{1}\big)\nonumber\\
h\big(\brace{\Deltacb{2}\modplus m\Deltarb{2}\modplus\phase{\abs{\mbf{x}_{2,m}}+\mbf{z}_{2,m}}}_{0}^{1}\bgiven|\rvec{x}_2|,\brace{|\mbf{y}_{2,m}|}_{0}^{1}\big)\label{eq:epsilon_a_final}
\end{align}

where (a) holds because conditioning reduces entropy, (b) follows directly from (<ref>) where $\phase{\mbf{y}}_{1,m} = \thetab_{1,m} \modplus \phase{\mbf{x}}_{1,m}\modplus \phase{|\mbf{x}_{1,m}| + \mbf{z}_{1,m}}$, and (c) holds since $\thetab_{2,m} \modsub \thetab_{1,m} = \Deltacb{2}\modplus m\Deltarb{2} $from (<ref>)–(<ref>); finally, in (d) we use that $-\phase{|\mbf{x}_{1,m}|+\mbf{z}_{1,m}}$ has the same distribution as $\phase{|\mbf{x}_{2,m}|+\mbf{z}_{2,m}}$, because the phase function is symmetric and the pairs $(|\rvec{x}_{1}|,|\rvec{x}_{2}|)$ and $(\mbf{z}_{1,m},\mbf{z}_{2,m})$ are exchangeable since they have the same distribution due to the iid assumption of the input distribution.

We continue with the second term on the RHS of (<ref>)

\begin{align}
h\paren{\{\thetab_{2,m}\}_{0}^{1}\given \vec{\thetab}_1} &\note{a}{=}h\paren{\thetacb{2},\thetacb{2}\modplus\thetarb{2}\given \thetacb{1},\thetarb{1}}\nonumber\\
& \note{b}{=} h\paren{\thetacb{2},\thetarb{2}\given \thetacb{1},\thetarb{1}}\nonumber\\
& = h(\Deltacb{2},\Deltarb{2}),\label{eq:epsilon_b_final}
\end{align}

where (a) follows since the pair $(\thetacb{1},\thetarb{1})$ is sufficient statistic for $\vec{\thetab}_1$ and (b) holds since the pair $(\thetacb{2},\thetarb{2})$ is sufficient statistics for the pair $(\thetacb{2},\thetacb{2}\modplus\thetarb{2})$.

Substituting (<ref>) and (<ref>) into (<ref>) gives (<ref>) and the proof is completed.

§.§ Proof of Lemma <ref>

By setting $\lambda=\lambda^{\ast} = (M-1)\log e$ and $\vec{\alpha} = \vec{\alpha}^{\ast}$ in (<ref>) we get

\begin{align}
R_{\lambda^{\ast},\vec{\alpha}^{\ast}}(\rho,\vec{s}) &=\frac{1}{2}\Eb{\log|s_0+\mbf{z}_0|^2}-h\paren{|s_0+\mbf{z}_0|^2\bgiven s_0}\nonumber\\
&\quad+\frac{1}{2}\Eb{\log|s_1+\mbf{z}_1|^2}-h\paren{|s_1+\mbf{z}_1|^2\bgiven s_1},\label{eq:R_prime}
\end{align}

and employing Lemma <ref> gives

\begin{align}
\lim_{s_0 \to \infty} &\frac{1}{2}\Eb{\log|s_0+\mbf{z}_0|^2}\!-\!h\paren{|s_0+\mbf{z}_0|^2\bgiven s_0} =\!-\frac{1}{2}\log (4\pi e),\label{eq:lim_log_a}\\
\lim_{s_1 \to \infty}&\frac{1}{2}\Eb{\log|s_1+\mbf{z}_1|^2}\!-\!h\paren{|s_1+\mbf{z}_1|^2\bgiven s_1}=\!-\frac{1}{2}\log (4\pi e),\label{eq:lim_log_b}
\end{align}

which proves (<ref>).

To characterize the asymptotic behavior of (<ref>), we use Lemma <ref> and that the phases $\phase{s_0+\mbf{z}_0} \to 0$, $\phase{s_1+\mbf{z}_1} \to 0$, and $\phase{\norm{\vec{s}}+\mbf{v}} \to 0$ as $\vec{s}\to \infty$. Then,

\begin{align}
\lim_{\substack{\vec{s} \to \infty}} h\!\fparen{\brace{\Deltacb{}\!\modplus\! m\Deltarb{}\!\modplus\!\phase{{s}_{m}+\mbf{z}_{m}}}_{0}^{1}\bgiven|\vec{s}+\rvec{z}|} = h(\Deltacb{},\Deltarb{})\label{eq:lim_F_a}
\end{align}

\begin{align}
\lim_{\substack{\vec{s} \to \infty}} h(\Deltarb{})+h\fparen{\Deltacb{}\!\modplus\!\phase{\norm{\vec{s}}+\mbf{v}}\,\Big|\,\big|\!\norm{\vec{s}}+\mbf{v}\big|}= h(\Deltacb{},\Deltarb{}),\label{eq:lim_F_b}
\end{align}

which results in (<ref>) and the proof is concluded.

§.§ Proof of Lemma <ref>

We can start with the definition of $J(\cdot)$ from (<ref>) and write

\begin{align}
J(\alpha'_m(\gamma), \gamma) &= \alpha'_m(\gamma) + \frac{e^{-\gamma} \gamma^{\alpha'_m(\gamma)}}{\Gamma(\alpha'_m(\gamma),\gamma)} \nonumber\\
&\note{a}{\le}     \alpha'_m(\gamma) + \frac{e^{-\gamma} \gamma^{\alpha'_m(\gamma)}}{\Gamma(1,\gamma)}\nonumber\\
&\note{b}{=} \alpha'_m(\gamma) + \gamma^{\alpha'_m(\gamma)},\label{eq:upper_bound_on_Es}
\end{align}

where (a) follows because $\Gamma(\alpha'(\gamma),\gamma) \ge \Gamma(1,\gamma)$ from Lemma <ref> and (b) follows as $\Gamma(1,\gamma) = e^{-\gamma}$. Note that in (a) we could utilize Lemma <ref> because $\gamma \le x_\mr{max} < x_0$ and $0\le\alpha'(\gamma)\le 1$.

Now, substituting (<ref>) into (<ref>) we can write

\begin{align}
\alpha'_m(\gamma) + \gamma^{\alpha'_m(\gamma)} &= \alpha_m^* - c_m(\gamma) + \frac{\gamma^{\alpha_m^*}}{\gamma^{c_m(y)}}\nonumber\\
& \note{a}{=} \alpha_m^* - c_m(\gamma) +c_m(\gamma)\nonumber\\
& = \alpha_m^*,
\end{align}

where (a) follows from (<ref>) in Lemma <ref>. Thus,

\begin{align}
J(\alpha'_m(\gamma), \gamma) \le \alpha^*_m,
\end{align}

which proves (<ref>).

Next we prove (<ref>), which requires to show $\lim_{\gamma \to 0^{+}} c_m(\gamma) = 0$. Based on the definition of the Lambert W function we have

\begin{align}
W_{\mr{L}}(x) = \frac{x}{e^{W_{\mr{L}}(x)}}. \label{eq:lambert_def}
\end{align}

Thus, we can write

\begin{align}
\lim_{\gamma \to 0^{+}} c_m(\gamma) &= \lim_{\gamma \to 0^{+}} \frac{\gamma^{\alpha_m^*}\log_{e}\gamma}{e^{W_{\mr{L}}(\gamma^{\alpha_m^*}\log_{e}\gamma)}\log_{e}\gamma} \nonumber\\
& = \lim_{\gamma \to 0^{+}} \frac{\gamma^{\alpha_m^*}}{e^{W_{\mr{L}}(\gamma^{\alpha_m^*}\log_{e}\gamma)}}\nonumber\\
& = \frac{0}{1}\nonumber\\
&= 0,
\end{align}

which proves (<ref>).

§.§ Proof of Lemma <ref>

In this proof, we need to characterize $g(m,\vec{s})$ for a deterministic vector

\begin{equation}
\vec{s}>\sqrt{\mu\gamma}.\label{eq:app:sm_lowerbound}
\end{equation}

For all $m\in\{0,\dots,M-1\}$, consider iid $\mbf{z}_m \sim \CN(0,1)$ and let

\begin{align}
\bs{\varphi}_m &= \phase{{s}_m+\mbf{z}_m},\label{eq:app:varphi} \\
\mbf{r}_m &= \abs{{s}_m+\mbf{z}_m}\nonumber\\
&= s_m \fparen{1+O\fparen{1/{\sqrt{\mu\gamma}}}}.\label{eq:app:rm}
\end{align}

Then, according to Lemma <ref>, given $s_m$ and $\mbf{r}_m = r_m$, we have $\bs{\varphi}_m \sim \mathcal{VM}(0,2{s}_m r_m)$ with its pdf denoted by $f_{\bs{\varphi}_m}^\mathrm{VM}(\varphi_m; 0, 2{s}_m r_m)$.
Using Lemma <ref>, the pdf of a von Mises distribution can be approximated by a wrapped normal distribution as

\begin{align}
f_{\bs{\varphi}_m}^\mathrm{VM}(\varphi_m;0,2{s}_m r_m) &\!\note{a}{=} \!f_{\bs{\varphi}_m}^\mathrm{WN}\!\fparen{\!{\varphi}_m;0,\frac{1}{2{s}_m r_m}\!}\!+\!O\!\bigg(\!\frac{1}{\sqrt{{s}_m r_m}}\!\bigg)\!.
\label{eq:appc_wn}
\end{align}

We need to evaluate the two terms on the RHS of (<ref>), which can be reformulated as

\begin{align}
h\fparen{\phase{{s}_{m}+\mbf{z}_{m}}\bgiven {s}_{m},\abs{{s}_{m}+\mbf{z}_{m}}} &= h\fparen{\bs{\varphi}_m\bgiven {s}_{m},\mbf{r}_m},\label{eq:app:termB}\\
h\fparen{\phi(m,\vec{s})\bgiven \vec{s},\brace{|{s}_{i}+\mbf{z}_{i}|}_{i=0}^{m}} &= h\fparen{\phi(m,\vec{s})\bgiven \vec{s},\brace{\mbf{r}_i}_{i=0}^{m}}. \label{eq:app:termA}
\end{align}

To evaluate the RHS of (<ref>), we use Lemma <ref> and write

\begin{align}
h\fparen{\bs{\varphi}_m \given {s}_{m},\mbf{r}_m} &= \frac{1}{2}\log(2\pi e)-\frac{1}{2}\log\fparen{2s_m r_m} +O\fparen{\frac{1}{\sqrt{s_m r_m}}}\nonumber\\
& \note{a}{=}\frac{1}{2}\log(2\pi e)-\frac{1}{2}\log\fparen{2s_m^2}+O\fparen{\frac{1}{\sqrt{\mu\gamma}}} ,\label{eq:app:gm_b}
\end{align}

where (a) follows from (<ref>).

To evaluate the RHS of (<ref>), we approximate each term of $\phi(m,\vec{s})$ in (<ref>) with a wrapped normal random variable. Thus, for given $m$ and $\vec{s}$, the conditional distribution of $\phi(m,\vec{s})$
can be approximated by a summation of wrapped normal random variables

\begin{align}
f_{\phi(m,\vec{s})}(\varphi) =f_{\phi(m,\vec{s})}^\mathrm{WN}\fparen{\varphi;0, a(m,\vec{s})}+O\fparen{a(m,\vec{s})},
\end{align}

\begin{align}
a(m,\vec{s}) &= \begin{cases}
\frac{1}{2s_2 r_2}+\frac{2}{s_1 r_1}+\frac{1}{2s_0 r_0}, & \hspace{7ex} m=2,\\
\sum_{i=m-3}^{m} \frac{1}{2s_i r_i}, &\hspace{7ex} m>2,
\end{cases}\nonumber\\
&= \begin{cases}
\frac{1}{2s_2^2}+\frac{2}{s_1^2}+\frac{1}{2s_0^2}+O\fparen{\frac{1}{\sqrt{\mu\gamma}}}, & m=2, \\
\sum_{i=m-3}^m \frac{1}{2s_i^2} +O\fparen{\frac{1}{\sqrt{\mu\gamma}}}, & m>2.
\end{cases}\label{eq:app.D10:temp}
\end{align}

Using Lemma <ref> and (<ref>),
we can write

\begin{align}
h\!\fparen{\phi(m,\vec{s})\bgiven \vec{s},\{\mbf{r}_i\}_{i=0}^{m}} &= \frac{1}{2} \log\fparen{2\pi e} + \frac{1}{2} \log\fparen{a(m,\vec{s})}+ O\fparen{\sqrt{a(m,\vec{s})}}\nonumber\\
&= \frac{1}{2} \log\fparen{2\pi e} + \begin{cases}
\frac{1}{2}\log\fparen{\frac{1}{2s_2^2}+\frac{2}{s_1^2}+\frac{1}{2s_0^2}}+O(\frac{1}{\sqrt{\mu\gamma}}), & m=2,\\
\frac{1}{2}\log\fparen{\sum_{i=m-3}^{m} \frac{1}{2s_i^2}} + O(\frac{1}{\sqrt{\mu\gamma}}), & m>2.
\end{cases} \label{eq:app:gm_a}
\end{align}

Finally, substituting (<ref>) and (<ref>) into (<ref>) and (<ref>), respectively; then, (<ref>) and (<ref>) into (<ref>) gives (<ref>), and the proof is completed.