# RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory
Sound Classification

June-Woo Kim1,2 Miika Toikkanen2 Sangmin Bae2,3 Minseok Kim4†‡ Ho-Young Jung1†
{kaen2891<EMAIL_ADDRESS><EMAIL_ADDRESS>*This research was supported
by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information
Technology Research Center) support program (IITP-2024-2020-0-01808)
supervised by the IITP (Institute of Information & Communications Technology
Planning & Evaluation) and by Brian Impact Foundation, a non-profit
organization dedicated to the advancement of science and technology for
all.†Corresponding authors ‡Work done before joining Amazon1Department of
Artificial Intelligence, Kyungpook National University2RSC LAB, MODULABS
3KAIST AI 4Amazon

###### Abstract

Recent advancements in AI have democratized its deployment as a healthcare
assistant. While pretrained models from large-scale visual and audio datasets
have demonstrably generalized to this task, surprisingly, no studies have
explored pretrained speech models, which, as human-originated sounds,
intuitively would share closer resemblance to lung sounds. This paper explores
the efficacy of pretrained speech models for respiratory sound classification.
We find that there is a characterization gap between speech and lung sound
samples, and to bridge this gap, data augmentation is essential. However, the
most widely used augmentation technique for audio and speech, SpecAugment,
requires 2-dimensional spectrogram format and cannot be applied to models
pretrained on speech waveforms. To address this, we propose RepAugment, an
input-agnostic representation-level augmentation technique that outperforms
SpecAugment, but is also suitable for respiratory sound classification with
waveform pretrained models. Experimental results show that our approach
outperforms the SpecAugment, demonstrating a substantial improvement in the
accuracy of minority disease classes, reaching up to 7.14%.

##  I INTRODUCTION

Recent remarkable advances in machine learning are rapidly transforming the
landscape of healthcare, offering unprecedented opportunities for early
disease detection and personalized treatment pathways. Within this evolving
paradigm, respiratory sound classification has gained significant attention
due to its potential to non-intrusively identify lung diseases based on
respiratory sound. However, the high cost of collecting lung sound data has
been an obstacle to the wide adoption of respiratory sound classifier models
because a substantial amount of data is typically required to train a
sufficiently reliable machine learning model.

To cope with the respiratory data deficiency, previous studies [1, 2, 3, 4, 5]
for respiratory sound classification have usually exploited the following two-
staged approach: $(i)$ _pretraining_ with sufficient data from other domains
(e.g., image or audio data) to enhance general feature representation
capability, and $(ii)$ then _fine-tuning the pretrained model with data
augmentation_ on respiratory sound to avoid the overfitting issues caused by
lack of diverse data. The state-of-the-art approach [1] for respiratory sound
classification demonstrated impressive performance by exploiting Audio
Spectrogram Transformer (AST) [6] pretrained on audio and image datasets, and
fine-tuning with the help of SpecAugment [7] that augments respiratory sound
samples in the format of spectrogram images.

Previous successful approaches to respiratory sound classification have been
limited to image-based models where the audio data is transformed into
spectrograms. Nevertheless, such transformation can lead to considerable loss
of information, so end-to-end learning that directly leverages the dense and
rich information of the original waveforms has gained significant attention in
recent speech domain studies [8, 9, 10, 11]. With the same motivation, our
research goal is to investigate the lung sound classification performance by
directly using the original waveforms. Instead of models pretrained on images,
we first explore speech-based pretrained encoders [9, 10, 11] that focus on
learning speech representations while having access to more rich information
present in the waveform.

With the change in the input domain, we can no longer apply SpecAugment [7],
which is limited to image formats (i.e., 2D spectrogram), to fine-tune the
models. In this regard, there is a call for an _input-agnostic_ approach that
can be applied universally to the strong capability of pretrained models with
_any_ input type for respiratory sound classification. In this paper, we
propose _RepAugment_ , a novel and efficient input-agnostic augmentation
technique applied at the representation level. The proposed RepAugment
operates directly on the model’s output representations before classifier,
rendering it input-agnostic and applicable to a wide range of model
architectures. In a synergistic effect between representation masking and
noise generation, RepAugment reduces the reliance on particular features,
while expanding the model’s understanding of the under-represented classes.

Experimental results demonstrate the effectiveness of RepAugment, specifically
when applied to models pretrained with audio or speech datasets, outperforming
the conventional SpecAugment. Moreover, our approach also shows accuracy gains
up to 7.14% for the most under-represented classes, highlighting its potential
to improve the diagnostics of abnormal cases. Our main contributions are
summarized as follows:

  * •

We explore the capability of pretrained speech models, and compare them to the
models pretrained on image and audio datasets commonly used for respiratory
sound classification.

  * •

We propose RepAugment, a simple and novel input-agnostic representation-level
augmentation method that can leverage the strong capability of pretrained
models with any input type for respiratory sound classification.

  * •

We demonstrate the effectiveness of RepAugment, particularly for improving the
performance of minority categories, when fine-tuning respiratory samples under
several pretraining conditions.

##  II RELATED WORKS

###  II-A Pretrained Models

Numerous deep learning methods have emerged for the lung sound classification
task, particularly those based on CNN, such as ResNet [12] and EfficientNet
[13]. Moreover, prior work has predominantly used large-scale visual and audio
datasets like ImageNet [14] and AudioSet [15], and their mixtures [6], which
have been shown to notably improve the performance of models on the ICBHI [16]
dataset. Specifically, Bae et al. [1] conducted a comprehensive study
comparing the performance of different models on respiratory sound
classification with and without using pretrained weights from ImageNet and
AudioSet. This comparison revealed notable differences in effectiveness across
models, highlighting the generalizability of pretrained weights over full
training in the case of respiratory sound classification tasks.

Unlike previous works, our goal is to tap into the ability of such large
models’ that have been pretrained on massive speech data [17, 18, 19], and
have so far remained unused in respiratory sound classification tasks.

###  II-B Data Augmentation Strategy

Data augmentations aim to maintain the class identity of a sample while
varying other aspects such as style to improve generalization. They are
related to domain randomization [20], which involves the random alteration of
samples while ensuring that they remain closely aligned with the underlying
data manifold. Mixup [21] provides variety of benefits for image
classification by applying such operations on data, while Manifold Mixup [22]
achieves the same on the hidden representations of the neural network. The
simple approach of perturbing samples with Gaussian noise has been found
effective at domain generalization [23, 24], few shot classification [25], and
addressing class imbalance by using variance in head classes to augment tail
classes [26].

In the fields of speech recognition, audio classification, and respiratory
sound classification, SpecAugment [7] is a popular data-augmentation technique
that directly operates on the spectrograms, masking out continuous blocks of
time-steps and frequency bins. However, this does not apply to waveform data
used by the state-of-the-art models [9, 10, 11], so there is a call for input-
agnostic augmentation. To this end, we apply a similar strategy on the
representation level to remain independent from the input modality. Masking
the features may be considered analogous to dropout [27], which is not a data
augmentation technique, but can also be viewed as kind of noising data-
augmentation [28]. We augment the features directly by masking out values from
the feature representations, while dropout randomly drops neurons from the
network.

##  III METHOD: REPAUGMENT

###  III-A Data Augmentation Policy

Our objective is to introduce an augmentation policy that operates on the
model’s output representations before classifier, making it independent of
input specifics and adaptable to a diverse array of model architectures. Our
_RepAugment_ strategy, denoted as $A$, is based on two operations. Masking
augmentation _Rep-Mask_ , denoted as $A_{mask}$, and noise generation
augmentation _Rep-Gen_ , denoted as $A_{gen}$.

Rep-Mask operation attempts to reduce the model’s reliance on particular
features by restricting access to them randomly during training. This is done
by masking the feature $z\in\mathbb{R}^{d}$ using a _mask_ $M$ consisting of
values 0 and 1, where 1 indicates the masked and 0 denotes the unmasked part.
Subsequently, we replace the masked parts with the vector $\overline{z}$,
which is the mean of $z$ computed over all dimensions and broadcasted back
into the shape of $z$. This approach was found to work better than using zero
values.

$A_{mask}(z)=z\odot(1-M)+\overline{z}\odot M$ (1)

Here $\odot$ denotes an elementwise multiplication.

Given a feature $z$, in order to generate the masking index of $M$, we apply
$k$ number of mask bands to the feature with different ranges. We set the
masking ranges to $[i$, $i+j)$ positions in $z$. The start index $i$ is chosen
from $[0,d-j)$, and $j$ is chosen from $[0,L)$. We empirically used $k=2$ and
$L=288$ in our experiments. Masking consecutive features channels is not
crucial in experiments of this paper as they are spatially independent, but
becomes more important when applied to 2D features that include the time-axis.

Reg-Gen operation attempts to expand the model’s experience of the
underrepresented classes by adding random variation to them. To achieve this,
we perturb the minority classes of $c_{crackle}$, $c_{wheeze}$, and $c_{both}$
with random noise as follows.

$A_{gen}(z)=z+z_{noise}$ (2)

We generate a Gaussian noise tensor $z_{noise}\sim N(\mu,\sigma^{2})$ and add
it to the original feature $z$ encoded from input $x$ corresponding to class
$c$. We used the standard normal distribution, where $\mu=0$ and $\sigma=1$.

The full augmentation A then becomes the following equation, where the samples
from the normal class $c_{normal}$, only receive the Rep-Mask augmentation.

$A=\begin{cases}A_{mask}(z),&\text{if }c=c_{normal}\\\
A_{gen}(A_{mask}(z)),&\text{else}\\\ \end{cases}$ (3)

The masked values also get perturbed by Rep-Gen thereby also reducing the
unnatural appearance of masking with constant values.

Figure 1: Difference between the input-level and feature-level augmentation
for respiratory sound classification. Our proposed input-agnostic _RepAugment_
which consists of Rep-Mask and Rep-Gen (at bottom right) can be employed for
any input type, whereas the SpecAugment (at top left) can only be applied to
the input spectrogram.

###  III-B Model Architectures

####  III-B1 Pretrained Speech Models

We first explore the capability of pretrained speech models for respiratory
sound classification tasks. In this regard, we use various pretrained models
as our backbone: wav2vec2-base [9] pretrained on the LibriSpeech 960 hours
dataset [17], wav2vec2-large on the LibriLight 60k hours dataset [18], HuBERT-
base [10] on LibriSpeech, HuBERT-large on LibriLight, and XLS-R-300 [11] on
Multilingual LibriSpeech [19] combined with other speech datasets as described
in [11], amounting to a cumulative duration of approximately 436k hours. To
fine-tune these models for the respiratory sound classification tasks, we take
the last hidden states of each pretrained model as input for the lung sound
classifier.

####  III-B2 Pretrained Image and Audio Models

To apply the proposed input-agnostic RepAugment to the fine-tuning process for
various input types, we further use diverse pretrained models on large-scale
visual and audio datasets such as ImageNet [14] and AudioSet [15], including
their combinations. To this end, we use pretrained ResNet [12] and
EfficientNet [13] models on ImageNet, along with CNN6 [29] on AudioSet, and
also leverage the Audio Spectrogram Transformer (AST) [6] for both ImageNet
and AudioSet.

####  III-B3 Applying RepAugment

The proposed RepAugment is applied to the feature representations where they
are obtained from feature encoders. In detail, for both pretrained speech and
image as well as audio models, we aggregate the last hidden states from each
pretrained model by averaging their values across temporal information. We
have empirically found that using average pooling provides a slight
performance improvement than employing mean values of feature representations
except for the AST model. Note that we use the average values of _class_ and
_distill_ tokens in the AST model. These aggregated feature representations
are then augmented with RepAugment and fed into the lung sound classifier
consisting of a 4-dimensional linear layer with LayerNorm [30].

##  IV EXPERIMENTS

###  IV-A Dataset

We used the ICBHI [16] dataset that serves as a popular open benchmark for the
respiratory sound classification task. This dataset has an official split into
training and test sets, maintaining the ratios of 60–40% for the training and
test sets respectively, with no patient overlap between them. Consequently,
the training and test sets comprised 4,142 and 2,756 samples respectively, and
are categorized into four classes: _normal_ (49.8%–57.29%), _crackle_
(29.30%–23.55%), _wheeze_ (12.10%–13.97%), and _both_ (8.80%–5.19%). For the
performance metrics, we used the _Specificity_ , _Sensitivity_ and their
arithmetic mean, hereinafter denoted as $S_{p}$, $S_{e}$, and $Score$,
following the definitions outlined in [16].

###  IV-B Experimental Setting

For training, we standardized the duration of all samples to 8 seconds. In the
case of pretrained speech models, we used the raw waveform of lung sound
samples as model input (i.e., without normalization). The pretrained weights
are from the S3PRL toolkit [31]. For the models pretrained on image and audio,
we used the 128-dimensional log Mel filterbank features with a window size of
25ms and an overlap size of 10ms. The log Mel filterbank features were
normalized with zero mean unit variance, except for the AST, for which we used
the mean and standard deviation of –4.27 and 4.57, as described in [1]. For
the ResNet, EfficientNet, and CNN6, we used the Adam optimizer with a learning
rate of 1e–3 and a batch size of 64 for 400 epochs. For the AST and all
pretrained speech models, we adopted the same optimizer with a learning rate
of 5e–5 and a batch size of 8 for 50 epochs. To ensure result stability, we
conducted the model training with a consistent set of five random seeds across
all experiments. Consequently, the reported values for $S_{p}$, $S_{e}$, and
$Score$ include both the mean and variance.

###  IV-C Experimental Results

TABLE I: Comparison of respiratory sound classification performance under
different pretrained models and augmentation configurations. Bold denotes the
best result.

pretrained on image & audio | Model | # Params | Pretrain | Augmentation |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---|---|---
ResNet [12] | 11.7M | ImageNet | No Aug. | $\text{{76.25}}_{\pm 5.65}$ | $\text{{28.87}}_{\pm 5.59}$ | $\text{{52.57}}_{\pm 1.38}$
SpecAugment | $\text{{{76.70}}}_{\pm 5.65}$ | $\text{{{33.47}}}_{\pm 4.03}$ | $\text{{{55.09}}}_{\pm 0.82}$
_RepAugment_ | $\text{{75.10}}_{\pm 7.63}$ | $\text{{31.63}}_{\pm 8.75}$ | $\text{{53.37}}_{\pm 1.37}$
EfficientNet [13] | 5.3M | ImageNet | No Aug. | $\text{{{79.37}}}_{\pm 3.58}$ | $\text{{{32.67}}}_{\pm 3.26}$ | $\text{{{56.02}}}_{\pm 0.52}$
SpecAugment | $\text{{{78.21}}}_{\pm 3.56}$ | ${{{34.44}}}_{\pm 2.84}$ | $\text{{{56.33}}}_{\pm 0.43}$
_RepAugment_ | $\text{{{75.10}}}_{\pm 7.63}$ | $\text{{{37.23}}}_{\pm 7.41}$ | $\text{{{56.17}}}_{\pm 0.65}$
CNN6 [29] | 4.8M | AudioSet | No Aug. | $\text{{{72.83}}}_{\pm 4.61}$ | $\text{{{39.64}}}_{\pm 3.6}$ | $\text{{{56.62}}}_{\pm 1.25}$
SpecAugment | $\text{{{77.00}}}_{\pm 3.27}$ | $\text{{{37.35}}}_{\pm 3.15}$ | $\text{{{57.17}}}_{\pm 0.81}$
_RepAugment_ | $\text{{{75.93}}}_{\pm 4.88}$ | $\text{{{38.95}}}_{\pm 6.41}$ | $\text{{{57.44}}}_{\pm 1.64}$
AST [1, 6] | 87.7M |  ImageNet + AudioSet | No Aug. | $\text{{{76.57}}}_{\pm 2.41}$ | $\text{{{41.72}}}_{\pm 1.26}$ | $\text{{{59.14}}}_{\pm 1.66}$
SpecAugment | $\text{{{77.14.}}}_{\pm 3.35}$ | $\text{{{43.07}}}_{\pm 2.80}$ | $\text{{{59.55}}}_{\pm 0.88}$
_RepAugment_ | $\text{{{82.47}}}_{\pm 5.39}$ | $\text{{{40.55}}}_{\pm 4.70}$ | $\text{{{61.51}}}_{\pm 0.51}$
pretrained on speech | wav2vec2-Base [9] | 95M | LibriSpeech | No Aug. | $\text{{{71.98}}}_{\pm 7.74}$ | $\text{{{30.86}}}_{\pm 9.10}$ | $\text{{{51.42}}}_{\pm 1.66}$
_RepAugment_ | $\text{{{68.71}}}_{\pm 14.25}$ | $\text{{{35.07}}}_{\pm 14.2}$ | $\text{{{51.89}}}_{\pm 0.44}$
wav2vec2-Large [9] | 310M | LibriLight | No Aug. | $\text{{{68.61}}}_{\pm 2.02}$ | $\text{{{41.09}}}_{\pm 2.94}$ | $\text{{{54.85}}}_{\pm 0.65}$
_RepAugment_ | $\text{{62.80}}_{\pm 2.59}$ | $\text{{{47.51}}}_{\pm 1.51}$ | $\text{{{55.16}}}_{\pm 1.03}$
HuBERT-Base [10] | 94M | LibriSpeech | No Aug. | $\text{{{68.08}}}_{\pm 8.50}$ | $\text{{{38.37}}}_{\pm 8.72}$ | $\text{{{51.22}}}_{\pm 0.31}$
_RepAugment_ | $\text{{{64.80}}}_{\pm 2.51}$ | $\text{{{42.26}}}_{\pm 3.73}$ | $\text{{{53.53}}}_{\pm 1.72}$
HuBERT-Large [10] | 310M | LibriLight | No Aug. | $\text{{{68.03}}}_{\pm 5.19}$ | $\text{{{42.18}}}_{\pm 2.35}$ | $\text{{{55.10}}}_{\pm 1.42}$
_RepAugment_ | $\text{{{67.75}}}_{\pm 3.21}$ | $\text{{{43.45}}}_{\pm 3.40}$ | $\text{{{55.60}}}_{\pm 0.38}$
XLS-R-300M [11] | 310M |  Multilingual LibriSpeech | No Aug. | $\text{{{69.36}}}_{\pm 2.58}$ | $\text{{{43.23}}}_{\pm 2.06}$ | $\text{{{56.29}}}_{\pm 0.81}$
_RepAugment_ | $\text{{{68.62}}}_{\pm 6.53}$ | $\text{{{44.83}}}_{\pm 4.46}$ | $\text{{{56.73}}}_{\pm 1.19}$

####  IV-C1 Using Pretrained Speech Models

Table I provides an overview of the respiratory classification performance
obtained with various pretrained models and datasets. To ascertain the impact
of architecture and pretraining datasets, we conducted fine-tuning on the
models without other additional learning techniques. The most performant
pretrained speech model is XLS-R-300M, with the ICBHI Score of 56.29%. Despite
employing robust pretrained speech models, the performance in respiratory
sound classification fell short of the initial expectation of at least being
comparable with AST at Score of 59.14%. We hypothesize that the lack of
architecture-specific hyperparameter optimization for AST and other pretrained
speech models may contribute to this as well as data distribution
inconsistency between lung sound and human speech. However, our result reveals
that the performance scales with the pretrained speech model size, as is
evident from the 310M large model outperforming the 95M base model in both
wav2vec2 and HuBERT architectures.

TABLE II: Overall performance on the ICBHI dataset for the official 60–40%
train–test split task of respiratory sound classification. Here, IN, AS, and
MLS refer to ImageNet [14], AudioSet [15], and Multilingual LibriSpeech [19]
together with other speech datasets as described in [11], respectively. Best
and second best are highlighted.

Method | Model | Pretrain | Venue |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---|---
RespireNet [32] (CBA+BRC+FT) | ResNet34 | IN | EMBC‘21 | 72.30 | 40.10 | 56.20
Ren et al. [33] | CNN8-Pt | - | ICASSP‘22 | 72.96 | 27.78 | 50.37
Chang et al. [34] | CNN8-dilated | - | INTERSPEECH‘22 | 69.92 | 35.85 | 52.89
Wang et al. [5] (Splice) | ResNeSt | IN | ICASSP‘22 | 70.40 | 40.20 | 55.30
Moummad et al. [4] (SCL) | CNN6 | AS | WASPAA‘23 | 75.95 | 39.15 | 57.55
Bae et al. [1] (Fine-tuning) | AST | IN + AS | INTERSPEECH‘23 | 77.14 | 41.97 | 59.55
Bae et al. [1] (Patch-Mix CL) | AST | IN + AS | INTERSPEECH‘23 | 81.66 | 43.07 | 62.37
Kim et al. (AFT on Mixed-500) [3] | AST | IN + AS | NeurIPSW‘23 | 80.72 | 42.86 | 61.79
Kim et al. (DAT) [2] | AST | IN + AS | ICASSP‘24 | 77.11 | 42.50 | 59.81
Kim et al. (SG-SCL) [2] | AST | IN + AS | ICASSP‘24 | 79.87 | 43.55 | 61.71
RepAugment [ours] | XLS-R-300M | MLS | - | $\text{{68.62}}_{\pm 6.53}$ | $\text{\bf 44.83}_{\pm 4.46}$ | $\text{56.73}_{\pm 1.19}$
RepAugment [ours] | AST | IN + AS | - | $\text{{82.47}}_{\pm 5.39}$ | $\text{40.55}_{\pm 4.70}$ | $\text{61.51}_{\pm 0.51}$

####  IV-C2 Effectiveness of RepAugment

As shown in Table I, SpecAugment was the most useful for models pretrained on
ImageNet, such as ResNet, EfficientNet, whereas our proposed RepAugment
demonstrated performance gains for all models, and in the case of CNN6
pretrained on AudioSet as well as AST pretrained on ImageNet and AudioSet, it
is the strongest method. Moreover, RepAugment also achieved the best
performance on all pretrained speech models, regardless of the datasets on
which they were pretrained. These findings not only highlight the versatility
of applying our approach across various model input formats but also show the
effectiveness of the proposed RepAugment for pretrained models both on audio
and speech datasets.

####  IV-C3 Comprehensive Comparison

Table II shows an overall comparison of different methods for lung sound
classification on the ICBHI dataset. Scores of 56.73% and 61.51% were achieved
with our proposed method applied to XLS-R-300M and AST, respectively. We would
like to highlight that the proposed RepAugment on the AST achieved a
remarkable improvement of 2.37% over the Score of AST with no augmentation,
and outperformed the Score of AST with SpecAugment by an improvement of 1.96%
as well. Furthermore, results show that our method achieves state-of-the-art
performance, particularly in terms of $S_{p}$ with XLS-R-300M and $S_{e}$ with
AST, respectively. We believe that our method can be helpful when fine-tuning
with various other datasets.

TABLE III: Accuracy (%) of the under-represented classes on the ICBHI test set
for AST and XLS-R-300M fine-tuning under different augmentation strategies.
Bold denotes the best result.

|  |  | Data Augmentation Strategy
---|---|---|---
Model | Class | Ratio (%) | No Aug. | SpecAugment | Rep-Mask | Rep-Gen | RepAugment
AST |  normal ($c_{normal}$) | 57.29 | $\text{76.82}_{\pm 2.47}$ | $\text{{75.52}}_{\pm 7.03}$ | $\text{77.71}_{\pm 6.38}$ | $\text{80.57}_{\pm 2.00}$ | $\text{{82.48}}_{\pm 5.39}$
crackle ($c_{crackle}$) | 23.55 | $\text{51.62}_{\pm 4.17}$ | $\text{{59.32}}_{\pm 3.75}$ | $\text{53.68}_{\pm 5.04}$ | $\text{51.96}_{\pm 3.79}$ | $\text{52.30}_{\pm 3.84}$
wheeze ($c_{wheeze}$) | 13.97 | $\text{35.56}_{\pm 2.87}$ | $\text{33.92}_{\pm 4.35}$ | $\text{{40.15}}_{\pm 3.17}$ | $\text{39.82}_{\pm 3.85}$ | $\text{40.08}_{\pm 4.84}$
both ($c_{both}$) | 5.19 | $\text{14.54}_{\pm 4.71}$ | $\text{18.46}_{\pm 4.73}$ | $\text{17.48}_{\pm 3.62}$ | $\text{20.14}_{\pm 4.93}$ | $\text{{21.68}}_{\pm 4.65}$
XLS-R |  normal ($c_{normal}$) | 57.29 | $\text{68.28}_{\pm 2.89}$ | - | $\text{70.87}_{\pm 9.14}$ | $\text{68.08}_{\pm 1.02}$ | $\text{{70.89}}_{\pm 2.25}$
crackle ($c_{crackle}$) | 23.55 | $\text{52.81}_{\pm 2.76}$ | - | $\text{54.27}_{\pm 8.68}$ | $\text{53.56}_{\pm 2.61}$ | $\text{{56.24}}_{\pm 5.14}$
wheeze ($c_{wheeze}$) | 13.97 | $\text{31.52}_{\pm 4.76}$ | - | $\text{{31.64}}_{\pm 9.80}$ | $\text{{39.74}}_{\pm 6.45}$ | $\text{36.99}_{\pm 5.22}$
both ($c_{both}$) | 5.19 | $\text{18.32}_{\pm 7.97}$ | - | $\text{15.94}_{\pm 3.26}$ | $\text{19.58}_{\pm 6.87}$ | $\text{{24.68}}_{\pm 8.28}$

(a) t-SNE results of pretrained HuBERT-Large

(b) t-SNE results of pretrained XLS-R-300M)

Figure 2: t-SNE results of pretrained HuBERT-Large and XLS-R-300M on
LibriSpeech and ICBHI test sets.

####  IV-C4 Effectiveness of Masking and Noise Generation

To validate the effectiveness of RepAugment, we study how each augmentation
strategy impacts each class of lung sounds. As shown in Table III, _Rep-Mask_
appears to benefit the more represented classes best and contributes to the
overall accuracy. In fact, in the case of XLS-R, small accuracy degradation
occurs for the smallest category $c_{both}$. In the case of AST where
SpecAugment is available, it exhibits a similar pattern as Rep-Mask, where
improvement in the largest class can be observed. _Rep-Gen_ on the other hand
contributes to generalization of the minority classes, and we see the opposite
pattern where the accuracy on the larger category $c_{crackle}$ remains
similar to the baseline. Finally, _RepAugment_ combines Rep-Mask and Rep-Gen
in a synergistic effect that leads to improvements in all categories without
regard to number of samples, but especially in the smallest category.

####  IV-C5 Distribution Gap between Speech and Respiratory Samples

To investigate the reasons behind the lower performance of pretrained speech
models compared to models pretrained on image and audio datasets, we analyze
the data distribution between speech and respiratory samples. Figure 2
provides t-SNE results generated from the output of the XLS-R-300M encoder
that performed best among the pretrained speech models in our experiments
using the LibriSpeech and ICBHI test sets. Notably, the red and green samples
exhibit a distinct separation, indicating that the models, which are primarily
trained on human speech data, significantly differ from respiratory sounds.
This domain gap can lead to challenges in transferring learned knowledge and
achieving optimal performance in lung sound classification tasks.
Consequently, further studies are necessary to enhance the applicability of
pretrained speech models for this task.

##  V CONCLUSIONS

In this paper, we explored the potential of pretrained speech models for
respiratory sound classification. We find that there exists characterization
gap between speech and respiratory sounds, which limits their efficacy.
Further, compared to spectrogram-based approaches that also benefit greatly
from SpecAugment, waveform-based approaches perform poorly. To alleviate the
issue, we propose an input-agnostic data augmentation that operates on the
feature representations of the network. We find that our approach is effective
as data augmentation for waveform-based models and even outperforms
SpecAugment in some cases.

## References

  * [1] S. Bae, J.-W. Kim, W.-Y. Cho, H. Baek, S. Son, B. Lee, C. Ha, K. Tae, S. Kim, and S.-Y. Yun, “Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification,” in _Proc. INTERSPEECH 2023_ , 2023, pp. 5436–5440.
  * [2] J.-W. Kim, S. Bae, W.-Y. Cho, B. Lee, and H.-Y. Jung, “Stethoscope-guided supervised contrastive learning for cross-domain adaptation on respiratory sound classification,” _arXiv preprint arXiv:2312.09603_ , 2023.
  * [3] J.-W. Kim, C. Yoon, M. Toikkanen, S. Bae, and H.-Y. Jung, “Adversarial fine-tuning using generated respiratory sound to address class imbalance,” in _Deep Generative Models for Health Workshop NeurIPS 2023_ , 2023. [Online]. Available: https://openreview.net/forum?id=z1AVG5LDQ7
  * [4] I. Moummad and N. Farrugia, “Pretraining respiratory sound representations using metadata and contrastive learning,” in _2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2023, pp. 1–5.
  * [5] Z. Wang and Z. Wang, “A domain transfer based data augmentation method for automated respiratory classification,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9017–9021.
  * [6] Y. Gong, Y.-A. Chung, and J. Glass, “AST: Audio Spectrogram Transformer,” in _Proc. Interspeech 2021_ , 2021, pp. 571–575.
  * [7] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _Interspeech 2019_ , Sep 2019.
  * [8] N. Zeghidour, N. Usunier, G. Synnaeve, R. Collobert, and E. Dupoux, “End-to-end speech recognition from the raw waveform,” _arXiv preprint arXiv:1806.07098_ , 2018.
  * [9] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in neural information processing systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [10] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [11] A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. von Platen, Y. Saraf, J. Pino _et al._ , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” _arXiv preprint arXiv:2111.09296_ , 2021.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [13] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [15] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _Proc. IEEE ICASSP 2017_ , New Orleans, LA, 2017.
  * [16] B. Rocha, D. Filos, L. Mendes, I. Vogiatzis, E. Perantoni, E. Kaimakamis, P. Natsiavas, A. Oliveira, C. Jácome, A. Marques _et al._ , “A respiratory sound database for the development of automated classification,” in _Precision Medicine Powered by pHealth and Connected Health: ICBHI 2017, Thessaloniki, Greece, 18-21 November 2017_. Springer, 2018, pp. 33–37.
  * [17] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [18] J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen _et al._ , “Libri-light: A benchmark for asr with limited or no supervision,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7669–7673.
  * [19] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A Large-Scale Multilingual Dataset for Speech Research,” in _Proc. Interspeech 2020_ , 2020, pp. 2757–2761.
  * [20] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2017, pp. 23–30.
  * [21] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _arXiv preprint arXiv:1710.09412_ , 2017.
  * [22] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in _International conference on machine learning_. PMLR, 2019, pp. 6438–6447.
  * [23] P. Li, D. Li, W. Li, S. Gong, Y. Fu, and T. M. Hospedales, “A simple feature augmentation for domain generalization,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 8886–8895.
  * [24] K. Zhou, Y. Yang, Y. Qiao, and T. Xiang, “Domain generalization with mixstyle,” _arXiv preprint arXiv:2104.02008_ , 2021.
  * [25] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang, “Cross-domain few-shot classification via learned feature-wise transformation,” _arXiv preprint arXiv:2001.08735_ , 2020.
  * [26] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li, “Deep representation learning on long-tailed data: A learnable embedding augmentation perspective,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2970–2979.
  * [27] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [28] X. Bouthillier, K. Konda, P. Vincent, and R. Memisevic, “Dropout as data augmentation,” _arXiv preprint arXiv:1506.08700_ , 2015.
  * [29] Q. Kong, Y. Cao, T. Iqbal, Y. Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 2880–2894, 2020.
  * [30] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [31] S. wen Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. tik Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H. yi Lee, “SUPERB: Speech Processing Universal PERformance Benchmark,” in _Proc. Interspeech 2021_ , 2021, pp. 1194–1198.
  * [32] S. Gairola, F. Tom, N. Kwatra, and M. Jain, “Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting,” in _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_. IEEE, 2021, pp. 527–530.
  * [33] Z. Ren, T. T. Nguyen, and W. Nejdl, “Prototype learning for interpretable respiratory sound analysis,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9087–9091.
  * [34] Y. Chang, Z. Ren, T. T. Nguyen, W. Nejdl, and B. W. Schuller, “Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis,” in _Proc. Interspeech 2022_ , 2022, pp. 4003–4007.


11institutetext: Georgia Institute of Technology 22institutetext: Google
Research

# Planes vs. Chairs: Category-guided 3D shape learning without any 3D cues

Zixuan Huang 11 Stefan Stojanov 11 Anh Thai 11
Varun Jampani 22 James M. Rehg 11

###### Abstract

We present a novel 3D shape reconstruction method which learns to predict an
implicit 3D shape representation from a single RGB image. Our approach uses a
set of single-view images of multiple object categories without viewpoint
annotation, forcing the model to learn across multiple object categories
without 3D supervision. To facilitate learning with such minimal supervision,
we use category labels to guide shape learning with a novel categorical metric
learning approach. We also utilize adversarial and viewpoint regularization
techniques to further disentangle the effects of viewpoint and shape. We
obtain the first results for large-scale (more than 50 categories) single-
viewpoint shape prediction using a single model without any 3D cues. We are
also the first to examine and quantify the benefit of class information in
single-view supervised 3D shape reconstruction. Our method achieves superior
performance over state-of-the-art methods on ShapeNet-13, ShapeNet-55 and
Pascal3D+.

##  1 Introduction

Figure 1: (a) We present the first method to learn 3D shape reconstruction
from single-viewpoint images over _multiple object categories simultaneously_
, without 3D supervision or viewpoint annotations. (b) To facilitate shape
learning under such a challenging scenario, We leverage category labels to
perform metric learning in the shape embedding space. For each category, we
learn a shape center. Then with a given sample, we minimize the distance
between its shape embedding and the corresponding category center, while
contrasting this distance with other inter-category distances.

Reconstructing the 3D shape of objects from 2D images is a fundamental
computer vision problem. Recent works have demonstrated that high-quality 3D
shape reconstructions can be obtained from a single RGB input image [4, 7, 10,
33]. In the most straight-forward approach, explicit 3D supervision from
ground-truth (GT) object shapes is used to train a shape reconstruction model
[48, 33, 44, 56]. While this approach yields strong performance, GT 3D shape
is difficult to obtain on a large scale. This limitation can be addressed via
multi-view supervision [57, 50, 58, 27, 19, 23, 29, 41], in which the learner
optimizes a reprojection loss over a set of viewpoints. This approach becomes
more difficult when fewer viewpoints are available during training. The
limiting case is single-view supervision, which is challenging due to the
well-known entanglement between shape and pose.111When only a single view of
each object instance is available for training, there are an infinite number
of possible 3D shapes that could explain the image under appropriate camera
viewpoints. Prior works (see Table 1) have addressed this challenge in two
ways. The first approach assumes that the camera viewpoint is known at
training time [28, 22]. While this strong assumption effectively reduces
entanglement, it is not scalable as it requires pose annotations.

A second line of attack trains separate models for each category of 3D objects
[20, 49, 8, 26, 16, 15, 60, 40, 54]. This single category approach provides
the learner with a strong constraint – each model only needs to learn the
“pattern” of shape and pose entanglement associated with a single object
category. This approach has the disadvantage that data cannot be pooled across
categories: pooling data can be beneficial to tasks such as generalization to
unseen categories of objects (zero-shot [21, 62] or few-shot [58]) and
viewpoint learning. For example, tables and chairs have many features (e.g.
legs) in common, and a multi-class reconstruction model could leverage such
similarities in constructing shared feature representations.

We introduce a novel _multi-category, single-view (MCSV)_ 3D shape learning
method, shown in Fig. 1 (a), that does not use shape or viewpoint supervision,
and can learn to represent multiple object categories within a single
reconstruction model. Our method exploits the observation that shapes within a
category (e.g. planes) are more similar to each other than they are to the
shapes in other categories (e.g. planes vs. chairs). We use a shape metric
learning approach, illustrated in Fig. 1 (b), which simultaneously learns
category shape embeddings (category centroids denoted as green/blue dots)
together with instance shape embeddings (green/blue triangles) and minimizes
the distance between shape instances and their corresponding category centers
(solid arrow) while maximizing the distance to the other categories (dotted
arrow). This method has two benefits. First, the learned shape manifold
captures a strong prior knowledge about shape distance, which helps to
eliminate erroneous shapes that might otherwise explain the input image.
Second, similar shapes tend to be clustered together, enabling the supervision
received by a particular shape to affect its neighbors, and helping to
overcome the limitation of having only a single viewpoint. As a consequence of
this approach, we are able to train a single model on all 55 object categories
in ShapeNet-55, which has not been accomplished in prior work.

Another important aspect of our approach is that we represent object shape as
an implicit signed distance field (SDF) instead of the more commonly used mesh
representation, in order to accommodate objects with varying topologies. This
is an important goal in scaling MCSV models to handle large numbers of object
categories. Specifically, our model (Fig. 1 (a)) takes a single input image
and produces an implicit SDF function for the 3D object shape. In contrast to
mesh and voxel representations, SDF representations create challenges in
training because they are less constrained and require a more complex
differentiable rendering pipeline. A beneficial aspect of our approach is the
use of adversarial learning [16, 60] and cycle-consistency [39, 60] for
regularization.

In summary, our 3D reconstruction learning approach has the following
favorable properties:

  * •

Extremely weak supervision. We use only single-view masked images with
category labels during training (considered as ‘self-supervised’ in prior
works).

  * •

Categorical shape metric learning. We make effective use of class labels with
our novel shape metric learning.

  * •

Single template-free multi-category model. We learn a single model to
reconstruct multiple object categories without using any category-specific
templates.

  * •

Implicit SDF representation. We use implicit SDFs, instead of meshes, to
represent diverse object topologies.

  * •

Order of magnitude increase in object categories learned in a single model.
Due to the scalability of our approach, we are the first to train a MCSV model
on all 55 categories of ShapetNetCoreV2 [2].

  * •

State-of-the-art reconstructions. Experiments on both ShapeNet [2] and
Pascal3D+ [55] demonstrate consistently better performance than previous
methods.

##  2 Related Work

Single-View Supervision. The body of work that is most closely-related to this
paper are methods that use single-view supervision to learn 3D shape
reconstruction with a back-projection loss [20, 49, 8, 26, 16, 15, 6, 60, 40,
22, 54, 28]. These works can be organized as depicted in Table 1 and largely
differ in their choice of 1) learning a single multi-category (multi-class) vs
multiple single-category models for reconstruction; 2) shape representation
(e.g. implicit SDF vs explicit mesh); 3) known vs. unknown viewpoint
assumption. We are the first to demonstrate the feasibility of single-view,
multi-category learning of an implicit shape representation (SDF) under the
unknown viewpoint condition via the use of category labels. We are also the
first to provide results on all of the ShapeNet-55 classes in a single model,
an order of magnitude more than prior works.

Table 1: Single-view supervised methods for shape reconstruction. For methods without supervision, they are mainly verified under a limited setting [15] or for specific objects [54]. K: keypoints, T: category templates, M: masks, m: mesh, vox: voxel, pc: pointcloud, im: implicit representation. Model | [20] | [49] | [8] | [26] | [6] | [16] | [15] | [60] | [40] | [28] | [54] | [22] | [45] | Ours
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Multi-Class Rec. | - | - | - | - | - | - | - | - | - | - | - | ✓ | ✓ | ✓
Supervision | K,M | M,T | M,T | M | M | M | - | M | M | M,V | - | M,V | M,V | M
3D Rep. | m | m | m | m | vox | vox | m | m | pc | im | d | m | m | im

Within this body of work [22, 28, 60] and the concurrent work of [45] are the
most closely related. Kato et. al. [22] (VPL) and Simoni et. al. [45] are the
only prior works to address multi-category shape learning with a single model,
but both assume access to ground truth (GT) viewpoint and use mesh
representations. Similar to our work, Ye et. al. [60] do not use viewpoint GT,
but they train one model for each object category. Further, their mesh
prediction is refined from a low-resolution voxel prediction by optimizing on
each individual sample. The latter is less efficient than feedforward models
and can potentially harm the prediction of concave shapes or details. In
contrast with [23, 45, 60] we use an implicit shape representation. Further,
as shown in Table 1, some earlier approaches use fixed meshes as deformable
shape templates. In contrast with implicit functions, meshes with fixed
topology restrict the possible 3D shapes that can be accurately represented.

Lin et. al. [28] is the only prior work to use an implicit SDF representation
for single view supervision, and we adopt their SDF-SRN network architecture
and associated reprojection losses in our formulation. This work trains
single-category models and assumes that the camera viewpoint is known, whereas
one of the main goals of our work is to remove these assumptions and to
demonstrate that the unconstrained SDF representation can be successful in the
multi-category, unknown viewpoint case. We consistently outperform a version
of SDF-SRN modified for our setting in Table 2, which highlights our
innovations.

Shape Supervision. Prior works on single image 3D reconstruction with explicit
3D geometric supervision have achieved impressive results [4, 7, 10, 33, 56].
However, the requirement of 3D supervision limits the applicability of these
methods. To overcome this, subsequent works employ multi-view images for
supervision. The use of image-only supervision is enabled by differentiable
rendering, which enables the generation of 2D images/masks from the predicted
3D shape and the comparison to the ground truth reference images as
supervision. These methods can be grouped by their representation of shape,
including voxels [57, 50, 58], pointclouds [27, 19], meshes [23, 29] and
implicit representation [41]. In contrast to these works, we allow only _a
single image per object instance_ in the training dataset.

Category Information. Few prior works have explored using category-specific
priors for few-shot shape reconstruction [51, 34], where they assume voxel
templates are available for each category. Our method uses category priors for
shape reconstruction, but by only leveraging the significantly weaker
supervision of category labels.

Deep Metric Learning. To harness shape learning with category labels, our
method also shares ideas with metric learning. The key idea of metric learning
is to learn an embedding space where similar instances are together and
dissimilar instances are far according to some distance metric. This paradigm
has shown success in self-supervised [3, 13] and supervised [24] learning, as
well as many downstream tasks e.g. person re-identification [17]. We use the
idea of metric learning for single-view shape reconstruction and demonstrate
its effectiveness for the first time.

##  3 Approach

Figure 2: Approach Overview. Given the input image, the encoders
$\boldsymbol{E}$ and $\boldsymbol{V}$ predicts the shape $\boldsymbol{s}$ and
texture $\boldsymbol{t}$ codes and viewpoint $\boldsymbol{v}$. The
hypernetwork $\boldsymbol{H}$ takes both latent codes and outputs the
parameters $\boldsymbol{\theta_{S}}$, $\boldsymbol{\theta_{T}}$ of the
implicit functions $\boldsymbol{f_{S}}$ and $\boldsymbol{f_{T}}$. These
functions are sampled with learnable ray tracer $\boldsymbol{R}$ to render the
image with the predicted viewpoint $\boldsymbol{v}$ and thus to compute the
reconstruction loss. We also render an extra image from a random viewpoint for
adversarial and viewpoint regularizations. Our shape metric learning technique
is performed over the shape latent code.

Given a collection of $n$ images concatenated with their masks
$\\{I_{i}\in\mathbb{R}^{h\times w\times 4}\\}_{i=1}^{n}$ and class labels
$\\{\mathbf{y}_{i}\in\\{0,1,\cdots,c\\}\\}$, our goal is to learn a single-
view 3D reconstruction model without any 3D, viewpoint, or multi-view
supervision. In contrast to most existing work, we learn a single network that
works across all object categories. We represent shape using an implicit shape
representation function
$\boldsymbol{f_{S}}:\mathbb{R}^{3}\rightarrow\mathbb{R}$, a multi-layer
perceptron (MLP) that maps 3D coordinates to signed distance function (SDF)
values $s\in\mathbb{R}$. Following a standard framework for learning 3D shape
via differentiable rendering, our model first infers shape, texture and
viewpoints from the input images, which are then rendered into images. The
rendered image is then compared to the input image, providing a training
signal for model learning. However, in our challenging multi-category setting,
without viewpoint or 3D supervision, providing supervision only by comparing
the rendered image and input images of the object results in poor performance.
To mitigate this, we propose a set of regularization techniques which improves
reconstruction quality and stabilizes model training. We first present an
overview of different modules of our approach in Section 3.1, and then
introduce our category-based shape metric learning in Section 3.2. Finally, we
present other regularization methods in Section 3.3. Implementation details
and license are described in the appendix.

###  3.1 Network Overview

Our model is trained in an end-to-end manner and can be decomposed into four
main network modules (Fig. 2).

Image encoder. The image encoder $\boldsymbol{E}$ maps an image with mask
channel $I\in\mathbb{R}^{h\times w\times 4}$ to a latent shape vector
$\boldsymbol{s}\in\mathbb{R}^{l}$ for the downstream shape predictor, and a
latent texture vector $\boldsymbol{t}\in\mathbb{R}^{l}$ for texture predictor.

Shape and texture prediction module. Following the design of hypernetworks
[11, 28], the shape and texture prediction module uses latent codes
$\boldsymbol{s}$ and $\boldsymbol{t}$ to predict the parameters of shape and
texture implicit functions $\boldsymbol{f_{S}}$ and $\boldsymbol{f_{T}}$ that
map 3D query points $\boldsymbol{x}$ to SDF and texture predictions:

$\displaystyle{\boldsymbol{\theta_{S}},\boldsymbol{\theta_{T}}}\leftarrow\boldsymbol{H}(\boldsymbol{s},\boldsymbol{t};\boldsymbol{\theta_{H}})$
(1)
$\displaystyle\boldsymbol{f_{S}}:(\boldsymbol{x};\boldsymbol{\theta_{S}})-\text{SDF
prediction}$ (2)
$\displaystyle\boldsymbol{f_{T}}:(\boldsymbol{x};\boldsymbol{\theta_{T}})-\text{Texture
prediction}$ (3)

$\boldsymbol{f_{S}}$ and $\boldsymbol{f_{T}}$ are MLPs with predefined
structure and parameters estimated by the hypernet $\boldsymbol{H}$ with
parameters $\boldsymbol{\theta_{H}}$.

Viewpoint prediction module. In our model, the viewpoint is represented as the
trigonometric functions of Euler angles for continuity [1], i.e.
$\boldsymbol{v}=[\cos\boldsymbol{\gamma},\sin\boldsymbol{\gamma}]$ with
$\boldsymbol{\gamma}$ denoting 3 Euler angles. The viewpoint prediction
network $\boldsymbol{V}(I;\boldsymbol{\theta_{V}})$ predicts the viewpoint
$\boldsymbol{v}\in\mathbb{R}^{6}$ of the object in the input image $I$ with
regard to a canonical pose (can be different from the human-defined canonical
pose), where $\boldsymbol{\theta_{V}}$ are the learnable parameters.

Differentiable renderer. We use an SDF-based implicit differentiable renderer
from [28] which takes the shape, texture and the viewpoint as inputs and
renders the corresponding RGB image. Formally, we denote it as a functional,
$\boldsymbol{R}(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v};\boldsymbol{\theta_{R}})$,
which maps shape SDF, texture implicit function and the viewpoint into 2D RGB
image with an alpha channel, in $\mathbb{R}^{h\times w\times 4}$. The renderer
itself is also learnable with $\boldsymbol{\theta_{R}}$ parameters. Note that
as the renderer in [28] cannot render masks, we modify it to render an extra
alpha channel from SDF field following [59].

Before we describe different regularizations, we briefly discuss the major
challenge in learning our model. During training, the only supervision signals
are the input images and the masks of the objects. The common approach is to
minimize a reconstruction loss via differentiable rendering222Masks are
leveraged with additional losses that enforce consistency between the shape
and the input mask. For conciseness, we omit them here and refer readers to
[28] for more details.

$\mathcal{L}_{recon}=\|I-\boldsymbol{R}(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v};\boldsymbol{\theta_{R}})\|^{2}_{2}.$
(4)

This learning scheme works well with multi-view supervision [41], or even
single-view images with viewpoint ground truth [28]. However, when there is
only single-view supervision without viewpoint ground truth, this is
significantly more difficult. There are infinite combinations of shapes and
viewpoints that perfectly render the given image, and the model does not have
any guidance to identify the correct shape-viewpoint combination. To tackle
this problem, we utilize category labels to guide the learning of shape via
metric learning. We also make use of other regularizations including
adversarial learning and cycle consistency to facilitate learning.

###  3.2 Shape metric learning with category guidance

Our novel shape metric learning approach is the key ingredient that enables us
to train a single model containing 55 shape categories. The key idea is to
learn a metric which maps shapes within the same category (e.g. chairs) to be
close to each other, while mapping shapes in different categories (e.g. planes
vs. chairs) to be farther apart. This approach leverages qualitative label
information to obtain a continuous metric for shape similarity, resulting in
two benefits. First, the shape manifold defined by the learned metric
constrains the space of possible 3D shapes, helping to eliminate spurious
shape solutions that would otherwise be consistent with the input image (see
Fig. 3 (a) for an example). Second, the shape manifold tends to group similar
shapes together, creating a neighborhood structure over the training samples.
For example, without category guidance it can be difficult for the network to
learn the relationship between two images of different chairs captured from
different, unknown, camera viewpoints. However, with the learned shape
manifold these instances are grouped together, facilitating the sharing of
supervisory signals from the rendering-based losses. A qualitative
reconstruction comparison in Fig. 3 (a) and t-SNE visualizations of shape
embedding space in Fig. 3 (b) provide further insight into these benefits.
Note that as the image discriminator used for adversarial regularization uses
images and labels as input, category information is still available during
training even without metric learning. Therefore, the current comparison
demonstrates the effectiveness of metric learning for further utilizing
category label information.

Figure 3: (a) Category-guided metric learning is beneficial for eliminating
erroneous shapes that can explain input image. In the first view (close to
input viewpoint), both shapes looks plausible and can explain input image. But
from another view, it is clear that our metric-learned shape is more
reasonable as a chair, while the baseline can be closer to a table/nightstand.
(b) t-SNE visualization of the shape embedding space on ShapeNet-55. Category-
guided metric learning results in a more structured representation and
facilitates joint learning.

Formally, for a batch of $n$ images ${\\{I_{i}\\}}_{i=1}^{n}$, we extract both
shape latent codes and texture latent codes via an image encoder
$\boldsymbol{E}$. For the $i^{th}$ sample, we denote the latent shape code as
$\boldsymbol{s}_{i}$ and the category label as $y_{i}\in\mathcal{C}$ where
$\mathcal{C}$ is the set of integer labels. There are two types of losses
commonly used for metric learning: embedding losses [12, 53, 42] that compare
different instances (e.g., triplet loss), and proxy-based losses [52, 30, 61]
which learn parameters representing categories (e.g. weight vectors in
normalized softmax loss or ProxyNCA [38]). As the embedding losses usually
require specific sampling strategies to work well, we follow the latter for
metric learning in this work.

Specifically, for each category we define a learnable category center vector
(or category embedding) ${\\{\boldsymbol{c}_{k}\\}}_{k=1}^{c}$ and optimize a
distance metric so that instances of the same category are close and instances
of different categories are far apart, as illustrated in the green shaded
region of Fig. 2 and Fig. 1(b). We optimize the following loss between learned
shape instance embeddings $\boldsymbol{s}$ and category center embeddings
$\boldsymbol{c}$:

$\mathcal{L}_{metric}=-\sum_{i=0}^{N}log\frac{exp(d(\boldsymbol{s}_{i},\boldsymbol{c}_{y_{i}})/\tau)}{\sum_{k\in\mathcal{C}}exp(d(\boldsymbol{s}_{i},\boldsymbol{c}_{k})/\tau)},$
(5)

where $d$ is a similarity measure and $\tau$ is the temperature. Following
normalized softmax loss [52] we use cosine similarity as a similarity measure,

$d(\boldsymbol{s},\boldsymbol{c})=\frac{\boldsymbol{s}\cdot\boldsymbol{c}}{\|\boldsymbol{s}\|\|\boldsymbol{c}\|}.$
(6)

The centers are randomly initialized from a uniform distribution and updated
directly through gradient descent. We set temperature $\tau$ to be 0.3 across
all our experiments.

###  3.3 Shape and viewpoint regularization

Adversarial regularization. To further facilitate learning, we use adversarial
regularization to ensure that predicted shapes and texture fields will render
realistic images from any viewpoints in addition to the input view. Given the
training image collection, we estimate the data manifold via an image
discriminator. If the rendered images also lie on the training data manifold,
most erroneous shapes and texture predictions will be eliminated. To achieve
this, similar to [16, 60], we use a discriminator with the adversarial
training [9]. Furthermore, because our multi-category setting results in a
complex image distribution, we condition our discriminator on category labels
for easier training following [37]. While our work is not the first work to
use GANs for shape learning in general, we believe it still provides useful
new insights by combining adversarial training with unconstrained implicit
representation and class conditioning under our challenging MCSV setting.

Formally, suppose
$I_{recon}=\boldsymbol{R}(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v})$
is the output of the renderer for estimated shape, texture and viewpoint given
an input image. We sample another random viewpoint $\boldsymbol{v}^{\prime}$
from a prior distribution $P_{v}(\boldsymbol{v})$ and render another image,
$I_{rnd}=R(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v}^{\prime})$
from this viewpoint. We match the distribution of ${I}$ and
${I_{recon},I_{rnd}}$ with adversarial learning as shown in Fig. 2.
Specifically, we optimize the objective $\mathcal{L}_{gan}$ below by
alternatively updating our model (the
$\boldsymbol{E},\boldsymbol{H},\boldsymbol{R},\boldsymbol{V}$ networks) and
the discriminator $\boldsymbol{D}(I;\boldsymbol{\theta_{D}})$

$\min_{\boldsymbol{\theta_{E}},\boldsymbol{\theta_{H}},\boldsymbol{\theta_{V}},\boldsymbol{\theta_{R}}}\max_{\boldsymbol{\theta_{D}}}\mathcal{L}_{gan}=\mathbb{E}[\log\boldsymbol{D}(I)]+\mathbb{E}[\log(1-\boldsymbol{D}([I_{recon},I_{rnd}]))].$
(7)

Here, $[I_{recon},I_{rnd}]$ is the stack of reconstructed and randomly
rendered images (on batch dimension). For the update step of the
reconstruction model, we follow a non-saturating scheme [9] where we maximize
$\mathbb{E}[\log(\boldsymbol{D}([I_{recon},I_{rnd}]))]$ instead of minimizing
$\mathbb{E}[\log(1-\boldsymbol{D}([I_{recon},I_{rnd}]))]$. We also use $R_{1}$
regularizer [32] and spectral normalization [36] to stabilize the training
process.

Viewpoint regularization via cycle-consistency. We regularize the viewpoint
prediction module via cycle-consistency [63], with a similar approach as the
state-of-the-art at self-supervised viewpoint estimation [39] and Ye et al.
[60]. However, they both rely on a strong shape symmetry assumptions to
facilitate the learning and Ye et al. further utilize shape embedding in the
cycle. In our approach, we do not use any such restrictive assumptions, and we
only consider viewpoint itself in this regularization.

Figure 4: Viewpoint regularization of our method. With a given shape and
texture field, we sample a random viewpoint and render an image as the input
to our viewpoint predictor. The viewpoint predictor is then supervised with
the randomly sampled viewpoint, which forms a consistency cycle.

The key idea of viewpoint regularization is that given an image, we require
that the viewpoint predictor accurately predicts the viewpoint of this image.
Different from real images, we can render an arbitrary number of images by
sampling viewpoints with a given shape and texture. This can be thought of as
creating pseudo data-label pairs for the viewpoint predictor.

Formally, given shape $\boldsymbol{f_{S}}$, texture field $\boldsymbol{f_{T}}$
and a random viewpoint $\boldsymbol{v_{r}}$, we render an image
$\hat{I}=\boldsymbol{R}(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v_{r}})$.
The goal is to minimize the distance between $\boldsymbol{v_{r}}$ and
$\hat{\boldsymbol{v_{r}}}=\boldsymbol{V}(\hat{I})$. With the trigonometric
function representation, we maximize the cosine similarity between
$\boldsymbol{v_{r}}$ and $\hat{\boldsymbol{v_{r}}}$ by minimizing

$\mathcal{L}_{cam}=1-\langle\boldsymbol{v_{r}}\,,\hat{\boldsymbol{v_{r}}}\rangle=1-\langle\boldsymbol{v_{r}}\,,\boldsymbol{V}(\boldsymbol{R}(\boldsymbol{f_{S}},\boldsymbol{f_{T}},\boldsymbol{v_{r}}))\rangle,$
(8)

where $\langle\cdot\,,\cdot\rangle$ denotes dot product. In practice, we apply
this regularization on the reconstructed image $I_{recon}$ and the randomly
rendered image $I_{rnd}$, together with the viewpoints that render them. This
is computationally efficient as both images are also used in other losses. We
use $\mathcal{L}_{cam}$ only to train the viewpoint prediction module, as we
stop the gradients to shape, texture and rendering modules.

##  4 Experiments

This section presents the empirical findings of applying our method to
multiple synthetic and real datasets, as well as ablations of its individual
components. An additional goal is to quantify the value of category labels in
the MCSV setting. By comparing category label-based supervision with a
standard two-view reconstruction approach, we find that the value of the
category label is approximately equal to 20% of an additional view. See the
appendix for details. Following an overview of the datasets, metrics and
baselines, we give results on synthetic and real data.

###  4.1 Datasets

For synthetic data we use two splits of the ShapeNet [2] dataset, and for real
data we use Pascal3D+[55].

ShapeNet-13. This dataset consists of images from the 13 biggest categories of
ShapeNet, originally used by Kato et al. [23] and following works learning 3D
shape without explicit 3D supervision [41, 28]. 24 views per object are
generated by placing cameras at a fixed $30^{\circ}$ elevation and with
azimuthal increments around the object of $15^{\circ}$. Following SDF-SRN
[28], we use a 70/10/20 split for training, validation and testing, resulting
in 30643, 4378 and 8762 objects respectively. We use only one view per object
instance, which is randomly pre-selected out of the 24 views for each object.

ShapeNet-55. To create a more challenging setting in comparison to prior work,
we use all of the categories of ShapeNet.v2. This is challenging due to (1)
approximately four times more shape categories; and (2) uniformly sampling at
random the azimuth in the $[0^{\circ},360^{\circ}]$ range and elevation in
$[20^{\circ},40^{\circ}]$. As a result, the image and shape distributions that
the model needs to learn are significantly more complex. To reduce the data
imbalance, we randomly sample at most 500 objects per category. Like
ShapeNet13 we use only one image per object instance and a 70/10/20 split,
with 15031, 1705, 3427 images respectively.

Figure 5: Qualitative comparison on ShapeNet-13. Our method learns both better
global structures and details on various categories.

Pascal 3D+. This dataset contains real-world images with 3D ground truth
obtained by CAD model alignment. Its challenges compared to ShapeNet come from
(1) inaccurate object masks that include noisy backgrounds—an especially
difficult setting for adversarial learning; and (2) the viewpoints vary in
azimuth, elevation and tilt, creating challenges because different categories
have different and unknown viewpoint distributions. We combine the commonly
used motorcycle and chair categories for a total of 2307 images, and as in
[28] we use the ImageNet [5] subset of Pascal3D+ with 1132/1175 images for
training/testing.

Evaluation Metrics We use Chamfer Distance (CD) and F-score under different
thresholds to evaluate shape reconstruction performance following [28, 10, 47,
48]. We use the Marching Cubes algorithm [31] to convert the implicit
representation to meshes prior to computing the metrics. Shapes are
transformed into camera coordinates for evaluation, see the appendix for
details.

Chamfer Distance. Following [41, 28], CD is defined as an average of accuracy
and completeness. Given two surface point sets $S_{1}$ and $S_{2}$, CD is
given as:

$d_{CD}(S_{1},S_{2})=\frac{1}{2|S_{1}|}\sum_{x\in S_{1}}\min_{y\in
S_{2}}\|x-y\|_{2}+\frac{1}{2|S_{2}|}\sum_{y\in S_{2}}\min_{x\in
S_{1}}\|x-y\|_{2}$ (9)

F-score. Compared to CD which accumulates distances, F-score measures accuracy
and completeness by thresholding the distances. For a threshold $d$
(F-Score@$d$), precision is the percentage of predicted points that have
neighboring ground truth points within distance $d$, recall is the percentage
of ground truth points that have neighboring predicted points within $d$.
F-score, the harmonic mean of precision and recall, can be intuitively
interpreted as the percentage of surface that is correctly reconstructed.

###  4.2 Baselines

There are no directly comparable state-of-the-art methods under our MCSV
setting. Instead, we slightly alter the setting of two recent methods SDF-SRN
[28] and Ye et al. [60] to compare with our model.

SDF-SRN [28] learns to predict implicit shapes using single images and known
camera poses in training. For comparison purposes, we attach our viewpoint
prediction module to SDF-SRN and train it across multiple categories
simultaneously. We compare to SDF-SRN on all datasets.

Ye et al. [60] learns category-specific voxel prediction without viewpoint
GT.333Their method has an optional per-sample test-time optimization to
further refine the voxels and convert into meshes. Using the authors’
implementation of this optimization did not lead to improved results in our
experiments, so we use the voxel prediction as the output for evaluation. We
compare our method with Ye et al. on Pascal3D+.444Evaluating it on ShapeNet is
quite challenging as it requires training numerous category-specific models
with shape alignment for each one. For Ye et al., we train different models
for different categories following their original setting and compute the
average metric over categories. We use GT masks as supervision for all
experiments.

###  4.3 ShapeNet-13

We perform experiments on ShapeNet-13 and show quantitative and qualitative
results in Table 2 and Figure 5.

Table 2: Quantitative result measured by CD and F-score on ShapeNet-13. Our method performs favorably to baselines and other SOTA methods.  Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  CD$\downarrow$
---|---|---|---|---
w/o category | 0.1589 | 0.6261 | 0.8527 | 0.520
w/o $\mathcal{L}_{metric}$ | 0.1875 | 0.6864 | 0.8805 | 0.458
w/o $\mathcal{L}_{cam}$ | 0.1837 | 0.6741 | 0.8758 | 0.463
w/o $\mathcal{L}_{gan}$ | 0.1846 | 0.6437 | 0.8422 | 0.532
Ours | 0.2005 | 0.7168 | 0.8949 | 0.430
SDF-SRN | 0.1606 | 0.5441 | 0.7584 | 0.682

Ablation Study. We first analyze the results of ablating the multiple
regularization techniques used in our approach. In Table 2, ‘w/o category’
shows the results of the model without any category information and ‘w/o
$\mathcal{L}_{metric}$’ is the model with GAN conditioning only, by removing
the metric learning component. Comparing them to our full model, we clearly
see our metric learning helps utilize category information in a more efficient
way (w/o $\mathcal{L}_{metric}$ vs Ours). We also see the benefit of having
category information for shape reconstruction tasks (w/o category vs Ours). We
further ablate on our camera (w/o $\mathcal{L}_{cam}$ vs Ours) and GAN
regularization (w/o $\mathcal{L}_{gan}$ vs Ours) and quantify their value. Our
qualitative result verifies these findings as well. We include more
quantitative findings about the value of category labels for shape learning in
the appendix.

SOTA Comparison. Comparing with SDF-SRN in Table 2 we see that our approach
improves on the adapted SDF-SRN by a 23.8% (w/o category) and 37.0% (with
category, Ours) CD decrease. Qualitatively, our approach captures both the
overall shape topology and details (legs of table in the bottom row) whereas
SDF-SRN fails. These all demonstrated the effectiveness of our proposed model.

###  4.4 ShapeNet-55

Compared to ShapeNet-13, ShapeNet-55 is significantly more challenging due to
both the number of categories and viewpoint variability. For results see Table
3 and Fig. 6 (a).

Table 3: Quantitative result measured by CD and F-score on ShapeNet-55. Our method performs favorably to baselines and other SOTA methods.  Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  CD$\downarrow$
---|---|---|---|---
w/o category | 0.0977 | 0.4365 | 0.6815 | 0.801
w/o $\mathcal{L}_{metric}$ | 0.1431 | 0.5758 | 0.8047 | 0.620
Ours | 0.1619 | 0.6164 | 0.8386 | 0.541
SDF-SRN [28] | 0.0707 | 0.2750 | 0.4806 | 1.172
Figure 6: Qualitative comparison on ShapeNet55 and Pascal3D+. Our method
learns both better global 3D structure and shape details on various
categories.

Ablation Study. On this dataset, we again ablate on the value of category
labels and our proposed shape metric learning. Similar to ShapeNet-13, we
clearly see our metric learning method utilizes category information more
effectively than GAN conditioning only (w/o $\mathcal{L}_{metric}$ vs Ours in
Table 3). Meanwhile, we demonstrate the huge benefit of having category
information for shape reconstruction on this dataset (w/o category vs Ours,
32.4% CD decrease). This is much more significant than ShapeNet-13 because
ShapeNet-55 has much higher data complexity, where a better structured shape
latent space can be quite beneficial to shape learning. Qualitatively, without
category regularization the model cannot capture even the global shape
successfully, whereas without shape metric learning it only partially succeeds
at highly symmetric objects (two vases in Figure 6 (a)). For others (e.g.
bench at row 2) its reconstruction can only explain the input view. This
further strengthen the discussion in Fig. 3 (a), where shape metric learning
can help eliminate erroneous shape predictions.

SOTA Comparison. Comparing with row 4 of Table 3 we see that our approach
improves on the adapted SDF-SRN by 31.6% (without category) and 53.8% (with
category) CD decrease. The significant performance improvement justifies the
effectiveness of our overall framework, as well as the effectiveness of the
categorical shape metric learning. Qualitatively, SDF-SRN collapses and fails
to capture the topology of most inputs, while our method with still
demonstrate a great qualitative reconstruction performance. More qualitative
results on various categories and reconstructed textures are included in the
appendix.

###  4.5 Pascal3D+

We compare our method to SDF-SRN [28] and Ye et al. [60] on Pascal3D+, as
shown in Table 4 and Fig. 6 (b). It is clear that our method outperforms the
SOTA on both metrics, and our metric learning method is critical to the good
performance. We also compare reconstructions qualitatively. As shown in Fig. 6
(b), SDF-SRN collapses to a thin flake that can only explain input images,
while Ye et al. lack some important details. This further verifies the
effectiveness of our proposed methods.

Table 4: Quantitative result measured by CD and F-score on Pascal3D+. Our method performs favorably to other SOTA methods.  Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  CD$\downarrow$
---|---|---|---|---
SDF-SRN [28] | 0.0954 | 0.4656 | 0.7474 | 0.777
Ye et al. [60] | 0.1195 | 0.5429 | 0.8252 | 0.625
Ours w/o $\mathcal{L}_{metric}$ | 0.1038 | 0.5108 | 0.7799 | 0.673
Ours | 0.1139 | 0.5460 | 0.8455 | 0.580

##  5 Limitation and Discussion

One limitation is that reconstruction accuracy decreases for some samples with
large concavity. For example, for bowls or bathtubs on ShapeNet-55, our method
cannot capture the concavity of the inner surface. Concavity is hard to model
without explicit guidance, and this can potentially be improved by explicitly
modeling lighting and shading. On the other hand, ShapeNet-55 represents a
class imbalance challenge, where images in different categories range from 40
to 500 images. This makes learning on rare classes difficult.

We also observe a training instability of our model caused by the usage of
adversarial regularization. Meanwhile, we think our shape metric learning can
be further improved by explicitly modeling the multi-modal nature of some
categories. This could be achieved by using several proxies for each category.
We leave this to future work. Please see appendix for more discussion and
qualitative examples on limitations.

##  6 Conclusion

We have presented the first 3D shape reconstruction method with an SDF shape
representation under the challenging _multi-category, single-view (MCSV)_
setting without viewpoint supervision. Our method leverages category labels to
guide implicit shape learning via a novel metric learning approach and
additional regularization. Our results on ShapeNet-13, ShapeNet-55 and
Pascal3D+, demonstrate the superior quantitative and qualitative performance
of our method over prior works. Our findings are the first to quantify the
benefit of category information in single-image 3D reconstruction.

## References

  * [1] Beyer, L., Hermans, A., Leibe, B.: Biternion nets: Continuous head pose regression from discrete training labels. In: German Conference on Pattern Recognition. pp. 157–168. Springer (2015)
  * [2] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  * [3] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [4] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In: European conference on computer vision. pp. 628–644. Springer (2016)
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [6] Gadelha, M., Maji, S., Wang, R.: 3d shape induction from 2d views of multiple objects. In: 2017 International Conference on 3D Vision (3DV). pp. 402–411. IEEE (2017)
  * [7] Gkioxari, G., Malik, J., Johnson, J.: Mesh r-cnn. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [8] Goel, S., Kanazawa, A., Malik, J.: Shape and viewpoint without keypoints. In: European Conference on Computer Vision. pp. 88–104. Springer (2020)
  * [9] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)
  * [10] Groueix, T., Fisher, M., Kim, V.G., Russell, B., Aubry, M.: AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [11] Ha, D., Dai, A., Le, Q.V.: Hypernetworks. arXiv preprint arXiv:1609.09106 (2016)
  * [12] Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). vol. 2, pp. 1735–1742. IEEE (2006)
  * [13] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9729–9738 (2020)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] Henderson, P., Ferrari, V.: Learning single-image 3d reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision pp. 1–20 (2019)
  * [16] Henzler, P., Mitra, N.J., Ritschel, T.: Escaping plato’s cave: 3d shape from adversarial rendering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9984–9993 (2019)
  * [17] Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017)
  * [18] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [19] Insafutdinov, E., Dosovitskiy, A.: Unsupervised learning of shape and pose with differentiable point clouds. arXiv preprint arXiv:1810.09381 (2018)
  * [20] Kanazawa, A., Tulsiani, S., Efros, A.A., Malik, J.: Learning category-specific mesh reconstruction from image collections. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 371–386 (2018)
  * [21] Kar, A., Häne, C., Malik, J.: Learning a multi-view stereo machine. arXiv preprint arXiv:1708.05375 (2017)
  * [22] Kato, H., Harada, T.: Learning view priors for single-view 3d reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9778–9787 (2019)
  * [23] Kato, H., Ushiku, Y., Harada, T.: Neural 3d mesh renderer. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3907–3916 (2018)
  * [24] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. arXiv preprint arXiv:2004.11362 (2020)
  * [25] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [26] Li, X., Liu, S., Kim, K., De Mello, S., Jampani, V., Yang, M.H., Kautz, J.: Self-supervised single-view 3d reconstruction via semantic consistency. In: European Conference on Computer Vision. pp. 677–693. Springer (2020)
  * [27] Lin, C.H., Kong, C., Lucey, S.: Learning efficient point cloud generation for dense 3d object reconstruction. In: proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
  * [28] Lin, C.H., Wang, C., Lucey, S.: Sdf-srn: Learning signed distance 3d object reconstruction from static images. arXiv preprint arXiv:2010.10505 (2020)
  * [29] Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7708–7717 (2019)
  * [30] Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L.: Sphereface: Deep hypersphere embedding for face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 212–220 (2017)
  * [31] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163–169 (1987)
  * [32] Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for gans do actually converge? In: International conference on machine learning. pp. 3481–3490. PMLR (2018)
  * [33] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4460–4470 (2019)
  * [34] Michalkiewicz, M., Parisot, S., Tsogkas, S., Baktashmotlagh, M., Eriksson, A., Belilovsky, E.: Few-shot single-view 3-d object reconstruction with compositional priors. In: European Conference on Computer Vision. pp. 614–630. Springer (2020)
  * [35] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: European Conference on Computer Vision. pp. 405–421. Springer (2020)
  * [36] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y.: Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018)
  * [37] Miyato, T., Koyama, M.: cgans with projection discriminator. arXiv preprint arXiv:1802.05637 (2018)
  * [38] Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 360–368 (2017)
  * [39] Mustikovela, S.K., Jampani, V., Mello, S.D., Liu, S., Iqbal, U., Rother, C., Kautz, J.: Self-supervised viewpoint learning from image collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3971–3981 (2020)
  * [40] Navaneet, K., Mathew, A., Kashyap, S., Hung, W.C., Jampani, V., Babu, R.V.: From image collections to point clouds with self-supervised shape and pose networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1132–1140 (2020)
  * [41] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3504–3515 (2020)
  * [42] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
  * [43] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32, 8026–8037 (2019)
  * [44] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2304–2314 (2019)
  * [45] Simoni, A., Pini, S., Vezzani, R., Cucchiara, R.: Multi-category mesh reconstruction from image collections. arXiv preprint arXiv:2110.11256 (2021)
  * [46] Sun, X., Wu, J., Zhang, X., Zhang, Z., Zhang, C., Xue, T., Tenenbaum, J.B., Freeman, W.T.: Pix3d: Dataset and methods for single-image 3d shape modeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [47] Tatarchenko, M., Richter, S.R., Ranftl, R., Li, Z., Koltun, V., Brox, T.: What do single-view 3d reconstruction networks learn? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3405–3414 (2019)
  * [48] Thai, A., Stojanov, S., Upadhya, V., Rehg, J.M.: 3d reconstruction of novel object shapes from single images. arXiv preprint arXiv:2006.07752 (2020)
  * [49] Tulsiani, S., Kulkarni, N., Gupta, A.: Implicit mesh reconstruction from unannotated image collections. arXiv preprint arXiv:2007.08504 (2020)
  * [50] Tulsiani, S., Zhou, T., Efros, A.A., Malik, J.: Multi-view supervision for single-view reconstruction via differentiable ray consistency. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2626–2634 (2017)
  * [51] Wallace, B., Hariharan, B.: Few-shot generalization for single-image 3d reconstruction via priors. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3818–3827 (2019)
  * [52] Wang, F., Xiang, X., Cheng, J., Yuille, A.L.: Normface: L2 hypersphere embedding for face verification. In: Proceedings of the 25th ACM international conference on Multimedia. pp. 1041–1049 (2017)
  * [53] Weinberger, K.Q., Saul, L.K.: Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research 10(2) (2009)
  * [54] Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1–10 (2020)
  * [55] Xiang, Y., Mottaghi, R., Savarese, S.: Beyond pascal: A benchmark for 3d object detection in the wild. In: IEEE winter conference on applications of computer vision. pp. 75–82. IEEE (2014)
  * [56] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711 (2019)
  * [57] Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H.: Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. arXiv preprint arXiv:1612.00814 (2016)
  * [58] Yang, G., Cui, Y., Belongie, S., Hariharan, B.: Learning single-view 3d reconstruction with limited pose supervision. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 86–101 (2018)
  * [59] Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Basri, R., Lipman, Y.: Multiview neural surface reconstruction by disentangling geometry and appearance. arXiv preprint arXiv:2003.09852 (2020)
  * [60] Ye, Y., Tulsiani, S., Gupta, A.: Shelf-supervised mesh prediction in the wild. arXiv preprint arXiv:2102.06195 (2021)
  * [61] Zhai, A., Wu, H.Y.: Classification is a strong baseline for deep metric learning. arXiv preprint arXiv:1811.12649 (2018)
  * [62] Zhang, X., Zhang, Z., Zhang, C., Tenenbaum, J.B., Freeman, W.T., Wu, J.: Learning to reconstruct shapes from unseen classes. arXiv preprint arXiv:1812.11166 (2018)
  * [63] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2223–2232 (2017)

##  Appendix 0.A Overview

This appendix is structured as follows: In Section 0.B we provide more
experimental results on Pix3D [46]; In Section 0.C we empirically quantify the
value of using category labels as opposed to multi-view supervision; In
Section 0.D we present additional SOTA comparison on ShapeNet-13; In Section
0.E we give implementation and training details for our model; In Section 0.F
we discuss the limitations of our approach and in Section 0.G we present
additional qualitative results on our large-scale ShapeNet-55 renderings.

##  Appendix 0.B Additional Experiments on Pix3D

We additionally evaluate our methods on Pix3D [46]. For Pix3D, we use 4
categories including bookcase, chair, table and wardrobe. We split the data
with a 70/10/20 percentage into training, validation and testing similar to
our experiments on ShapeNet. We compare our method to SDF-SRN [28] and Ye et
al. [60] on Pix3D, as shown in Table 5 and Fig. 7. Again, it is clear that our
method outperforms the SOTA methods quantitatively and qualitatively. These
results further verify the effectiveness of our proposed methods.

Figure 7: Qualitative comparison on Pix3D. Our method learns both better global 3D structure and shape details on various categories. Table 5: Quantitative result measured by CD and F-score on Pix3D. Our method performs favorably to other SOTA methods.  Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  CD$\downarrow$
---|---|---|---|---
SDF-SRN [28] | 0.1370 | 0.5622 | 0.7996 | 0.625
Ye et al. [60] | 0.1325 | 0.5308 | 0.7994 | 0.585
Ours | 0.1745 | 0.6604 | 0.8988 | 0.421

##  Appendix 0.C Quantifying the value of category labels

We further quantify the value of category label for the Multi-Category Single-
View (MCSV) reconstruction task in this section. Our goal is to compare our
method with category-guided shape metric learning with a model trained with
multi-view supervision (more than a single view available per object
instance). In this section, we train all models without adversarial
regularization or the viewpoint predictor. We assume the viewpoint is known in
this set of experiments. The baseline models are trained using an additional
view as supervision. We follow a similar training process as [41] by randomly
sampling a view per object for each epoch. To empirically determine the value
of category labels, we vary the portion of the data that has multi-view
annotations and compare that to our single-view category guided model. The
quantitative results are shown in Figure 8. As shown in the figure, having
access to category labels can roughly lead to the reconstruction accuracy
using 15% to 20% two-view annotation, measured by Chamfer Distance. Given the
availability of category label compared to multi-view data, we believe that
this is a promising finding.

Figure 8: We evaluate the value of category label in MCSV reconstruction with
camera pose. Under this quantitative evaluation, we show the category label
will lead to similar performance of having access to around 15% to 20% two-
view annotation.

##  Appendix 0.D Additional Comparison on ShapeNet-13

We additionally compare to Ye et al. [60] on ShapeNet-13. We train Ye et al.
in their category-specific way (13 different models) and the comparison is
shown in Table 6. Our method outperforms Ye et al. significantly, even though
the computation/parameters required for the category-specific models are much
higher than ours.

Table 6: Additional comparison measured by CD and F-score on ShapeNet-13. Methods |<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|<EMAIL_ADDRESS>|  CD$\downarrow$
---|---|---|---|---
Ye et al. [60] | 0.1349 | 0.5419 | 0.7777 | 0.669
Ours | 0.2005 | 0.7168 | 0.8949 | 0.430

##  Appendix 0.E Implementation details

In this section, we provide more details about our architecture, loss
functions, training and evaluation.

###  0.E.1 Network architecture

Our architecture of image encoder, shape/texture module as well as the
differentiable renderer share a similar design to SDF-SRN [28], while shape
and texture modules are made more lightweight for computational efficiency.
The image encoder and the viewpoint predictor are based on ResNet18 [14]. The
hypernetwork that generates the weights of shape and texture field from latent
code is a set of 6-layer MLPs of hidden dimension 512. Each MLP in this set
generates the weights of a single layer of either shape or texture. The MLP
representing $\boldsymbol{f_{S}}$ also has a 6-layer structure, with first 4
hidden layer 64 neurons and last hidden layer 32 neurons. The texture MLP has
2 layers, with a hidden dimension of 128. It is conditioned on the shape
embedding following [59]. Note that both shape and texture MLPs use Positional
Encoding [35] to encode input coordinates for better details. The
differentiable renderer we use is from SDF-SRN [28], where a LSTM [18] learns
to perform the ray marching steps. The LSTM predicts a step length for each
rendering step based on local implicit feature as input and previous steps
encoded as hidden state. We follow IDR [59] (equation 7) to render the extra
alpha channel, where we use the negative minimum SDF value on each ray with
Sigmoid to represent the alpha value. In practice, we use the minimum SDF
value from the ray-marcher steps instead of sampling numerous depths for each
ray, and the SDF value is scaled by 30 before Sigmoid to increase the
sharpness of the mask.

###  0.E.2 Loss function

Our overall loss function is a weighted summation of the reconstruction loss,
regularization losses and the losses that facilitate the learning of the
renderer and the shape field as in SDF-SRN [28]:

$\mathcal{L}_{total}=\mathcal{L}_{recon}+\lambda_{1}\mathcal{L}_{metric}+\lambda_{2}\mathcal{L}_{gan}+\lambda_{3}\mathcal{L}_{cam}+\lambda_{4}\mathcal{L}_{SDF-
SRN}$ (10)

In our experiments, we set $\lambda_{2}=0.2$, $\lambda_{3}=0.03$,
$\lambda_{4}=1$ for all datasets. We use a $\lambda_{1}$ of 0.1 for
ShapeNet-55 and Pix3D, 0.05 for ShapeNet-13 and 0.03 for Pascal3D+.
$\mathcal{L}_{render}$ itself is a weighted summation of several losses as
well, we refer to [28] for more details. Since we render an extra alpha
channel, we also use the commonly used soft IOU loss [29] with coefficient 0.1
to supervise the predicted soft masks with GT masks.

###  0.E.3 Training and inference

To train our model, we iterate between the reconstruction step and the
adversarial step. In the reconstruction step, the whole model except the
discriminator is updated to minimize $\mathcal{L}_{total}$. All the
regularizations we propose are activated during this step. In the adversarial
step, only the discriminator is updated by maximizing
$\lambda_{1}\mathcal{L}_{gan}$, with all other loss disabled. For viewpoint
sampling, we follow uniform distributions for azimuth, with a range of
$[0^{\circ},360^{\circ}]$ across all datasets. The elevation is also sampled
uniformly within $[20^{\circ},40^{\circ}]$ on ShapeNet-55. We sample elevation
and tilt following Gaussian distributions on Pascal3D+ and treat mean and
standard deviation as hyperparameters, similar to [60]. Please see Fig. 9 for
the comparison of our prior distribution and the groundtruth camera
distribution.

Figure 9: Comparison between our prior camera distribution (green) for
training and the GT (blue) distributions on Pascal3D+.

We use a learning rate of 0.0001 for both steps, and the optimizer is Adam
[25] with $\beta_{1}=0$, $\beta_{2}=0.9$ and a batch size of 12. We did not
use weight decay, learning rate scheduling or data augmentations. Our model is
trained on a single NVIDIA GTX TITAN V for 80 epochs, which takes 2-3 days
depending on the dataset size. The shape field is pretrained with the SDF
values of a sphere for a better initialization as in SDF-SRN [28]. During
inference, we only keep the image encoder and the shape prediction module. We
implement our method in PyTorch [43] and will release the code upon
publication.

###  0.E.4 Evaluation details

We use the Marching Cubes algorithm [31] to convert the implicit
representation to meshes prior to computing the metrics. Specifically, we
sample SDF values with a $128^{3}$ spatial grid and extract the 0-isosurface
for marching cubes. We further sample 100000 points from each mesh for
calculating the metrics. To align the predicted and GT shapes under the same
canonical space, we transform both shapes to view-centered frames.

Specifically, when training our models on Pascal3D+ and Pix3D, we assume weak-
perspective cameras, and perform center crop and scaling over the input
images. Since we do not know where the center of each GT shape is located
w.r.t. the cropped and scaled image under a weak perspective camera, we align
the meshes by registering shape predictions to the ground truth using the
Iterative Closest Point (ICP) algorithm. This is in line with prior works such
as SDF-SRN [28].

###  0.E.5 License

We develop our code based on the code of SDF-SRN
555https://github.com/chenhsuanlin/signed-distance-SRN under the MIT license.
We use ShapeNetV2 [2], of which the license is specified at
https://shapenet.org/terms. We use Pascal3D+ [55] under a MIT license and
Pix3D [46] under a Creative Commons Attribution 4.0 International License.

##  Appendix 0.F Limitations

We further discuss the limitations of our method in this section with more
qualitative examples. When our model fails to reconstruct accurate shapes for
some samples we observe it is primarily due to 3 reasons: 1) concavity, 2)
class imbalance and 3) complex topology.

Figure 10: Illustration of limitation on concavity modeling. Our model fails
to reconstruct concave regions for these samples.

Figure 11: Illustration of limitation with class imbalance. Our model fails to
reconstruct the shape of these rare categories accurately.

Concavity. As discussed in the experiments, our method cannot model highly
concave shapes such as bowls or hats. The masks for such shapes do not provide
any information that reveals concavity. On the other hand, our method does not
have access to explicit lighting or shading information. These factors make
the learning of concavity hard. We demonstrate this issue in Fig. 11 with
examples from ShapeNet-55, including bowls, bath tubs and trash bins. We think
it will be interesting to explore the explicit modeling of lighting/shading
for future works.

Class imbalance. We see a strong class imbalance on ShapeNet-55, where several
classes have 500 samples while some only have 40 samples. Such an imbalance
makes the learning challenging for some categories, as the gradient update
within a minibatch can be dominated by major categories. We illustrate this
issue in Fig. 11 by showing the reconstruction on 3 rare categories. We think
it will be interesting to systematically explore the imbalance issue for shape
reconstruction.

Figure 12: Illustration of complex topologies. For samples that have complex
topologies, our model can only reconstruct a rough global structure.

Complex topologies. Due to the lack of 3D or multi-view supervision, it is
still quite challenging to learn accurate shapes when the topologies are
complex. We illustrate this issue in Fig. 12 by showing three shelves from
ShapeNet-55. We believe it is still an open problem to learn accurate shapes
for such examples under the challenging _multi-category, single-view (MCSV)_
setting without viewpoint supervision.

We hope these limitations are beneficial observations to inform and guide
future research under similar challenging settings. On the other hand, despite
these limitations, our method can reconstruct accurate shapes for the majority
of images or categories. We believe this is a significant step toward fully
unsupervised shape learning.

##  Appendix 0.G Additional Qualitative Results.

In this section, we show more qualitative results of our model on ShapeNet-55
across various categories, as in Fig. 13, Fig. 14, Fig. 15 and Fig. 16.

Figure 13: Additional qualitative results on ShapeNet-55. Figure 14:
Additional qualitative results on ShapeNet-55. Figure 15: Additional
qualitative results on ShapeNet-55. Figure 16: Additional qualitative results
on ShapeNet-55.