# Improving Query Representations for Dense Retrieval with Pseudo Relevance
Feedback

HongChien Yu  Carnegie Mellon University<EMAIL_ADDRESS>,  Chenyan Xiong
Microsoft Research<EMAIL_ADDRESS>and  Jamie Callan  Carnegie
Mellon University<EMAIL_ADDRESS>
(2021)

###### Abstract.

Dense retrieval systems conduct first-stage retrieval using embedded
representations and simple similarity metrics to match a query to documents.
Its effectiveness depends on encoded embeddings to capture the semantics of
queries and documents, a challenging task due to the shortness and ambiguity
of search queries. This paper proposes ANCE-PRF, a new query encoder that uses
pseudo relevance feedback (PRF) to improve query representations for dense
retrieval. ANCE-PRF uses a BERT encoder that consumes the query and the top
retrieved documents from a dense retrieval model, ANCE, and it learns to
produce better query embeddings directly from relevance labels. It also keeps
the document index unchanged to reduce overhead. ANCE-PRF significantly
outperforms ANCE and other recent dense retrieval systems on several datasets.
Analysis shows that the PRF encoder effectively captures the relevant and
complementary information from PRF documents, while ignoring the noise with
its learned attention mechanism.

Dense retrieval; query representation; pseudo relevance feedback

††journalyear: 2021††copyright: rightsretained††conference: Proceedings of the
30th ACM International Conference on Information and Knowledge Management;
November 1–5, 2021; Virtual Event, QLD, Australia††booktitle: Proceedings of
the 30th ACM International Conference on Information and Knowledge Management
(CIKM ’21), November 1–5, 2021, Virtual Event, QLD, Australia††doi:
10.1145/3459637.3482124††isbn: 978-1-4503-8446-9/21/11††ccs: Information
systems Retrieval models and ranking

##  1\. Introduction

Dense retrieval systems first encode queries and documents into a dense
embedding space and then perform information retrieval by finding a query’s
nearest neighbors in the embedding space (Lee et al., 2019; Reimers and
Gurevych, 2020; Karpukhin et al., 2020; Xiong et al., 2021). With the
advancement of pre-trained language models (Devlin et al., 2019; Lu et al.,
2021), dedicated training strategies (Karpukhin et al., 2020; Xiong et al.,
2021), and efficient nearest neighbor search (Johnson et al., 2021; Guo et
al., 2020), dense retrieval systems have shown effectiveness in a wide range
of tasks, including web search (Nguyen et al., 2016), open domain question
answering (Kwiatkowski et al., 2019), and zero-shot IR (Thakur et al., 2021).

Retrieval with dense, fully-learned representations has the potential to
address some fundamental challenges in sparse retrieval. For example,
vocabulary mismatch can be solved if the embeddings accurately capture the
information need behind a query and maps it to relevant documents. However,
decades of IR research demonstrates that inferring a user’s search intent from
a concise and often ambiguous search query is challenging (Croft et al.,
2010). Even with powerful pre-trained language models, it is unrealistic to
expect an encoder to perfectly embed the underlying information need from a
few query terms.

A common technique to improve query understanding in sparse retrieval systems
is pseudo relevance feedback (PRF) (Croft et al., 2010; Xu and Croft, 1996;
Lavrenko and Croft, 2001), which uses the top retrieved documents from an
initial search as additional information to enrich the query representation.
Whether PRF information is used via query expansion (Xu and Croft, 1996;
Jaleel et al., 2004) or query term reweighting (Bendersky et al., 2011), its
efficacy has been consistently observed across various search scenarios,
rendering PRF a standard practice in many sparse retrieval systems.

This work leverages PRF information to improve query representations in dense
retrieval. Given the top retrieved documents from a dense retrieval model,
e.g., ANCE (Xiong et al., 2021), we build a PRF query encoder, ANCE-PRF, that
uses a BERT encoder (Devlin et al., 2019) to consume the query and the PRF
documents to refine the query representation. ANCE-PRF is trained end-to-end
using relevance labels and learns to optimize the query embeddings using the
rich information from PRF documents. It reuses the document index from ANCE to
avoid duplicating index storage.

In experiments on MS MARCO and TREC Deep Learning (DL) Track passage ranking
benchmarks, ANCE-PRF is consistently more accurate than ANCE and several
recent dense retrieval systems that use more sophisticated models and training
strategies (Zhan et al., 2020; Luan et al., 2021). We also observe large
improvements on DL-HARD (Mackie et al., 2021) queries, a curated set to
include complex search intents challenging for neural systems. To the best of
our knowledge, ANCE-PRF is among the best performing first-stage retrieval
systems on the highly competitive MARCO passage ranking leaderboard.

Our studies confirm that the advantages of ANCE-PRF reside in its ability to
leverage the useful information from the PRF documents while ignoring the
noise from irrelevant PRF documents. The PRF encoder allocates substantially
more attention to terms from the relevant PRF documents, compared to those
from the irrelevant documents. A case study shows that the encoder focuses
more on PRF terms that are complementary to the query terms in representing
search intents. These help ANCE-PRF learn better query embeddings that are
closer to the relevant documents and improve the majority of testing
queries.111Our code, checkpoints, and ranking results are open-sourced at
https://github.com/yuhongqian/ANCE-PRF.

##  2\. Related Work

In dense retrieval systems, queries and documents are encoded by a dual-
encoder, often BERT-based, into a shared embedding space (Karpukhin et al.,
2020; Xiong et al., 2021; Luan et al., 2021; Khattab and Zaharia, 2020).
Recent research in dense retrieval mainly focuses on improving the training
strategies, especially the negative sampling part, including random sampling
in batch (Xiong et al., 2021), sampling from BM25 top negatives (Lee et al.,
2019; Gao et al., 2021), sampling from an asynchronously (Xiong et al., 2021)
updated hard negatives index, constructing hard negatives using document index
from an existing dense retrieval model (Zhan et al., 2020), or reranking
models (Hofstätter et al., 2021; Lin et al., 2020). Most dense retrieval
systems encode a document using a constant number of embedding vectors
(Karpukhin et al., 2020; Xiong et al., 2021; Luan et al., 2021), often one per
document. There are also approaches using one vector per document token
(Khattab and Zaharia, 2020), similar to the interaction-based neural IR
approaches (Guo et al., 2016; Xiong et al., 2017). In this work, we focus on
models that only use one vector per document, whose retrieval efficiency is
necessary for real production systems (Xiong et al., 2021).

In recent research, PRF information has been leveraged by neural networks to
combine feedback relevance scores (Li et al., 2018), modify query-document
interaction using encoded feedback documents (Ai et al., 2018; Chen et al.,
2021), or learn contextualized query-document interactions (Yu et al., 2021).
A parallel work (Wang et al., 2021) expands multi-vector query representations
with feedback embeddings extracted using a clustering technique.

##  3\. Method

A typical dense retrieval system encodes query $q$ and document $d$ using a
BERT-style encoder and then calculates the matching score $f(q,d)$ using
simple similarity metrics:

(1) $f(q,d)=\texttt{BERT}^{q}(\texttt{[CLS]}\circ
q\circ\texttt{[SEP]})\cdot\texttt{BERT}^{d}(\texttt{[CLS]}\circ
d\circ\texttt{[SEP]}),$

where BERTq and BERTd respectively output their final layer [CLS] embeddings
as the query and the document embeddings. Eq. (1) is fine-tuned using standard
ranking losses and with various negative sampling techniques (Karpukhin et
al., 2020; Luan et al., 2021). The initial retrieval system this work uses,
ANCE, conducts negative sampling from an asynchronously updated document index
(Xiong et al., 2021).

ANCE-PRF leverages PRF documents retrieved by ANCE to enrich query
representations. Given the top $k$ documents $d_{1},...,d_{k}$ from ANCE,
ANCE-PRF trains a new PRF query encoder to output the query embedding
$\bm{q^{\text{prf}}}$:

(2) $\bm{q^{\text{prf}}}=\texttt{BERT}^{prf}(\texttt{[CLS]}\circ
q\circ\texttt{[SEP]}\circ d_{1}\circ\texttt{[SEP]}\circ...\circ
d_{k}\circ\texttt{[SEP]}).$

It then conducts another retrieval with PRF embeddings:

(3)
$f^{\text{prf}}(q,d)=\bm{q^{\text{prf}}}\cdot\texttt{BERT}^{d}(\texttt{[CLS]}\circ
d\circ\texttt{[SEP]}).$

The training uses the standard negative log-likelihood loss:

(4)
$\mathcal{L}=-\log\frac{\exp(\bm{q^{\text{prf}}}\cdot\bm{d^{+}})}{\exp(\bm{q^{\text{prf}}}\cdot\bm{d^{+}})+\sum_{\bm{d^{-}}\in
D^{-}}\exp(\bm{q^{\text{prf}}}\cdot\bm{d^{-}})},$

where $\bm{d^{+}}$ and $\bm{d^{-}}$ are embeddings of relevant and irrelevant
documents. ANCE-PRF uses document embeddings from the initial dense retrieval
model to avoid maintaining a separate document index for PRF. Therefore, only
$\bm{q}^{prf}$ is newly learned.

Eq. (4) trains the query encoder to identify the relevant PRF information
using its Transformer attention. Specifically, the attention from the [CLS]
embedding in the last layer of Eq. (2) to the $j$th token $t_{j}$ of the input
sequence $s$ is:

(5)
$cls\\_attention(t_{j})=\sum_{i}\frac{\exp(\bm{q}_{cls}^{i}\cdot\bm{k}_{j}^{i})}{\sum_{l=1}^{|s|}\exp(\bm{q}_{cls}^{i}\cdot\bm{k}_{l}^{i})},$

where $\bm{q}_{cls}^{i}$ and $\bm{k}_{j}^{i}$ are the “query” vector and $j$th
input token’s “key” vector of the $i$th attention head (Vaswani et al., 2017).
Ideally, the PRF encoder should learn to yield

(6)
$\sum_{j^{+}}cls\\_attention(t_{j^{+}})>\sum_{j^{-}}cls\\_attention(t_{j^{-}}),$

where $j^{+}$ are indexes of the meaningful tokens from the PRF documents, and
$j^{-}$ are those of the irrelevant PRF tokens.

ANCE-PRF can be easily integrated with any dense retrieval models. With the
document embeddings and index unchanged, the only computational overheads are
one more query encoder forward pass (Eq. (2)) and one more nearest neighbor
search (Eq. (3)), a minor addition to the dense retrieval process (Xiong et
al., 2021).

Table 1. Ranking results. ANCE-PRF uses 3 feedback documents. All baseline results except BM25 and BM25+RM3 are reported by previous work. Statistically significant improvements over baselines are indicated by $\ast$ (BM25), $\dagger$ (BM25+RM3), $\ddagger$ (DE-BERT), $\mathsection$ (ME-BERT), and $\mathparagraph$ (ANCE) with $p\leq 0.05$ in t-test. Per query results of those underlined are not available for significance tests.  | MARCO Dev | MARCO Eval | TREC DL 2019 | TREC DL 2020
---|---|---|---|---
Method | NDCG@10 | MRR@10 | R@1K | MRR@10 | NDCG@10 | R@1K | HOLE@10 | NDCG@10 | R@1K | HOLE@10
BM25 |  0.238† | -38.7% |  0.191† | -42.1% | 0.858 | -10.5% | - | - | 0.506 | -21.9% | 0.750 | -0.1% | 0.000 | 0.480 | -25.7% | 0.786 | +1.3% | 0.006
BM25+RM3 | 0.219 | -43.6% | 0.171 | -48.2% |  0.872∗§ | -9.1% | - | - | 0.518 | -20.1% | 0.800∗¶ | +6.0% | 0.000 | 0.482 | -25.4% | 0.822∗¶ | +5.9% | 0.002
DPR (Karpukhin et al., 2020; Xiong et al., 2021) | - | - | 0.311 | -5.8% | 0.952 | -0.1% | - | - | 0.600 | -7.4% | - | - | - | 0.557 | -13.8% | - | - | -
DE-BERT (Luan et al., 2021) |  0.358∗† | -7.7% | 0.302 | -8.5% | - | - | 0.302 | -4.7% | 0.639 | -1.4% | - | - | 0.165 | - | - | - | - | -
ME-BERT (Luan et al., 2021) |  0.394∗†‡ | +1.5% |  0.334∗†‡ | +1.2% | 0.855 | -10.8% | 0.323 | +1.9% | 0.687 | +6.0% | - | - | 0.109 | - | - | - | - | -
LTRe (Zhan et al., 2020) | - | - | 0.341 | +3.3% | 0.962 | +0.0% | - | - | 0.675 | +4.2% | - | - | - | - | - | - | - | -
ANCE (Xiong et al., 2021) |  0.388∗†‡ | 0.0% |  0.330∗†‡ | 0.0% |  0.959∗†§ | 0.0% | 0.317 | 0.0% | $0.648^{\ast\dagger}$ | 0.0% | 0.755 | 0.0% | 0.149 | $0.646^{\ast\dagger}$ | 0.0% | 0.776 | 0.0% | 0.135
ANCE-PRF | 0.401∗†‡§¶ | +3.4% | 0.344∗†‡§¶ | +4.2% |  0.959∗†§ | +0.0% | 0.330 | +4.1% | $0.681^{\ast\dagger}$ | +5.1% | $0.791^{\mathparagraph}$ | +4.8% | 0.133 | $\textbf{0.695}^{\ast\dagger\mathparagraph}$ | +7.6% | 0.815 | +5.0% | 0.087
Table 2. Ranking accuracy with a varying number of PRF documents ($k$). $Avg\\_Rel$ is the average relevance score of PRF documents at position $k$. Superscriptsk mark statistically significant improvements over $k$. ANCE results are in the first row ($\ast$). |  MARCO Dev (Binary Label) |  TREC DL 2019 (0-3 Scale Label) |  TREC DL 2020 (0-3 Scale Label)
---|---|---|---
$\bm{k}$ | NDCG@10 | MRR@10 | R@1K | Avg_Rel | NDCG@10 | R@1K | HOLE@10 | Avg_Rel | NDCG@10 | R@1K | HOLE@10 | Avg_Rel
$\ast$ |  0.3880 |  0.3300 |  0.9590 | - | 0.648 | 0.755 | 0.149 | - | 0.646 | 0.776 | 0.135 | -
0 | 0.364 | 0.307 | 0.943 | - | 0.672 | 0.780 | 0.149 | - | 0.668 | 0.791 | 0.115 | -
1 |  0.393∗0 |  0.3340 | 0.963∗0 | 0.210 |  0.680∗ |  0.795∗ | 0.142 | 2.023 |  0.689∗ | 0.814 | 0.093 | 2.093
2 |  0.401∗01 |  0.343∗01 |  0.96203 | 0.112 | 0.678 | 0.797∗ | 0.133 | 1.651 | 0.696∗ | 0.816 | 0.085 | 1.870
3 |  0.401∗01 |  0.344∗01 |  0.9590 | 0.067 | 0.681 |  0.791∗ | 0.133 | 1.791 |  0.695∗ | 0.815 | 0.087 | 1.907
4 | 0.403∗015 | 0.346∗012 |  0.9610 | 0.046 | 0.675 |  0.796∗ | 0.130 | 1.535 | 0.696∗ | 0.821 | 0.093 | 1.556
5 |  0.400∗01 |  0.344∗01 |  0.96001 | 0.039 | 0.681 |  0.796∗ | 0.128 | 1.465 |  0.688∗ | 0.816 | 0.096 | 1.370
Table 3. Results on DL-HARD (Mackie et al., 2021). We use the same symbols as in Table 1 for statistically significant improvements. | DL-HARD
---|---
Method | NDCG@10 | R@1K | HOLE@10
BM25 |  0.304† | -9.0% | 0.669 | -12.8% | 0.504
BM25+RM3 | 0.273 | -18.3% | 0.703 | -8.3% | 0.508
ANCE | 0.334 | 0.0% | 0.767 | 0.0% | 0.570
ANCE-PRF | 0.365† | +9.3% | 0.761 | -0.1% | 0.544

##  4\. Experimental Setup

Next, we discuss the datasets, baselines, and implementation details.

Datasets. We use MS MARCO passage training data (Nguyen et al., 2016) which
includes 530K training queries. We first evaluate on its dev set with 7k
queries and also obtain the testing results by submitting to its leaderboard.
MARCO’s official metric is MRR@10.

We also evaluate the MARCO trained model on two additional evaluation
benchmarks, TREC DL (Craswell et al., 2020b; Craswell et al., 2020a) and DL-
HARD (Mackie et al., 2021). TREC DL (Craswell et al., 2020b; Craswell et al.,
2020a) includes 43 labeled queries from 2019 and 54 from 2020 for the MARCO
corpus. The official metric is NDCG@10 and Recall@1K, the latter with label
binarized at relevance point 2. Following Xiong et al. (Xiong et al., 2021),
we also report HOLE@10, the unjudged fraction of top 10 retrieved documents,
to reflect the coverage of pooled labels on dense retrieval systems. DL-HARD
(Mackie et al., 2021) contains 50 queries from TREC DL that were curated to
challenge neural systems in a prior TREC DL track. Its official metric is
NDCG@10.

Baselines include BM25 (Robertson and Zaragoza, 2009), RM3 (Jaleel et al.,
2004; Lavrenko and Croft, 2001), a classical PRF framework in sparse
retrieval. We also compare with several recent dense retrievers. ME-BERT (Luan
et al., 2021) was trained with hard-negative mining (Gillick et al., 2019),
and is the only one that uses multi-vector document encoding. DE-BERT (Luan et
al., 2021) is the single-vector version of ME-BERT. DPR (Karpukhin et al.,
2020) is trained with in-batch negatives. LTRe (Zhan et al., 2020) generates
hard negatives using document embeddings from an existing dense retrieval
model. ANCE (Xiong et al., 2021) uses hard negatives from asynchronously
updated dense retrieval index using the latest model checkpoint.

Implementation Details. In training, we initialize query encoder from the ANCE
FirstP model (Xiong et al., 2021)222https://github.com/microsoft/ANCE and kept
the document embeddings from ANCE (and thus also the ANCE negative index)
uncharged. All hyperparameters used in ANCE training are inherited in ANCE-
PRF. All models are trained on two RTX 2080 Ti GPUs with per-GPU batch size 4
and gradient accumulation step 8 for 450K steps. We keep the model checkpoint
with the best MRR@10 score on the MS MARCO dev set.

In inference, we first obtain ANCE top $k$ documents using Faiss IndexFlatIP
and Eq. (1), feed them into the ANCE-PRF query encoder (Eq. (2)) for updated
query embeddings, and run another Faiss search with Eq. (3) for final results.

##  5\. Experimental Results

In this section, we discuss our experimental results and studies.

###  5.1. Overall Results

Table 1 includes overall retrieval accuracy on MS MARCO and TREC DL datasets.
ANCE-PRF outperforms ANCE, its base retrieval system, on all datasets. On the
challenging DL-HARD (Table 3), ANCE-PRF improves NDCG@10 By 9.3% over ANCE,
indicating ANCE-PRF’s advantage in queries challenging for neural systems.
These results suggest that ANCE-PRF effectively leverages PRF information to
produce better query embeddings. ANCE-PRF also helps retrieve relevant
documents not recognized by ANCE, improving R@1K by about 5% on both TREC DL
sets.

ANCE-PRF rankings are significantly more precise than the sparse retrieval
baselines with large margins across all datasets. RM3 achieves the best R@1K
on both TREC DL sets, but its improvement is not as significant on DL-HARD.

ANCE-PRF also outperforms several strong dense retrieval baselines and
produces the most accurate rankings on almost all datasets. While Luan et al.
(Luan et al., 2021) discuss the theoretical benefits of higher dimensional
dense retrieval as in ME-BERT, our empirical results show that a well-informed
query encoder can achieve comparable results, while avoiding the computational
and spatial overhead caused by using multiple vectors per document.

(a) Dot product

(b) Attention on 1st PRF Doc

(c) Attention on 2nd PRF Doc

(d) Attention on 3rd PRF Doc

Figure 1. Fig. (a) shows the dot product between the ANCE-PRF query embedding
and document embeddings at different training steps (x-axis). Fig. (b)-(d) are
the cls$\\_$attention (y-axes) on “all”, “relevant”, and “irrelevant” feedback
documents ranked at positions 1-3 in the initial retrieval.  Figure 2. The
histograms show the per-query NDCG@10 differences between ANCE-PRF and ANCE
retrieval on TREC DL 2020’s 54 queries. ANCE-PRF wins on 34 queries, loses on
15, and ties on 5. The tables show the example queries and the two models’
first different retrieved passages. Terms receiving higher attention weights
are highlighted in darker red.

###  5.2. Ablation on PRF Depths

To understand the number of feedback documents ($k$) needed for effective
learning, we trained models using different $k$ and report the results in
Table 2. We trained $k=0$ as a controlled experiment, which is equivalent to
training ANCE for an extra 450K steps with fixed negatives.

Overall, we observe that models with $k>0$ are consistently better than ANCE
($k=\ast$) and $k=0$, showing that ANCE-PRF effectively utilizes the given
feedback relevance information. The Avg_Rel indicates that PRF documents at
$k>1$ contain noisy relevance information, which is a known challenge for
traditional PRF approaches (Collins-Thompson, 2009). Nevertheless, ANCE-PRF
yields stable improvements over ANCE for $k=1$ to 5, demonstrating the model’s
robustness against noisy feedback from deeper $k$.

###  5.3. Analyses of Embedding Space & Attention

In this group of experiments, we analyze the learned embeddings and attention
in ANCE-PRF.

Embedding Space. Fig. 1(a) shows the distance during training between the
ANCE-PRF query embedding and the embeddings of the original ANCE query, the
relevant documents, and the irrelevant documents. We use MARCO dev in this
study, in which about one out of the three PRF documents is relevant. In the
embedding space, ANCE-PRF queries are closest to the original query and then
the relevant documents, while further away from the irrelevant documents.
ANCE-PRF’s query embeddings effectively encode both the query and the feedback
relevance information.

Learned Attention. We also analyze the learned attention on the relevant and
the irrelevant PRF documents during training. We use TREC DL 2020 for this
study as its dense relevance labels provide more stable observations. We
calculate the average attention from the [CLS] token to each group
(“relevant”, “irrelevant”, and “all”) of PRF document (Eq. (5) & (6)), and
plot them in Fig. 1(b)-1(d).

As training proceeds, ANCE-PRF pays more and more attention to the relevant
PRF documents than the irrelevant ones, showing the effectiveness of its
learning. Note that the original query always attracts the most attention from
the PRF encoder, which is intuitive, as the majority of the search intent is
to be determined by the query. The PRF information is to refine the query
representation with extra information but not to invalidate it.

###  5.4. Case Study

Fig. 2 plots the per query win/loss of ANCE-PRF versus ANCE on TREC DL 2020
and shows one example each.

ANCE-PRF wins on more queries and with larger margins. We also notice the PRF
query encoder focuses more on terms that are complementary to the query. In
the winning example, ANCE-PRF picks up terms explaining what ”un fao” is and
does not mistake ”un” as ”uno”. On the other hand, ANCE-PRF may be misled by
information appearing in multiple feedback documents. This is a known
challenge for PRF because the correctness of information from multiple
feedback documents is its core assumption (Lavrenko and Croft, 2001). In the
losing example, “pattern their shotguns” occurs in multiple PRF documents,
attracting too much attention to allow ANCE-PRF to make a better choice.

##  6\. Conclusion

Existing dense retrievers learn query representations from short and ambiguous
user queries, thus a query representation may not precisely reflect the
underlying information need. ANCE-PRF addresses this problem with a new query
encoder that learns better query representations from the original query and
the top-ranked documents from a state-of-the-art dense retriever, ANCE.

Our experiments demonstrate that ANCE-PRF’s effectiveness in refining query
understanding and its robustness against noise from imperfect feedback. Our
studies reveal that ANCE-PRF learns to distinguish between relevant and
irrelevant documents. We show that ANCE-PRF successfully learns to identify
relevance information with its attention mechanism. Its query encoder pays
more attention to the relevant portion of the PRF documents, especially the
PRF terms that complement the query terms in expressing the information need.

ANCE-PRF provides a straightforward way to leverage the PRF information in
dense retrieval and can be used as a plug-in in embedding-based retrieval
systems. We observe that simply leveraging the classic PRF information in the
new neural-based retrieval regime leads to significant accuracy improvements,
suggesting that more future research can be done in this direction.

## References

  * (1)
  * Ai et al. (2018) Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018\.  Learning a Deep Listwise Context Model for Ranking Refinement. In _The 41st International ACM SIGIR Conference on Research Development in Information Retrieval_. ACM, 135–144.
  * Bendersky et al. (2011) Michael Bendersky, Donald Metzler, and W Bruce Croft. 2011\.  Parameterized concept weighting in verbose queries. In _Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval_. ACM, 605–614.
  * Chen et al. (2021) Xiaoyang Chen, Kai Hui, Ben He, Xianpei Han, Le Sun, and Zheng Ye. 2021.  Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context.  _arXiv preprint arXiv:2104.08523_ (2021).
  * Collins-Thompson (2009) Kevyn Collins-Thompson. 2009\.  Reducing the risk of query expansion via robust constrained optimization. In _Proceedings of the 18th ACM Conference on Information and Knowledge Management_. ACM, 837–846.
  * Craswell et al. (2020a) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020a.  Overview of the TREC 2020 Deep Learning Track. In _Proceedings of the Twenty-Ninth Text REtrieval Conference_ _(NIST Special Publication)_. National Institute of Standards and Technology (NIST).
  * Craswell et al. (2020b) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020b.  Overview of the TREC 2019 Deep Learning Track. In _Proceedings of the Twenty-Eighth Text REtrieval Conference_ _(NIST Special Publication)_. National Institute of Standards and Technology (NIST).
  * Croft et al. (2010) W Bruce Croft, Donald Metzler, and Trevor Strohman. 2010\.  _Search engines: Information retrieval in practice_. Vol. 520.  Addison-Wesley Reading.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Vol. 1. Association for Computational Linguistics, 4171–4186.
  * Gao et al. (2021) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2021\.  Complement Lexical Retrieval Model with Semantic Residual Embeddings. In _Advances in Information Retrieval - 43rd European Conference on IR Research_ _(Lecture Notes in Computer Science, Vol. 12656)_. Springer, 146–160.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego García-Olano. 2019\.  Learning Dense Representations for Entity Retrieval. In _Proceedings of the 23rd Conference on Computational Natural Language Learning_. Association for Computational Linguistics, 528–537.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016\.  A Deep Relevance Matching Model for Ad-hoc Retrieval. In _Proceedings of the 25th ACM International Conference on Information and Knowledge Management_. ACM, 55–64.
  * Guo et al. (2020) Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020.  Accelerating Large-Scale Inference with Anisotropic Vector Quantization. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_. PMLR, 3887–3896.
  * Hofstätter et al. (2021) Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021\.  Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In _The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 113–122.
  * Jaleel et al. (2004) Nasreen Abdul Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah S. Larkey, Xiaoyan Li, Mark D. Smucker, and Courtney Wade. 2004.  UMass at TREC 2004: Novelty and HARD. In _Proceedings of the Thirteenth Text REtrieval Conference_ _(NIST Special Publication, Vol. 500-261)_. NIST.
  * Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021\.  Billion-Scale Similarity Search with GPUs.  _IEEE Trans. Big Data_ 7, 3 (2021), 535–547.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 6769–6781.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. ACM, 39–48.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 452–466.
  * Lavrenko and Croft (2001) Victor Lavrenko and W. Bruce Croft. 2001.  Relevance-Based Language Models. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 120–127.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019\.  Latent Retrieval for Weakly Supervised Open Domain Question Answering. In _Proceedings of the 57th Conference of the Association for Computational Linguistics_ , Vol. 1. Association for Computational Linguistics, 6086–6096.
  * Li et al. (2018) Canjia Li, Yingfei Sun, Ben He, Le Wang, Kai Hui, Andrew Yates, Le Sun, and Jungang Xu. 2018.  NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 4482–4491.
  * Lin et al. (2020) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020\.  Distilling Dense Representations for Ranking using Tightly-Coupled Teachers.  _arXiv preprint arXiv:2010.11386_ (2020).
  * Lu et al. (2021) Shuqi Lu, Chenyan Xiong, Di He, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tieyan Liu, and Arnold Overwijk. 2021\.  Less is More: Pre-training a Strong Siamese Encoder Using a Weak Decoder.  _arXiv preprint arXiv:2102.09206_ (2021).
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, Dense, and Attentional Representations for Text Retrieval.  _Trans. Assoc. Comput. Linguistics_ 9 (2021), 329–345.
  * Mackie et al. (2021) Iain Mackie, Jeffrey Dalton, and Andrew Yates. 2021\.  How Deep is your Learning: the DL-HARD Annotated Deep Learning Dataset. In _Proceedings of the 44th International ACM SIGIR conference on research and development in Information Retrieval_. ACM, 2335–2341.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems_ _(CEUR Workshop Proceedings, Vol. 1773)_. CEUR-WS.org.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 4512–4525.
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The Probabilistic Relevance Framework: BM25 and Beyond.  _Foundations and Trends in Information Retrieval_ 3, 4 (2009), 333–389.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models.  _arXiv preprint arXiv:2104.08663_ (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems_. 5998–6008.
  * Wang et al. (2021) Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021.  Pseudo-Relevance Feedback for Multiple Representation Dense Retrieval.  _arXiv preprint arXiv:2106.11251_.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 55–64.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021\.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In _9th International Conference on Learning Representations_. OpenReview.net.
  * Xu and Croft (1996) Jinxi Xu and W. Bruce Croft. 1996.  Query Expansion Using Local and Global Document Analysis. In _Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, 4–11.
  * Yu et al. (2021) HongChien Yu, Zhuyun Dai, and Jamie Callan. 2021.  PGT: Pseudo Relevance Feedback Using a Graph-Based Transformer. In _Advances in Information Retrieval - 43rd European Conference on IR Research_ _(Lecture Notes in Computer Science, Vol. 12657)_. Springer, 440–447.
  * Zhan et al. (2020) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020.  Learning To Retrieve: How to Train a Dense Retrieval Model Effectively and Efficiently.  _arXiv preprint arXiv:2010.10469_ (2020).


11institutetext: Technical University of Munich
Munich, Germany
11email: {abhishek.saroha, tarun.yenamandra<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
#  Implicit Shape Completion via Adversarial Shape Priors

Abhishek Saroha  Marvin Eisenberger  Tarun Yenamandra  Daniel Cremers

###### Abstract

We present a novel neural implicit shape method for partial point cloud
completion. To that end, we combine a conditional Deep-SDF architecture with
learned, adversarial shape priors. More specifically, our network converts
partial inputs into a global latent code and then recovers the full geometry
via an implicit, signed distance generator. Additionally, we train a
PointNet++ discriminator that impels the generator to produce plausible,
globally consistent reconstructions. In that way, we effectively decouple the
challenges of predicting shapes that are both realistic, i.e. imitate the
training set’s pose distribution, and accurate in the sense that they
replicate the partial input observations. In our experiments, we demonstrate
state-of-the-art performance for completing partial shapes, considering both
man-made objects (e.g. airplanes, chairs, …) and deformable shape categories
(human bodies). Finally, we show that our adversarial training approach leads
to visually plausible reconstructions that are highly consistent in recovering
missing parts of a given object.

###### Keywords:

Shape Reconstruction and Completion, Neural Implicit Models, Generative
Adversarial Networks, Signed Distance Function

##  1 Introduction

In recent times, neural implicit models experienced a steep increase in
popularity as a tool for 3D shape modeling. The high flexibility and
representation power of such approaches make them often preferable to other,
alternative 3D representations like voxel grids, point clouds, polygonal
meshes, or grid-based SDFs. For once, neural implicit methods are grid-free
and can, in theory, represent objects up to an arbitrary resolution. In
practice, this enables them to adapt the level of precision to different
regions of a given surface – the resulting encoding is, at the same time,
compact and precise. In comparison to explicit surface representations like
meshes, they can represent geometries of an arbitrary topology, only requiring
the target surface to be watertight. Moreover, implicit surfaces can be
converted to other familiar representations in a straightforward manner, e.g.
voxel grids by thresholding, or triangle meshes via the classical marching
cubes algorithm [20].

A common approach in this line of work is to devise encoder-decoder
architectures [21, 24, 25, 6] to convert various types of input data into
high-fidelity 3D surfaces. The decoder, in this context, predicts the
occupancy probability or signed distance values for a set of xyz-point
coordinates. This methodology has been used successfully to convert point
clouds into watertight surfaces, reconstruct the 3D geometry from image
observations or perform voxel super-resolution [40, 6, 28]. On the other hand,
while such encoder-decoder approaches excel at reproducing objects from the
training distribution, they often lack robustness when generalizing to unseen
test poses.

One of the key insights of our work is that we can extend the generalization
capacity of neural implicit generative models by utilizing adversarial shape
priors. Generative Adversarial Networks (GANs) [10] are a ubiquitous tool for
generative modeling on the 2D image domain. By and large, the popularity of
GANs is based on their ability to generate novel data instances that imitate
the style of the training distribution. While standard encoder-decoder
architectures merely replicate the exact training samples, GANs are able to
generate truly novel datapoints. Motivated by this observation, in this work
we leverage the power of adversarial shape priors, in combination with a
neural implicit architecture [24, 17].

A central open problem in the context of 3D generative models is shape
completion from partial input observations. Common use-cases include shape
reconstruction from range sensors like LIDAR, or completing partially occluded
inputs. Many existing methods define shape completion as filling in local
information, completing the backside of a front-facing human [6], or
completing halves of symmetric objects (e.g. left side of airplanes) [24]. In
contrast, we consider shape completion of point clouds that show truly
arbitrary views of an object with varying degrees of partiality. By
definition, this necessitates a holistic view on a given input observation:
Completing a given snippet requires (1) encoding and categorizing the partial
input point cloud (2) inferring the appropriate global geometry and (3)
reconstructing the full shape, while taking both (1) and (2) into account. In
an encoder-decoder architecture, (1)-(3) are strongly correlated. In our
experiments, we demonstrate that, by including adversarial shape priors, our
method is able to disentangle (1) and (2) to a certain extent. Overall, this
leads to significantly more robust predictions on partial snippets of unseen
test poses. There are a number of recent works that aim at increasing the
fidelity of neural implicit surface decoders by leveraging more local
information [25, 6]. Our results indicate that this is not always preferable
for shape completion, which requires modeling semantic information about a
given object on the level of the global shape pose.

####  1.0.1 Contribution

Our contributions can be summarized as follows:

  1. 1.

We devise a novel neural network architecture based on implicit surfaces that
predicts complete, watertight meshes from partial input point cloud
observations.

  2. 2.

We learn enhanced shape priors via generative adversarial learning. In that
manner, our model effectively decouples the tasks of reconstructing meaningful
shapes and fitting them to partial inputs.

  3. 3.

Our method achieves state-of-the-art performance for completing partial point
cloud observations of both man-made objects (lamps, chairs, …) and non-rigidly
deformable shapes (human bodies).

##  2 Related work

The field of 3D generative models experienced a surge in popularity over the
last few years. In the following, we provide an overview of approaches most
closely related to ours. We focus on related neural implicit models and
methods that are specifically designed for 3D shape completion.

####  2.0.1 Neural implicit models

Implicit surface models have a long-standing history in computer vision, see
[23] for an overview of classical level-set approaches. Fuelled by the rise of
deep learning, the pioneering works on occupancy networks [21], DISN [40] and
Deep-SDF [24] simultaneously proposed to represent implicit functions as a
feed-forward neural network. This paradigm has a number of crucial advantages
compared to other 3D representations. For once, deep implicit models are
significantly more memory efficient than voxel grids [7, 13] or classical
signed distance functions [23]. In contrast to template-based [19, 18, 37, 15]
and general mesh-based [12] methods, they allow for reconstructing watertight
surfaces without making any assumptions about the input object’s topology.
While point cloud generation models have similar advantages [9, 41, 1], they
only produce sparse samples of a 3D surface without any notion of
connectivity.

The initial works [21, 40, 24] sparked countless follow-up works that build up
on the idea of generating 3D shapes with neural implicits. There are a number
of works combine neural implicit models with local, grid-based convolutions
for different applications like creating human avatars [28], representing 3D
scenes [25] or standard object-level reconstruction [40]. Similar to our
approach, [17] learn an adversarial prior for 3D shape synthesis. [22]
reconstructs 4D shape sequences via learnable flow fields. [42] devise a
morphable model for human faces based on neural implicits. Another related
line of work [38, 5, 35, 29] aims at reconstructing highly realistic human
avatars by averaging scans in multiple poses via deformable human models like
SMPL [19]. Note that, compared with such methods, our approach works out of
the box for a broad range of object categories, since it makes no additional
modeling assumptions that are exclusive to humans.

####  2.0.2 Shape completion

The central task we consider in this work is shape completion, i.e.
reconstructing a full object from a partial observation. A standard approach
to that end is learning shape priors via an encoder-decoder architecture and
then predicting optimal reconstructions by fixing the learned decoder. This
approach has been explored individually for both voxelized, man-made objects
[31] and deformable meshes with fixed topology [18]. Similarly, [24] solve
shape completion via a neural implicit encoder-decoder architecture. Our
approach builds up on this methodology while additionally learning adversarial
shape priors via a PointNet++ discriminator [27]. Another common approach is
learning to generate full point clouds from partial ones via learnable
decoders [44, 43] based on FoldingNet [41] or hierarchical shape decoding
[32]. Others propose to complete voxel grids via 3D convolutional networks
[30, 8]. Combining this idea with neural implicits, [6] learns hierarchical,
voxel-based features which are subsequently decoded into an occupancy-based
representation.

##  3 Method

The overall objective of our method is to perform shape completion of partial
point cloud observations. Specifically, the goal is to map a partial input
point cloud $\bm{V}_{\mathrm{part}}\in\mathbb{R}^{n\times 3}$ to a full,
reconstructed triangular mesh
$\bm{X}_{\mathrm{rec}}:=\bigl{(}\bm{V}_{\mathrm{rec}},\bm{F}_{\mathrm{rec}}\bigr{)}$,
where $\bm{V}_{\mathrm{rec}}\in\mathbb{R}^{m\times 3}$ and
$\bm{F}_{\mathrm{rec}}\subset\bm{V}_{\mathrm{rec}}\times\bm{V}_{\mathrm{rec}}\times\bm{V}_{\mathrm{rec}}$
are the sets of vertices and faces of $\bm{X}_{\mathrm{rec}}$, respectively.
We analogously write
$\bm{X}_{\mathrm{full}}:=\bigl{(}\bm{V}_{\mathrm{full}},\bm{F}_{\mathrm{full}}\bigr{)}$
for the full, watertight input training shapes. Furthermore, we denote
$\bm{U}=(\bm{u}_{i})_{i=1,\dots,k}\in\mathbb{R}^{k\times 3}$ as a set of
randomly sampled, uniform points from the unit sphere
$\bm{u}_{i}\in\bigl{\\{}\bm{x}:\|\bm{x}\|_{2}\leq 1\bigr{\\}}$. During
training, the points $\bm{u}_{i}$ are used as query points to train the Deep-
SDF signed distance prediction network. In the following, we define our neural
network architecture in Section 3.1 and loss function in Section 3.2. We
further provide additional implementation details in Section 3.3.

Figure 1: Overview. Our model takes a partial point cloud
$\bm{V}_{\mathrm{part}}$ and generates signed distance predictions
$\bm{d}_{\mathrm{rec}}$ for a uniform set of points $\bm{U}$, conditioned on
the encoding $\bm{z}:=\mathcal{E}(\bm{V}_{\mathrm{part}})$. Similarly, it
predicts distances for the input points $\bm{V}_{\mathrm{part}}$ in order to
generate a shape that matches the partial observations. At test time, the
resulting distance field $\bm{d}_{\mathrm{rec}}$ is converted to the full,
reconstructed shape $\bm{X}_{\mathrm{full}}$. The discriminator incentivizes
meaningful output shapes by learning to distinguish between generated distance
fields $\bm{d}_{\mathrm{rec}}$ and corresponding real samples
$\bm{d}_{\mathrm{full}}$.

###  3.1 Network

The core of our method is an adversarial [10], implicit surface neural network
architecture [21, 24], see Figure 1 for an overview. In the following, we
define its individual components:

####  3.1.1 Encoder

The encoder $\mathcal{E}$ is responsible for converting the partial input
point cloud $\bm{V}_{\mathrm{part}}$ into a high-dimensional latent code
$\bm{z}$.

We can write this mapping $\mathcal{E}$ as

$\mathcal{E}:\begin{cases}\mathbb{R}^{n\times 3}\to\mathbb{R}^{512}\\\
\bm{V}_{\mathrm{part}}\mapsto\bm{z},\end{cases}$ (1)

where $\bm{V}_{\mathrm{part}}\in\mathbb{R}^{n\times 3}$ is the partial input
point cloud and $\bm{z}\in\mathbb{R}^{{512}}$ is the $512$-dim predicted
latent encoding. More specifically, we choose $\mathcal{E}$ as a point cloud
architecture based on the classification network of PointNet [26]. In
comparison to more sophisticated point cloud processing networks [27, 39, 33],
PointNet has no local feature detection mechanism and mainly focuses on the
global geometric structure of a considered object. In fact, the main objective
of the sub network $\mathcal{E}$ is to encode the global pose context of
$\bm{V}_{\mathrm{part}}$ whereas, detecting local features is of secondary
importance. Additionally, this makes our method robust to varying input
sampling densities, which we demonstrate in Figure 5. Apart from being a
straightforward choice, we therefore found empirically that the bias of
PointNet towards global geometric features serves our purpose best in this
context.

####  3.1.2 Generator

The generator $\mathcal{G}$ (or decoder) is responsible for generating 3D
surfaces that match the underlying training distribution. The network
architecture of $\mathcal{G}$ is based on existing work on Deep-SDF decoders
[17, 24], see our supplementary material for more details. For a given set of
query points $\bm{x}_{1},\dots,\bm{x}_{k}$, which are provided to the network
as the input, the generator $\mathcal{G}$ predicts signed distance values to
the target surface.

In summary, we can define $\mathcal{G}$ as

$\mathcal{G}:\begin{cases}\mathbb{R}^{3}\times\mathbb{R}^{512}\to\mathbb{R}\\\
(\bm{x}_{i},\bm{z})\mapsto
d_{i}:=\mathrm{SDF}_{\mathcal{G},\bm{z}}(\bm{x}_{i}),\end{cases}$ (2)

where $\bm{x}_{1},\dots,\bm{x}_{k}\in\mathbb{R}^{{k\times 3}}$ are the query
points with the corresponding, predicted signed distance values
$d_{1},\dots,d_{k}\in\mathbb{R}$ and $\bm{z}\in\mathbb{R}^{{512}}$ is a
$512$-dim latent vector predicted by the encoder $\mathcal{E}$.

During training, we consider two different types of query points $\bm{x}_{i}$
as inputs to $\mathcal{G}$: uniformly sampled points $\bm{x}_{i}:=\bm{u}_{i}$
and partial input surface points $\bm{x}_{i}:=\bm{v}_{\mathrm{part},i}$. The
former represents the predicted output shape $\bm{X}_{\mathrm{rec}}$ and the
latter is used to train the reconstruction loss, see Section 3.2 for more
details. By construction of Equation 2, the generator $\mathcal{G}$ processes
each input point $\bm{x}_{i}$ independently. For convenience, we therefore
simply concatenate both of these signals
$\bm{U}=(\bm{u}_{i})_{i=1,\dots,k}\in\mathbb{R}^{k\times 3}$ and
$\bm{V}_{\mathrm{part}}=(\bm{v}_{\mathrm{part},i})_{i=1,\dots,n}\in\mathbb{R}^{n\times
3}$ in practice and obtain a single input point cloud
$(\bm{U},\bm{V}_{\mathrm{part}})\in\mathbb{R}^{(k+n)\times 3}$ with which we
query $\mathcal{G}$ during training. This corresponds to the orange/blue
matrices in Figure 1.

####  3.1.3 Discriminator

The final component of our method is the discriminator network
$\mathcal{D}_{\bm{U}}$, playing the role of the critic that classifies
generated point clouds into real/fake:

$\mathcal{D}_{\bm{U}}:~{}\mathbb{R}^{k}\to\mathbb{R},~{}~{}\bm{d}\mapsto c.$
(3)

The mapping $\mathcal{D}_{\bm{U}}$ takes a signed distance field
$\bm{d}=(d_{1},\dots,d_{k})\in\mathbb{R}^{k}$ as input. During training, we
consider either the distances
$\bm{d}_{\mathrm{rec}}:=\mathcal{G}(\bm{U},\bm{z})$ predicted by the generator
or the ground-truth field $\bm{d}_{\mathrm{full}}$ corresponding to the full
training shapes $\bm{X}_{\mathrm{full}}$, see Section 3.2 for more details.
The subscript $\bm{U}$ of the discriminator $\mathcal{D}_{\bm{U}}$ highlights
its implicit dependency on the uniform query point cloud
$\bm{U}\in\mathbb{R}^{k\times 3}$. In practice, we concatenate $\bm{U}$ and
$\bm{d}$ to a 4D input point cloud $(\bm{U},\bm{d})\in\mathbb{R}^{k\times 4}$
within the first layer of $\mathcal{D}_{\bm{U}}$. The output score
$c\in\mathbb{R}$ is used subsequently used to compute the GAN loss, see
Section 3.2 for more details.

Following the common methodology of GANs on images, the $\mathcal{D}_{\bm{U}}$
acts as a critic that encourages $\mathcal{G}$ to imitate the training shape
distribution and synthesize more realistic samples. To that end, we chose the
PointNet++ architecture [27] as the basis for the discriminator
$\mathcal{D}_{\bm{U}}$. Compared to the standard PointNet network [26],
PointNet++ decomposes a given input point cloud into multiple subregions in a
hierarchical manner. Hence, it is much more effective at detecting local
features, which is a major prerequisite for obtaining realistic
reconstructions. Note, that this is in stark contrast to the encoder network
$\mathcal{E}$, whose main objective is detecting the global pose features of a
considered object. To further illustrate this point, we show a comparison of
the two different discriminators, in an ablation study, see our supplementary
material.

###  3.2 Loss function

To train our model, we define the following loss function:

$\ell\bigl{(}\mathcal{E},\mathcal{G},\mathcal{D}_{\bm{U}}\bigr{)}:=\ell_{\mathrm{GAN}}\bigl{(}\mathcal{G},\mathcal{D}_{\bm{U}}\bigr{)}+\lambda_{\mathrm{rec}}\ell_{\mathrm{rec}}\bigl{(}\mathcal{E},\mathcal{G}\bigr{)}+\lambda_{\mathrm{norm}}\ell_{\mathrm{norm}}\bigl{(}\mathcal{E},\mathcal{G}\bigr{)}.$
(4)

Motivated by the standard GAN methodology, our generator $\mathcal{G}$ and
discriminator $\mathcal{D}_{\bm{U}}$ are trained in an alternating manner,
playing the role of each other’s adversary. The discriminator tries to
distinguish samples generated by the generator from real training samples,
which in turn drives $\mathcal{G}$ to synthesize more realistic shapes.

$\min_{\mathcal{E},\mathcal{G}}\max_{\mathcal{D}_{\bm{U}}}\ell\bigl{(}\mathcal{E},\mathcal{G},\mathcal{D}_{\bm{U}}\bigr{)}.$
(5)

In practice, we optimize Equation 5 in an alternating manner, taking single
update steps of $\mathcal{E},\mathcal{G}$ and $\mathcal{D}_{\bm{U}}$ for a
single batch, respectively. In the remainder of this section, we define the
individual components referred to in Equation 4.

####  3.2.1 GAN loss

We devise the GAN loss component of our method by following the popular image
model Wasserstein-GAN (WGAN) [2]

$\ell_{\mathrm{GAN}}={\mathbb{E}_{\bm{d}_{\mathrm{full}}}}\bigl{[}\mathcal{D}_{\bm{U}}(\bm{d}_{\mathrm{full}})\bigr{]}-{\mathbb{E}_{\bm{V}_{\mathrm{part}}}}\bigl{[}\mathcal{D}_{\bm{U}}\bigl{(}\mathcal{G}(\bm{U},\mathcal{E}(\bm{V}_{\mathrm{part}}))\bigr{)}\bigr{]}+\lambda_{\mathrm{GP}}GP$
(6)

where we sample from both the training distributions of real samples
$\bm{d}_{\mathrm{full}}$ and partial conditional inputs
$\bm{V}_{\mathrm{part}}$. Here, $GP$ denotes a gradient penalty term, defined
as

$\mathbb{E}_{\lambda\sim\mathcal{U}(0,1)}\biggl{[}\bigl{(}\bigl{\|}\nabla_{\bm{d}}\mathcal{D}_{\bm{U}}\bigl{(}(1-\lambda)\bm{d}_{\mathrm{full}}+\lambda\bm{d}_{\mathrm{rec}}\bigr{)}\bigr{\|}_{2}-1\bigr{)}^{2}\biggr{]},$
(7)

where $\mathcal{U}(0,1)$ denote the uniform distribution on the interval
$(0,1)$. Furthermore, $\bm{d}_{\mathrm{full}}$ and
$\bm{d}_{\mathrm{rec}}=\mathcal{G}(\bm{U},\mathcal{E}(\bm{V}_{\mathrm{part}}))$
are defined as the signed distances of the full input training shapes and the
distance field predicted by the generator, respectively. Main idea of this
gradient penalty term is to enforce the approximate Lipschitz property
$\nabla_{\bm{d}}\mathcal{D}_{\bm{U}}(\cdots)\approx 1$ on the predictions of
the $\mathcal{D}_{\bm{U}}$ as a soft version of gradient clipping, see [2] for
further details.

####  3.2.2 Reconstruction loss

In addition to the adversarial loss, we employ a reconstruction loss on the
outputs of the generator.

$\ell_{\mathrm{rec}}={\mathbb{E}_{\bm{V}_{\mathrm{part}}}}\bigl{[}\bigl{\|}\mathcal{G}\bigl{(}\bm{V}_{\mathrm{part}},\mathcal{E}(\bm{V}_{\mathrm{part}})\bigr{)}\bigr{\|}_{2}\bigr{]},$
(8)

where $\bm{V}_{\mathrm{part}}$ are the surface points of the partial input
point cloud. The purpose of this loss is to enforce the predicted signed
distances of points on the input surface $\bm{V}_{\mathrm{part}}$ to be as
close as possible to zero.

Note, that $\ell_{\mathrm{GAN}}$ prevents the generator to opt for the
degenerate solution $\mathcal{G}(\cdot,\cdot)\equiv 0$ and instead take a
sample from the learned prior distribution, for which the points
$\bm{V}_{\mathrm{part}}$ are as close as possible to the surface of the
predicted shape.

####  3.2.3 Normal loss

We further utilize the outer normals of the partial input point clouds
$\bm{V}_{\mathrm{part}}$

$\ell_{\mathrm{norm}}={\mathbb{E}_{\bm{V}_{\mathrm{part}}}}\bigl{[}\bigl{\|}\nabla_{\bm{V}}\mathcal{G}\bigl{(}\bm{V}_{\mathrm{part}},\mathcal{E}(\bm{V}_{\mathrm{part}})\bigr{)}-\bm{N}_{\mathrm{part}}\bigr{\|}_{2}\bigr{]},$
(9)

where $\bm{N}_{\mathrm{part}}\in\mathbb{R}^{(n)\times 3}$ denotes the surface
normals corresponding to the point set $\bm{V}_{\mathrm{part}}$. As
demonstrated in [11], the loss in Equation 9 encourages the generator
$\mathcal{G}$ to replicate small scale features present in the training shape
distribution. In practice, we extract $\bm{N}_{\mathrm{part}}$ from the full
training mesh before sampling the partial point cloud
$\bm{V}_{\mathrm{part}}$.

###  3.3 Implementation details

####  3.3.1 Training protocol

To train our model for the task of shape completion, we consider a training
set, defined as a collection of full, watertight triangular meshes
$\bigl{\\{}\bm{X}_{\mathrm{full}}^{(1)},\dots,\bm{X}_{\mathrm{full}}^{(N)}\bigr{\\}}$,
where
$\bm{X}_{\mathrm{full}}^{(i)}=\bigr{(}\bm{V}_{\mathrm{full}}^{(i)},\bm{F}_{\mathrm{full}}^{(i)}\bigr{)}$,
see the purple shape in Figure 1. In each training iteration, we sample a
random half-space that removes parts of the full input shape to generate the
partial inputs $\bm{V}_{\mathrm{part}}$, see the blue point cloud in Figure 1.
The relative ratio of retained points after cut-off is chosen randomly from
the uniform distribution $\mathcal{U}(0.5,1)$. In that way, the model learns
to complete shapes with varying degrees of partialities. At test time, we
consider cut-off ratios sampled from $\mathcal{U}(0.5,0.55)$ to focus on the
challenging case of $\approx 50\%$ retained points. We show additional
partiality settings at test time in our ablation study, see Table 2.

Our model is trained in a coarse-to-fine manner: We start training the network
with $n=m=1024$ sampled points for both $\bm{U}$ and $\bm{V}_{\mathrm{part}}$.
After every 300 epochs, we double the number of points in
$\bm{U},\bm{V}_{\mathrm{part}}$ to $n=m=2048,4096,\dots,32768$ points. For
refinement, we append another 1000 epochs on the highest resolution
$n=m=32768$. This progressive training schedule is motivated by previous works
[16, 17] which show that it improves the training stability and overall
synthesis quality.

####  3.3.2 Data preparation

As specified in Section 4, we consider datasets that focus on both man-made
objects and deformable human bodies, respectively. For data preparation, we
follow the protocol described by [17, 24]. The input to our method is a
training set of triangular meshes
$\bigl{\\{}\bm{X}_{\mathrm{full}}^{(1)},\dots,\bm{X}_{\mathrm{full}}^{(N)}\bigr{\\}}$.
We first scale the vertices of the meshes $\bm{X}_{\mathrm{full}}^{(i)}$ to a
unit sphere for data normalization. For each training shape, we then uniformly
sample a set of $100,000$ points $\bm{u}_{j}$ in the ambient domain and
compute the ground-truth signed distance values $\bm{d}_{\mathrm{full},j}$ of
each point $\bm{u}_{j}$ from the surface boundary of
$\bm{X}_{\mathrm{full}}^{(i)}$. To convert the mesh
$\bm{X}_{\mathrm{full}}^{(i)}$ into a point cloud, each mesh is rendered from
multiple camera angles to obtain depth buffers, which are projected back into
the object space, see [17] for more details. We then sample the partial inputs
$\bm{V}_{\mathrm{part}}$, as well as their corresponding normals
$\bm{N}_{\mathrm{part}}$, as described in the previous paragraph. Shapes which
had either less than 1% of the samples points inside the shape, or a
discontinuous SDF, are discarded from the respective training and test sets.

In summary, for every processed mesh we get two input point clouds. the first
is the point cloud $\bm{U}$, signified by the orange box in fig. 1, which
contains randomly sampled points around the input object.
$\bm{V}_{\mathrm{part}}$ and $\bm{N}_{\mathrm{part}}$, on the other hand, are
the surface points and corresponding outer normals sampled from the meshes
$\bm{X}_{\mathrm{full}}$.

####  3.3.3 Parameter specifications

During training, we use RMSProp [34] as the optimizer for updating the
parameters of the various modules of our model. For the discriminator
optimization, the optimizer has a learning rate of 1e-5, while for the encoder
and the generator, the learning rate was set to 1e-3. We also set the relative
weight of the reconstruction loss to $\ell_{\mathrm{rec}}=8e-3$, while the
weight of the normal loss component is set to be $\ell_{\mathrm{norm}}=0.01$.
The Gradient Penalty defined (7) is weighed by a factor of
$\lambda_{\mathrm{GP}}=10$.

##  4 Experiments

###  4.1 Datasets

We choose two dataset to asssess our method’s ability to complete shape from
both rigid, man made object classes and non-rigidly deformable shape
categories. Specifically, we consider the following datasets:

####  4.1.1 ShapeNet

The ShapeNet dataset [4] is a large collection of man-made 3D models. Its
objects are classified into various different categories, such as airplanes,
sofas, chairs, cars, etc. Beyond the objects’ geometries, the ShapeNet dataset
also provides annotations such as correspondences, keypoints, various
orientation vectors, and more. More specifically, we use ShapeNetCore (v1),
which is a subset of the ShapeNet dataset, containing approximately 51,300
models that span 55 different categories. For our method, we specifically
focus on three different classes from the full dataset, which were lamps,
benches, and displays.

####  4.1.2 SURREAL Dataset

SURREAL [36] is a large-scale dataset which contains about 6 million 3D models
of synthetic humans. In addition to the pose information, they also provide
details such as optical flow, depth, normals, segmentation of body parts etc.
For our purposes, we use a subset of $1886$ shapes for training and $536$
different poses as our test set.

###  4.2 Evaluation

####  4.2.1 Metric

We evaluate different methods by measuring the Chamfer Distance (CD) between
the ground truth models $\bm{X}_{\mathrm{full}}$ and the predicted shapes
$\bm{X}_{\mathrm{rec}}$. The CD measures the distance between two point
clouds, defined as the mean distance of each individual point on the first
point cloud to the second point set:

$\mathrm{CD}\left(\bm{V}_{1},\bm{V}_{2}\right)=\frac{1}{\left|\bm{V}_{1}\right|}\sum_{\bm{x}\in\bm{V}_{1}}\min_{\bm{y}\in\bm{V}_{2}}\|\bm{x}-\bm{y}\|_{2}^{2}+\frac{1}{\left|\bm{V}_{2}\right|}\sum_{\bm{y}\in\bm{V}_{2}}\min_{\bm{x}\in\bm{V}_{1}}\|\bm{x}-\bm{y}\|_{2}^{2}.$
(10)

####  4.2.2 Baselines

We compare our approach to a broad range of recent approaches that allow for
shape completion. Specifically, we show state-of-the-art methods for three
relevant categories of baselines methods:

  * •

Occupancy Networks [21] ONet is a classical encoder-decoder method that like
ours encodes a partial input point cloud into a global feature vector $\bm{z}$
and subsequently predicts a full geometry via a neural implicit decoder.

  * •

Convolutional Occupancy Networks [25] The second baseline ConvONet is based on
ONet, but additionally leverages local features refinement on a latent 3D
feature grid with learnable 3D convolutions.

  * •

PCN [44] On top of showing state-of-the-art neural implicit approaches, we
also compare our methods to a recent point cloud based shape completion
approach to provide a more complete picture.

####  4.2.3 Discussion

We show a quantitative comparison between our method and the considered
baselines on ShapeNet in Table 1. We further provide corresponding qualitative
comparisons in Figure 3. Analogously, we show results on SURREAL in Figure 2.
To give a more complete picture, in the latter case we also plot the total
cumulative CD curves for each method, additionally to showing the mean CD
values. Qualitative comparisons on SURREAL are shown in Figure 4.

As seen in Table 1 and Figure 2, our method achieves excellent quantitative
performance on rigid objects such as Shapenet as well as non-rigid humans of
the SURREAL dataset. Especially on the deformable human shapes from SURREAL,
our method outperforms the considered baselines by a significant margin of
$91\%$ and $58\%$, respectively. Aside from the quantitative performance, our
qualitative comparisons in Figure 3 and Figure 4 further highlight the
superior visual quality of our obtained reconstructions on both datasets.

On SURREAL, the baselines ONet[21] and ConvONet[25] frequently fail to
complete specific missing details, such as the feet and hands, see Figure 4.
On the man-made objects from ShapeNet, our method shows a much more robust
capacity to generalize to unseen test poses. We believe that this can, by and
large, be attributed to our learned adversarial shape priors. While encoder-
decoder based baselines like ONet simply learn to encode the full training
pose $\bm{X}_{\mathrm{full}}$, they often fail to generalize to previously
unseen object geometries and instead opt for reconstructing the closest
example from the training set. While ConvONet is, in principle, able to
reproduce fine scale details, in our observation it also proves to be less
stable when faced with partial input geometries – it is mainly designed for
full object level reconstruction or reconstructing large-scale scenes. The
local 3D convolution refinement proves to be a disadvantage for shape
completion since, while providing more expressivity, it also tends to make the
decoder less robust for imperfect inputs.

CD ($\downarrow$) | Ours | ONet | ConvONet | PCN-Coarse | PCN-Dense
---|---|---|---|---|---
Lamp | 9.49 | 9.99 | 25.92 | 9.20 | 8.73
Display | 16.51 | 15.79 | 17.22 | 18.47 | 18.45
Benches | 18.63 | 18.77 | 21.00 | 22.13 | 21.81

Table 1: A comparison of our method with other baselines approaches for shape
completion on the Shapenet dataset. For each setting, we show the mean CD,
averaged over 30000 points. The results in the Table are scaled by a factor of
$10^{3}$ for readability. The partial inputs $\bm{V}_{\mathrm{part}}$ consist
of $n=500$ points for Ours, ONet[21] and ConvONet[25]. For PCN, we show
results for their two separate network settings, see [44] for more details.
Additionally, to maintain consistency with the training of PCN, we test this
specific baseline on a resolution of $n=8000$ points on the same partial
snippets as the other methods.

Figure 2: We show cumulative Chamfer distance curves on SURREAL for our
method, compared to the baselines ONet [21], ConvONet [25] and PCN [44]. Each
curve shows the ratio of points below a certain CD threshold for CD values
between $[0,0.05]$. The left plot reports the distance of the generated shape
$\bm{X}_{\mathrm{rec}}$ to the input $\bm{X}_{\mathrm{full}}$, corresponding
to the first term in Equation 10. Vice-versa, the right plot shows the
distance of $\bm{X}_{\mathrm{full}}$ to $\bm{X}_{\mathrm{rec}}$. Analogous to
Table 1, we further report the mean CD values in the lower right corner
(legend) of each plot.

\begin{overpic}[width=433.62pt]{figures/qualitative/bench_qual_001.png}
\put(3.0,16.0){$\bm{V}_{\mathrm{part}}$}
\put(20.0,16.0){$\bm{X}_{\mathrm{full}}$} \put(35.0,16.0){ConvONet}
\put(56.0,16.0){PCN} \put(73.0,16.0){ONet} \put(90.0,16.0){Ours} \end{overpic}

Figure 3: Qualitative comparison – ShapeNet. We show two separate qualitative
examples for all three object classes of ShapeNet considered in Table 1,
namely ’Bench’, ’Display’, Lamp’. These results highlight clearly that,
compared to other existing methods, our learned adversarial shape priors allow
for a more robust generalization performance on unseen test samples.

\begin{overpic}[width=433.62pt]{figures/qualitative/surreal_qual_001.png}
\put(6.0,23.0){$\bm{V}_{\mathrm{part}}$}
\put(23.0,23.0){$\bm{X}_{\mathrm{full}}$} \put(38.0,23.0){ConvONet}
\put(59.0,23.0){PCN} \put(76.0,23.0){ONet} \put(93.0,23.0){Ours} \end{overpic}

Figure 4: Qualitative comparison – SURREAL. We show several different
qualitative completion results corresponding to the analysis in Figure 2. We
observe that, while the considered baselines are able to predict a meaningful
global pose for most partial inputs, the predictions from our method are
generally more robust. Overall, our approach tends to produce more stable
results at the extremities (arms and legs).

###  4.3 Ablation study

To provide a deeper understanding our how our approach learns to complete
partial shapes, we show two additional experimental settings. Moreover, we
show additional ablation studies in our supplementary material.

####  4.3.1 Sampling density

We assess how varying the number of test points $n$ of the partial inputs
$\bm{V}_{\mathrm{part}}$ affects our performance on SURREAL. More
specifically, we gradually increase the number of input points from $n=50$ to
$n=8000$ and report the resulting accuracies in Figure 5. The overall insight
here is that our method is fairly robust to sparser inputs. We attribute this
to two factors: For once, our model is trained in a progressive manner which
allows it to learn the underlying adversarial priors at various scales, see
Section 3.3 for more details. Moreover, by choosing a PointNet architecture as
our encoder $\mathcal{E}$, the network has a certain degree of robustness
built-in. PointNet is based on independent feature refinement on the input
points and global feature pooling ($\max$ pool), both of which are relatively
stable under varying point densities.

####  4.3.2 Partial cut-off ratio

The second ablation we consider investigates how the degree of partiality
affects the prediction results at test time. More specifically, we take our
model trained on SURREAL with partiality ratios of $\mathcal{U}(0.5,1)$ and
test it for different degrees of partial test shapes $\bm{V}_{\mathrm{part}}$
see Table 2.

Figure 5: We show the effect of varying the density of sampled points $n$
(x-axis) on the partial inputs $\bm{V}_{\mathrm{part}}$. For each $n$, we
denote the test set performance of our model in terms of the CD on SURREAL,
analogous to the results in Figure 2. These results show that even for
extremely sparse samplings $n=50$ the CD to the ground-truth meshes does not
change by a significant margin. For $n>1000$, the performance plateaus,
indicating that increasing the resolution beyond this point does not benefit
our method.

Partial % | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0
---|---|---|---|---|---|---|---|---|---|---|---
CD | 253.883 | 170.93 | 58.08 | 32.42 | 20.93 | 13.19 | 8.24 | 5.65 | 4.24 | 3.32 | 2.88
Gen to g.t. CD | 6.54 | 6.03 | 6.48 | 5.62 | 4.46 | 3.36 | 2.47 | 2.01 | 1.78 | 1.62 | 1.62
G.t. to gen CD | 247.33 | 164.89 | 51.59 | 26.80 | 16.46 | 9.82 | 5.76 | 3.63 | 2.45 | 1.70 | 1.26

Table 2: Ablation study on SURREAL for test shapes with varying degrees of
partialites in the range $[0.05,1]$. In each setting, we train our model on
the standard training set with ratios $[0.5,1]$ of retained points, see
Section 3.3 for more details.

##  5 Conclusion

We presented a novel approach to shape completion that combines a neural
implicit surface architecture with learnable, adversarial shape priors. In
that manner, our network effectively decouples the task of shape completion
into (1) reconstructing objects that are meaningful and (2) fitting shapes
from the learned distribution to a conditional, partial observation. We
demonstrate state-of-the-art performance on both ShapeNet and SURREAL,
highlighting our network’s flexibility of completing both rigid, man-made
objects and non-rigidly deformable shape categories. Our method yields robust
predictions for various degrees of partiality and different sampling
densities. Finally, we show a variety of qualitative comparisons that
highlight the superior visual quality of the shape prior learned by our
adversarial model.

## Acknowledgements

This work was supported by the ERC Advanced Grant SIMULACRON and the Munich
School of Machine Learning.

## References

  * [1] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3d point clouds. In: International conference on machine learning. pp. 40–49. PMLR (2018)
  * [2] Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein gan (2017)
  * [3] Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016)
  * [4] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. Tech. Rep. arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago (2015)
  * [5] Chen, X., Zheng, Y., Black, M.J., Hilliges, O., Geiger, A.: Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11594–11604 (2021)
  * [6] Chibane, J., Alldieck, T., Pons-Moll, G.: Implicit functions in feature space for 3d shape reconstruction and completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6970–6981 (2020)
  * [7] Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In: European conference on computer vision. pp. 628–644. Springer (2016)
  * [8] Dai, A., Ruizhongtai Qi, C., Nießner, M.: Shape completion using 3d-encoder-predictor cnns and shape synthesis. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5868–5877 (2017)
  * [9] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 605–613 (2017)
  * [10] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks (2014)
  * [11] Gropp, A., Yariv, L., Haim, N., Atzmon, M., Lipman, Y.: Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002.10099 (2020)
  * [12] Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: A papier-mâché approach to learning 3d surface generation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 216–224 (2018)
  * [13] Häne, C., Tulsiani, S., Malik, J.: Hierarchical surface prediction for 3d object reconstruction. In: 2017 International Conference on 3D Vision (3DV). pp. 412–420. IEEE (2017)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
  * [15] Jiang, C., Huang, J., Tagliasacchi, A., Guibas, L.J.: Shapeflow: Learnable deformation flows among 3d shapes. Advances in Neural Information Processing Systems 33, 9745–9757 (2020)
  * [16] Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018), https://openreview.net/forum?id=Hk99zCeAb
  * [17] Kleineberg, M., Fey, M., Weichert, F.: Adversarial generation of continuous implicit shape representations (2020)
  * [18] Litany, O., Bronstein, A., Bronstein, M., Makadia, A.: Deformable shape completion with graph convolutional autoencoders. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1886–1895 (2018)
  * [19] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG) 34(6), 1–16 (2015)
  * [20] Lorensen, W.E., Cline, H.E.: Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163–169 (1987)
  * [21] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4460–4470 (2019)
  * [22] Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Occupancy flow: 4d reconstruction by learning particle dynamics. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 5379–5389 (2019)
  * [23] Osher, S., Fedkiw, R.: Signed distance functions. In: Level set methods and dynamic implicit surfaces, pp. 17–22. Springer (2003)
  * [24] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 165–174 (2019)
  * [25] Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., Geiger, A.: Convolutional occupancy networks. In: European Conference on Computer Vision. pp. 523–540. Springer (2020)
  * [26] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. arXiv preprint arXiv:1612.00593 (2016)
  * [27] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space (2017)
  * [28] Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2304–2314 (2019)
  * [29] Saito, S., Yang, J., Ma, Q., Black, M.J.: Scanimate: Weakly supervised learning of skinned clothed avatar networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2886–2897 (2021)
  * [30] Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., Funkhouser, T.: Semantic scene completion from a single depth image. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1746–1754 (2017)
  * [31] Stutz, D., Geiger, A.: Learning 3d shape completion from laser scan data with weak supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1955–1964 (2018)
  * [32] Tchapmi, L.P., Kosaraju, V., Rezatofighi, H., Reid, I., Savarese, S.: Topnet: Structural point cloud decoder. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 383–392 (2019)
  * [33] Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6411–6420 (2019)
  * [34] Tieleman, T., Hinton, G.: Lecture 6.5 - rmsprop, coursera: Neural networks for machine learning. technical report (2012)
  * [35] Tiwari, G., Sarafianos, N., Tung, T., Pons-Moll, G.: Neural-gif: Neural generalized implicit functions for animating people in clothing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11708–11718 (2021)
  * [36] Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M.J., Laptev, I., Schmid, C.: Learning from synthetic humans. In: CVPR (2017)
  * [37] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2mesh: Generating 3d mesh models from single rgb images. In: Proceedings of the European conference on computer vision (ECCV). pp. 52–67 (2018)
  * [38] Wang, S., Mihajlovic, M., Ma, Q., Geiger, A., Tang, S.: Metaavatar: Learning animatable clothed human models from few depth images. Advances in Neural Information Processing Systems 34 (2021)
  * [39] Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog) 38(5), 1–12 (2019)
  * [40] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems 32 (2019)
  * [41] Yang, Y., Feng, C., Shen, Y., Tian, D.: Foldingnet: Point cloud auto-encoder via deep grid deformation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 206–215 (2018)
  * [42] Yenamandra, T., Tewari, A., Bernard, F., Seidel, H.P., Elgharib, M., Cremers, D., Theobalt, C.: i3dmm: Deep implicit 3d morphable model of human heads. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12803–12813 (2021)
  * [43] Yu, X., Rao, Y., Wang, Z., Liu, Z., Lu, J., Zhou, J.: Pointr: Diverse point cloud completion with geometry-aware transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12498–12507 (2021)
  * [44] Yuan, W., Khot, T., Held, D., Mertz, C., Hebert, M.: Pcn: Point completion network. In: 2018 International Conference on 3D Vision (3DV). pp. 728–737. IEEE (2018)

##  Appendix 0.A Network architecture details

####  0.A.0.1 Encoder

The encoder $\mathcal{E}$ converts an input point cloud
$\bm{V}_{\mathrm{part}}\in\mathbb{R}^{n\times 3}$ with $n$ points into a high-
dimensional latent encoding $\bm{z}\in\mathbb{R}^{512}$. The specific
architecture is based on PointNet [26]. It comprises 4 linear layers followed
by leaky ReLU non-linearities and a max-pool operation over the points $n$
after the last linear layer. The linear layers change the feature dimension
from 3, representing the spatial coordinates, to 64, 128, 256, and finally to
512 dimensions. The max pool operation after the last linear layer reduces the
entire feature tensor to a single latent vector $\bm{z}\in\mathbb{R}^{512}$
per input shape.

####  0.A.0.2 Generator

The generator $\mathcal{G}$ is responsible for predicting the SDF values for a
set of query points $\bm{x}_{1},\dots,\bm{x}_{k}\in\mathbb{R}^{{k\times 3}}$.
Apart from the points $\bm{x}_{i}$, the second input to $\mathcal{G}$ is a
vector $\bm{z}\in\mathbb{R}^{512}$, containing a latent representation of the
partial input point cloud $\bm{V}_{\mathrm{part}}$ obtained from the encoder
$\mathcal{E}$. $\mathcal{G}$ comprises 8 linear layers, all operating on 128
dimension features, where each linear layer is followed by layer normalization
[3] and non-linear activation. In each layer, we use leaky ReLU as the non-
linearity, since these were shown to increase the generation robustness in the
context of generative adversarial networks. It furthermore contains a linear
layer that maps the input latent code of the partial shape $\bm{z}$ from a 512
(output size of the encoder defined in 3.1.1) to a 128 dimensional vector.
Similarly, the points $\bm{x}_{i}$ are converted from $3$ input dimensions to
$128$ latent dimensions in a linear layer. The two 128-dim latent vectors
produced by the initial input layers are added up before being passed to the
second layer.

To facilitate gradient flow and increase training stability, we use skip
connections [14] between the input and the 5th layer. Similar to the first
layer, $\bm{z}$ is first passed through a linear layer before being added to
the output of the fourth layer.

####  0.A.0.3 Discriminator

We use a standard PointNet++ [27] classification architecture without batch
normalization layers, as our discriminator $\mathcal{D}_{\bm{U}}$. The module
is composed of what the authors call ’set abstraction layers’, where the
points are recursively sampled, grouped and passed through individual PointNet
layers. The purpose of this scheme is to learn the features in a hierarchical
way, which allows it to extract fine grained details. After a series of
abstraction layers and subsequent pooling, the resulting feature tensor is
passed through three linear layers, the final one of which outputs a scalar
value corresponding to the discriminator’s output classification of
true/false.

##  Appendix 0.B Results on additional ShapeNet classes

In Table 1 (main paper), we show a quantitative comparison between our
approach and various baselines. The experiment in the main paper comprises of
3 specific classes of ShapeNet that we deemed to be most representative and
most suitable for capturing the pose and geometry diversity in the whole
dataset. For completeness, we also show quantitative results on the remaining
9 classes of ShapeNet here, see Table 3. The training set of each class is
defined by randomly selecting 500 samples, while the evaluation was performed
on the full test sets.

CD ($\downarrow$) | Ours | ONet | ConvONet | PCN-Coarse | PCN-Dense
---|---|---|---|---|---
Chair | 10.49 | 15.59 | 12.43 | 11.33 | 10.75
Cabinet | 18.27 | 24.09 | 20.03 | 21.92 | 21.34
Table | 19.38 | 23.13 | 19.52 | 22.05 | 21.23
Car | 2.34 | 2.32 | 3.08 | 1.79 | 1.41
Vessel | 5.01 | 4.63 | 4.45 | 4.11 | 3.71
Loudspeaker | 17.42 | 24.84 | 19.70 | 19.23 | 18.59
Sofa | 11.42 | 11.93 | 11.65 | 12.49 | 11.90
Telephone | 13.68 | 15.72 | 12.61 | 14.40 | 14.32
Airplane | 2.49 | 2.32 | 2.42 | 2.16 | 1.88

Table 3: We show additional quantitative comparisons of our method to our
baselines ONet [21], ConvONet [25], and PCN [44] for the task of shape
completion on the 9 ShapeNet classes in addition to the 3 classes in Table 1.
Similar to Table 1, the average CD is scaled by a factor of $10^{3}$ for
readability. We observe that different classes have results on different CD
scales, varying by up to one order of magnitude. Aside from the different
complexities of individual categories, this can be in part attributed to
varying vertex densities in the ground-truth meshes.

##  Appendix 0.C Additional ablations

We perform a number of additional ablations to provide further insights and
assess the role of the various sub-components of our network.

###  0.C.1 Training set size

We perform an ablation to study the behaviour of our network in the low-
training data regime, see Figure 6. Our results indicate that decreasing the
number of training shapes does not significantly impact the performance of our
model. In fact, while the CD error increases for small $\\#$ training samples,
the training remains relatively stable even when up to $\approx 99\%$ of the
training data is removed.

Figure 6: An ablation study assessing how the number of training shapes
impacts the quantitative performance of our model. Specifically, we report the
test set CD for the task of shape completion on the ’Benches’ class of
ShapeNet. The setting of $422$ training shapes (right end of the curve) is
equivalent to the result reported in Table 1, with a CD of $18.63$. We
conclude that our method is relatively robust, even if the number of samples
in the training set are reduced considerably.

###  0.C.2 Network ablation

Finally, we study how two specific components of our network impact the
reconstruction accuracy. For once, we compare how replacing our discriminator
$\mathcal{D}_{\bm{U}}$ based on PointNet++ [27] with a more primitive PointNet
[26] discriminator changes the results. For both discriminators
$\mathcal{D}_{\bm{U}}$, we assess the impact of removing the normal loss
$\ell_{\mathrm{norm}}$ from our total loss function $\ell$.

Results are summarized in Figure 7. We report the resulting test set
performance, equivalent to the shape completion experiment on SURREAL in
Figure 2 in the main paper. Additionally, we show sample reconstructions to
assess the qualitative impact of the different settings. It is immediately
evident that there is a noticeable advantage of ‘Ours’ which is based on
PointNet++ and includes the normal loss $\ell_{\mathrm{norm}}$. The PointNet
discriminator fails to capture fine details of the reconstructed shapes and is
relatively ineffective at inpainting missing parts. On the other hand, without
the normal loss $\ell_{\mathrm{norm}}$, the reconstructions lack fine details,
especially at salient regions such as the feet and hands of the human. These
trends are also clearly reflected in the quantitative comparisons in Figure 7.

Setting | CD ($\downarrow$)
---|---
(i) PN w/o $\ell_{\mathrm{norm}}$ | 12.30
(ii) PN w/ $\ell_{\mathrm{norm}}$ | 12.90
(iii) PN++ w/o $\ell_{\mathrm{norm}}$ | 10.82
(iv) Ours | 10.70

\begin{overpic}[width=433.62pt]{figures/ablation/abl_disc_001.png}
\put(6.0,29.0){$\bm{V}_{\mathrm{part}}$} \put(30.0,29.0){(i)}
\put(50.0,29.0){(ii)} \put(70.0,29.0){(iii)} \put(90.0,29.0){(iv)}
\end{overpic}

Figure 7: We show a quantitative and qualitative ablation study to assess the
role of two design choices in our model. We assess the impact of replacing the
PointNet++ [27] discriminator architecture $\mathcal{D}_{\bm{U}}$ with the
more primitive PointNet [26]. Additionally, we show a version of both networks
where we remove the normal loss component $\ell_{\mathrm{norm}}$. Overall, the
four settings are: (i) PointNet without $\ell_{\mathrm{norm}}$, (ii) PointNet
with $\ell_{\mathrm{norm}}$, (iii) PointNet++ without $\ell_{\mathrm{norm}}$,
and (iv) PointNet++ with $\ell_{\mathrm{norm}}$. The latter case (iv) is
equivalent to our full model, as introduced in the main paper. We show a
quantitative comparison on the test set of SURREAL (left side), as well as a
qualitative sample reconstruction (right side). These results indicate that
both the PointNet++ discriminator and the normal loss are crucial for an
optimal shape completion performance.