# Laser: Latent Set Representations
for 3D Generative Modeling

Pol Moreno &Adam R. Kosiorek∗ &Heiko Strathmann &Daniel Zoran &Rosalia
Schneider &Björn Winckler &Larisa Markeeva &Théophane Weber &Danilo J. Rezende
&DeepMind Equal contribution. Correspondence to<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
###### Abstract

Neural Radiance Field (nerf) provides unparalleled fidelity of novel view
synthesis—rendering a 3D scene from an arbitrary viewpoint. nerf requires
training on a large number of views that fully cover a scene, which limits its
applicability. While these issues can be addressed by learning a prior over
scenes in various forms, previous approaches have been either applied to
overly simple scenes or struggling to render unobserved parts. We introduce
Latent Set Representations for nerf-vae (Laser-NV)—a generative model which
achieves high modelling capacity, and which is based on a set-valued latent
representation modelled by normalizing flows. Similarly to previous amortized
approaches, Laser-NV learns structure from multiple scenes and is capable of
fast, feed-forward inference from few views. To encourage higher rendering
fidelity and consistency with observed views, Laser-NV further incorporates a
geometry-informed attention mechanism over the observed views. Laser-NV
further produces _diverse and plausible_ completions of occluded parts of a
scene while remaining consistent with observations. Laser-NV shows state-of-
the-art novel-view synthesis quality when evaluated on ShapeNet and on a novel
simulated City dataset, which features high uncertainty in the unobserved
regions of the scene. See laser-nv-paper.github.io for video results, code and
data.

##  1 Introduction

Probabilistic scene modelling aims to learn stochastic models for the
structure of 3D scenes, which are typically only partially observed (Eslami et
al., 2018; Kosiorek et al., 2021; Burgess et al., 2019). Such models need to
reason about unobserved parts of a scene in way that is consistent with the
observations and the data distribution. Scenes are usually represented as
latent variables, which are ideally compact and concise, yet expressive enough
to describe complex data.

Such 3D scenes can be thought of as projections of light rays onto an image
plane. Neural Radiance Field (nerf, Mildenhall et al. (2020)) exploits this
structure explicitly. It represents a scene as a radiance field (a.k.a. a
scene function), which maps points in space (with the corresponding camera
viewing direction) to color and mass density values. We can use volumetric
rendering to project these radiance fields onto any camera plane, thus
obtaining an image. Unlike directly predicting images with a CNN, this
rendering process respects 3D geometry principles. nerf represents scenes as
parameters of an mlp, and is trained to minimize the reconstruction error of
observations from a single scene—resulting in unprecedented quality of novel
view synthesis.

For generative modelling, perhaps the most valuable property of nerf is the
notion of 3D geometry embedded in the rendering process, which does not need
to be learned, and which promises strong generalisation to camera poses
outside the training distribution. However, since nerf’s scene representations
are high dimensional mlp parameters, they are not easily amenable to
generative modelling (Dupont et al., 2022). nerf-vae (Kosiorek et al., 2021)
embeds nerf in a generative model by conditioning the scene function on a
latent vector that is inferred from a set of ‘context views’. It then uses
nerf’s rendering mechanism to generate outputs. While nerf-vae admits
efficient inference of a compact latent representation, its outputs lack
visual fidelity. This is not surprising, given its simple latent structure,
and the inability to directly incorporate observed features. In addition,
nerf-vae does not produce varied samples of unobserved parts of a scene.

A number of recent deterministic methods (Yu et al., 2021; Trevithick & Yang,
2021; Wang et al., 2021) uses local image features to directly condition
radiance fields in 3D. This greatly improves reconstruction quality of
observed parts of the scene. However, these methods are still unable to
produce plausible multimodal predictions for the unobserved parts of a scene.

Figure 1:  Left: Laser-NV infers a set-valued latent $\mathcal{Z}$ from the
context $\mathcal{V}$ that consists of $N$ image and camera pairs
$(\mathbf{I}^{n},\mathbf{C}^{n})$. On querying the scene function at a point
$\mathbf{x}_{i}$ with direction $\mathbf{d}_{i}$, the latents are combined
with local features $\mathbf{H}^{n}$ that are back-projected from the context
views—producing color and density. Right: Rendering a novel viewpoint may
include observed (green, an example query point on the left) and unobserved
parts (red) of the scene. Conditioned on the context $\mathcal{V}$, Laser-NV
allows sampling multiple scene completions that are consistent with the
context views (green arrows) while providing varied explanations for the
unobserved parts of the scene (red arrows). We show two such samples for the
same target camera $\mathbf{C}^{t}$. Also see laser-nv-paper.github.io for
additional videos generated by the model.

In this work, we address nerf-vae’s shortcomings by proposing Latent Set
Representations for nerf-vae (Laser-NV). To increase modelling capacity, we
follow Perceiver (Jaegle et al., 2021) and use an arbitrarily-sized set of
latent variables instead of just one vector. Unlike Perceiver, Laser-NV models
this set probabilistically using normalizing flows. To further enable
producing samples which are consistent with observed parts, we make the
generative model conditional on a set of context views (as opposed to
conditioning only the approximate posterior). Laser-NV offers superior visual
quality with the ability to synthesise multiple varied novel views compatible
with observations. Figure 1 shows Laser-NV’s key components and abilities. See
laser-nv-paper.github.io for additional results, code and the City dataset.

Our contributions are as follows:

  * •

We introduce a novel set-valued latent representation modelled by purpose-
built permutation-invariant normalizing flows conditioned on context views. We
show that increasing the number of latent set elements improves modelling
performance, providing a simple way to trade off computation for quality
without adding new model parameters. We also verify that increasing latent
dimensionality in nerf-vae offers no such benefits. In contrast with
deterministic scene models in the literature, our probabilistic treatment over
the latent set allows covering multiple models when predicting novel views.

  * •

We develop a novel attention mechanism to condition the scene function on the
set-valued latent as well as additional local features computed from context
views. We show that including local features further improves visual quality.

  * •

We evaluate Laser-NV on three datasets: a category-agnostic ShapeNet dataset,
MultiShapeNet, and on a novel “City” dataset that contains a large simulated
urban area and poses significant challenges as a benchmark for novel view
synthesis due to high uncertainty in the unobserved parts of the scene. Our
model overcomes some of the main limitations of nerf-vae and also outperforms
deterministic nerf models on novel view synthesis in the face of uncertainty.

##  2 Background: NeRF & NeRF-VAE

Neural Radiance Field (nerf, (Mildenhall et al., 2020)) represents a 3D scene
as a scene function $F(\bm{x},\bm{d})$ of a point coordinate
$\bm{x}\in\mathbb{R}^{3}$ and direction $\bm{d}\in\mathbb{R}^{3}$, and outputs
the colour $\bm{c}\in\mathbb{R}^{3}$ and volume density $\sigma\geq 0$. This
function is parameterized using a neural network (typically a fully connected
MLP). To obtain a pixel colour, the scene function is evaluated at many points
along the corresponding ray in the volumetric rendering process.

This volumetric rendering integral is in practice approximated using numerical
integration; see (Mildenhall et al., 2020; Blinn, 1982) for details. We use
$\hat{\bm{I}}={\operatorname{\texttt{render}}}(F(\cdot),\bm{C})$ to denote the
image rendering process that outputs the image
$\hat{\bm{I}}\in\mathbb{R}^{H\times W\times 3}$ for the rays of a camera
$\bm{C}$ and a scene function $F$. The camera $\bm{C}=(\bm{K},\bm{R},\bm{t})$
is specified by its intrinsic parameters $\bm{K}\in\mathbb{R}^{3\times 3}$ and
extrinsic parameters given by a rotation $\bm{R}\in\mathbb{R}^{3\times 3}$ and
translation $\bm{t}\in\mathbb{R}^{3}$. In its original formulation, the
parameters of NeRF are optimized by minimizing an error function (typically
the mean-squared error) between rendered images and the ground-truth images
across many views of a single scene.

While nerf learns high-fidelity scene representations when many views are
available, it does not learn a prior over scenes, does not provide compact
scene representations, and does not admit efficient inference (i.e. fast
estimation of the scene parameters from a few input views). To address these
issues, nerf-vae (Kosiorek et al., 2021) embeds nerf as a decoder in a
variational auto-encoder (vae , (Kingma & Welling, 2014; Rezende et al.,
2014)). Each scene is represented by a latent vector $\bm{z}$ (instead of the
set of parameters of an MLP as in NeRF) which can be either sampled from a
Gaussian prior $p\left(\bm{z}\right)$ or from the approximate posterior
$q\left(\bm{z}\mathrel{}\middle|\mathrel{}\bm{C}\right)$ inferred from a set
of context views and associated cameras $\bm{C}$. The latent $\bm{z}$ is used
to condition an MLP that parameterizes the scene function,
$F_{\theta}(\cdot,\bm{z}):(\bm{x}_{i},\bm{d}_{i})\mapsto\bm{c}_{i},\bm{\sigma}_{i}$.
Here, parameters $\theta$ are shared between scenes, while $\bm{z}$ is scene
specific. nerf-vae uses a Gaussian likelihood
$p\left(\bm{I}\mathrel{}\middle|\mathrel{}\bm{z},\bm{C}\right)$ with the image
rendered from the sampled scene function as mean, and a fixed standard
deviation.

##  3 Latent Set Representations for nerf-vae

LASER-NV is a conditional generative model, conditioned on a set of input
views $\mathcal{V}\coloneqq\\{\bm{I}^{n},\bm{C}^{n}\\}^{N}_{n=1}$. Given these
views the model infers a latent representation that can be used for novel view
synthesis.

In order to represent large-scale and complex scenes we use Latent Set
Representations (lasers), a set of latents of the form
$\mathcal{Z}:={\left\\{\bm{z}_{k}\right\\}}_{k=1}^{K}$, where $K$ is a
hyperparameter. In the nerf-vae, the scene function MLP is directly
conditioned on the inferred latent. In contrast, Laser-NV’s scene function is
conditioned on the latent set $\mathcal{Z}$ and features
$\mathcal{H}:=\\{\bm{H}^{n}\\}_{n=1}^{N}$ extracted from $N$ input views, and
integrates information from both. This results in the form
$F_{\theta}(\cdot,\mathcal{Z},\mathcal{H}):(\bm{x_{i}},\bm{d_{i}})\mapsto\bm{c}_{i},\bm{\sigma}_{i}$,
which we detail further in Section 3.2.

We obtain the input features $\mathcal{H}$ by separately encoding the input
views using a convolutional neural net (CNN) that maintains spatial structure.
These feature maps are subsequently used in the conditional prior, the
posterior, and the resulting scene function, all of which we describe next.
Further architectural details are provided in Appendix A.

###  3.1 Conditional Prior and Posterior

Given a set of context views
$\mathcal{V}\coloneqq\\{\bm{I}^{n},\bm{C}^{n}\\}^{N}_{n=1}$ consisting of
images $\bm{I}^{n}$ and corresponding cameras $\bm{C}^{n}$, Laser-NV defines a
conditional prior $p(\mathcal{Z}|\mathcal{V})$ over a latent set of size $K$,
parameterized by a permutation-invariant normalizing flow— a distribution
model that allows for both fast and exact sampling and efficient density
evaluation (Rezende & Mohamed, 2015; Papamakarios et al., 2021).

Flows are defined by an invertible mapping $f$ of random variables, starting
from a base distribution $p_{0}\left(\mathcal{Z}^{{(}0{)}}\right)$. When $f$
is composed of multiple invertible mappings $f_{1},\ldots,f_{m}$, the
resulting density is given by

$p\left(\mathcal{Z}\mathrel{}\middle|\mathrel{}\mathcal{V}\right)=p_{0}\left(f^{-1}\left(\mathcal{Z},\mathcal{V}\right)\right)\prod_{m=1}^{M}\left|\det\frac{\delta
f_{m}\left(\mathcal{Z}^{{(}m-1{)}},\mathcal{V}\right)}{\delta\mathcal{Z}^{{(}m-1{)}}}\right|^{-1}\,.$
(1)

Our flow design closely follows that of Wirnsberger et al. (2020; 2021), who
parameterize the transformations in each layer using transformers, preserving
permutation invariance. Specifically, we propose a novel design to condition
the flow distribution on the features $\mathcal{H}$ extracted from the context
views. To achieve that, we use split coupling layers (Dinh et al., 2017),
where for each layer, we first apply self-attention to the set of latents,
followed by cross-attention to attend the context features. The output of the
second transformer defines the parameters for an invertible affine
transformation of the random variables. Implementation details of our
normalizing flow is in Section A.1.

Since Laser-NV is trained as a VAE, we define an _approximate posterior_
$q(\mathcal{Z}|\mathcal{U}\cup\mathcal{V})$, where
$\mathcal{U}=\\{\bm{I}^{t},\bm{C}^{t}\\}^{M}_{t=1}$ are target views that the
model is trained to reconstruct, denoted as posterior context. The posterior
and conditional prior are both distributions over the latent space, where the
posterior is conditioned on the union of prior and posterior context. We model
the posterior distribution using the same architecture as the conditional
prior, but with a separate set of learned parameters.

###  3.2 Decoder & Scene Function

When evaluating the scene function, i. e. computing colour and density for a
point $\mathbf{x}_{i},\mathbf{d}_{i}$, Laser-NV has access to both the latent
set $\mathcal{Z}$ as well as the features $\mathcal{H}$ extracted from the
context views $\mathcal{V}$. We now describe how to integrate those two, which
consists of querying the latent set for each point, and using 3D projection to
extract relevant features.

#### Querying the Latent Set

While the latent set representation $\mathcal{Z}$ contains information
regarding the entire scene, when querying for a specific point $\bm{x}_{i}$,
we would like to extract only relevant information from $\mathcal{Z}$. We
propose using a transformer cross-attention model to compute a single feature
vector from $\mathcal{Z}$. In the first layer, queries are computed from the
positional encoding of $\bm{x}_{i}$, and keys and values are computed from the
elements of $\mathcal{Z}$. In subsequent layers, queries are computed from
previous layers’ output. We denote the output of the latent querying as latent
features $\tilde{\bm{z}}_{i}$, each corresponding to a query point
$\bm{x}_{i}$.

#### Local Features

When synthesizing a novel view given a few input views, a number of existing
nerf-based methods uses 3D projection to determine which features from the
context views are useful for explaining a particular 3D point (Yu et al.,
2021; Trevithick & Yang, 2021; Wang et al., 2021). Laser-NV incorporates this
idea, and in particular follows the design of pixelNeRF (Yu et al., 2021) as
it is shown to achieve strong results in a wide range of domains.

We use the encoded input views $\\{\bm{H}^{n}\\}^{N}_{n=1}$ described above,
along with their corresponding camera matrices $\\{\bm{C}^{n}\\}^{N}_{n=1}$.
Given a query 3D point $\bm{x}_{i}$ and direction $\bm{d}_{i}$, we obtain the
point $\bm{p}^{n}_{i}\in\mathbb{R}^{2}$ in the image space of the
n${}^{\text{th}}$ context view using the known intrinsic parameters. The
corresponding local features are then bilinearly interpolated
$\bm{h}^{n}_{i}=\bm{H}^{n}[\bm{p}^{n}_{i}]$. We refer to $\bm{h}_{i}$ as
_local features_ since they are spatially localized based on the projected
coordinates, as opposed to being a global aggregation of all context features.
We finally obtain $\bm{\hat{h}}^{n}_{i}$ for each context view by separately
processing the features $\bm{h}^{n}_{i}$ using a residual MLP that has as an
additional conditioning on the image-space projected points and directions.

#### Integrating Latents and Local Features

In contrast to pixelNeRF, which does not have latents, Laser-NV integrates the
both local and the latent features. It does so with a transformer:

$\bm{f}_{i}=M_{\theta}(\tilde{\bm{z}}_{i};\bigcup_{n=1}^{N}\\{\bm{h}^{n}_{i}:\operatorname{\text{is\\_visible}}(\mathbf{x}_{i},\mathbf{C}^{n})\\}\cup\\{\tilde{\bm{z}}_{i}\\})$
(2)

The attention queries are computed from latent features $\tilde{\bm{z}}_{i}$
(see above), and attend to the combined set of local features and to
$\tilde{\bm{z}}_{i}$. Note that, for a point $\mathbf{x}_{i}$, we use only
those context views from which this point is visible. Using a transformer to
process encoded features from multiple views with epipolar constraints has
been proposed in prior work on Multi-view Stereo, see e.g. (He et al., 2020;
Xi et al., 2022; Wang et al., 2022; Ding et al., 2022).

The final step in evaluating the scene function is to use $\bm{f}_{i}$ to
condition a residual MLP similar to nerf-vae resulting in Laser-NV’s scene
function
$F_{\theta}(\cdot,\mathcal{Z},\mathcal{H}):(\mathbf{x}_{i},\mathbf{d}_{i})\mapsto\bm{c}_{i},\sigma_{i}$
which is used to volume-render image outputs.

###  3.3 Loss

Similarly to nerf-vae, our objective function is given by the evidence lower-
bound:

$\mathcal{L}\left(\mathcal{U},\mathcal{V}\right)=\mathbb{E}_{\mathcal{Z}\sim
q}\left[\sum^{M}_{t=1}\log
p\left(\bm{I}^{t}\mathrel{}\middle|\mathrel{}\bm{C}^{t},\mathcal{Z},\mathcal{V}\right){}\right]-\operatorname{KL}\left(q(\mathcal{Z}\mid\mathcal{U}\cup\mathcal{V})\mathrel{}\middle|\mathrel{}\mathrel{}\middle|\mathrel{}p\left(\mathcal{Z}\mathrel{}\middle|\mathrel{}\mathcal{V}\right){}\right)$
(3)

where $\bm{I}^{t}$ and $\bm{C}^{t}$ are the images and cameras from the
posterior context views $\mathcal{U}$. We use a Gaussian likelihood term for
$p\left(\bm{I}^{t}\mathrel{}\middle|\mathrel{}\bm{C}^{t},\mathcal{Z},\mathcal{V}\right)$
with a fixed variance.

##  4 Experiments

###  4.1 Datasets

We first evaluate Laser-NV on a novel dataset based on a set of synthetic,
procedurally generated, environments consisting of urban outdoor scenes. This
dataset contains posed images of a virtual city with objects of multiple
scales, such as house blocks, roads, buildings, street lights, etc (but no
transient objects such as cars or people). As such, it is an extremely
challenging scene representation benchmark, especially when representations
are inferred from few views only—resulting in partial observability and high
uncertainty. In this situation, when evaluated on unobserved parts of a scene,
we want a model to produce predictions that are varied, self-consistent, and
aligned with the data distribution. Each scene consists of a neighborhood
block, and a randomly chosen designated point of interest (road intersection
or park) is selected within. Images are then rendered by placing the camera at
random nearby positions and viewpoints from the point of interest. The dataset
has 100K training scenes, and 3200 test scenes. We provide further details and
dataset samples in Appendix B.

To test Laser-NV’s ability to produce near-deterministic scene completions in
a low-uncertainty setting, we compare it against a number of previous methods
on the ShapeNet NMR dataset (Kato et al., 2018), which consists of several
views of single objects from 13 different ShapeNet categories. Finally, we
evaluate on the more challenging MultiShapeNet-Hard (MSN-Hard) (Sajjadi et
al., 2022b), which tests the models capabilities in the presence of a large
number of cluttered objects of varying size with detailed textures and
realistic backgrounds.

###  4.2 Models & Evaluation

On the City dataset, we compare Laser-NV with Multi-view Conditional NeRF
(mvc-nerf) and nerf-vae; see Figure 7 in Appendix A for model diagrams. Mvc-
nerf (Figure 7(c)) is a variant of Laser-NV (Figure 7(a)) which does not use
Laser-NV’s latent set representations but instead only relies on
deterministically rendering new views using local features. It therefore
closely resembles the design of pixelNeRF but uses the same architecture for
the image encoder and scene function $F_{\theta}$ as Laser-NV to ensure they
are comparable. On ShapeNet, we further show quantitative results of a number
of non-NeRF novel view synthesis methods: Scene Representation Networks (SRN,
Sitzmann et al. (2019)), Differentiable Volumetric Rendering (DVR, Niemeyer et
al. (2020)), and Scene Representation Transformers (SRT, Sajjadi et al.
(2022b)). During training, Laser-NV uses a random subset of 4 of the 23 target
views as posterior context views111While using the 23 views as posterior
context would be the standard set up (Bayer et al., 2021), we find that using
4 randomly-sampled views is much cheaper computationally and yet sufficient
for training.. mvc-nerf and nerf-vae are trained to directly predict the 23
views given a single context view. At test time, Laser-NV samples the latents
from its conditional prior whereas nerf-vae uses its posterior (given that its
prior is unconditional). Importantly, all models in this evaluation use one
input view and predict 23 target views.

At evaluation time, we feed each model with a set of context views, and
evaluate _reconstructions_ of those same views, and _predictions_ of novel
views of the same scene. We evaluate reconstruction ability via likelihood
estimates, and standard image similarity metrics PSNR, and SSIM (Wang et al.,
2004). For novel view predictions we report the Fréchet Inception Distance
(FID, Heusel et al. (2017)) between the marginal distribution of samples and
the evaluation dataset222Image similarity metrics such as PSNR and SSIM are
not useful when there exists a large multimodal space of possible predictions
as is the case in the City.. This metric highlights the incapability of
deterministic models to generate varied and plausible samples in face of
uncertainty. Note that since nerf-vae is unconditional, we use the learned
approximate posterior as the conditional distribution. In the City, we thus
train a separate model (one to use for reconstruction metrics that has 8
posterior context views and another one with 2 views for predictions). For
ShapeNet results all models use one context view.

On MSN-Hard, we train Laser-NV with 5 prior context views and 3 target views.
We report the test PSNR of 1 novel view given 5 input views, and compare with
the results reported in Sajjadi et al. (2022a). The methods of comparison
include two amortized non-NeRF models based on light fields, Scene
Representation Transformers (SRT) and Object SRT (OSRT Sajjadi et al.
(2022a)); and two amortized NeRF methods, pixelNeRF and Volumetric OSRT
(VOSRT).

###  4.3 Ablations

We further ablate Laser-NV to isolate contributions of individual model
components. To quantify the effects of conditioning the prior we compare
Laser-NV to a conditional version of nerf-vae. Note that the original nerf-vae
used an unconditional prior (Kosiorek et al., 2021). We also compare Laser-NV
to a version that does not condition the scene function with local context
features, cf. Equation 2 and Figure 7(b) in Appendix A, denoted by Laser-NV
(no geom.). Laser-NV, Laser-NV (no geom.), nerf-vae variants, and mvc-nerf
share architectures for the image encoder and the scene function $F_{\theta}$.

###  4.4 Training and implementation details

We use the hierarchical sampling technique of (Mildenhall et al., 2020) when
rendering a pixel colour, which involves learning two separate sets of scene
function parameters as well as optimizing a corresponding colour likelihood
term for each step. For experiments with the City, we leverage ground-truth
depth maps using the method proposed by Stelzner et al. (2021) in order to
train all the models with fewer scene function evaluations per ray. Not using
ground-truth depth significantly increases training time while only marginally
lowering reconstruction PSNR, see Appendix D for a discussion and Fig. 16 in
Appendix B for visualisations. Importantly, when evaluating our models we
revert to volumetric rendering. All generative models are trained by
optimizing the evidence lower bound; where we anneal the KL term in Eq. 3. All
experiments use the Adam (Kingma & Ba, 2014) optimizer. We provide full
details of model architectures in Appendix A and training procedures in
Appendix E.

###  4.5 Results

Results on the City dataset are shown in Table 1. In terms of reconstructing
observed input views, both Laser-NV and mvc-nerf, due to the use of explicit
geometrical knowledge, result in excellent reconstructions. nerf-vae, however,
needs to reconstruct input views purely from its latent representation,
resulting in low fidelity reconstructions. For prediction of unseen views, the
output distribution of Laser-NV is closer to the actual evaluation data (lower
FID), in contrast to mvc-nerf which outputs a single prediction, and nerf-vae
which produces low quality outputs that do not vary much. We show
representative example reconstructions and predictions in Fig. 2. See laser-
nv-paper.github.io for additional results including fly-throughs of model
samples333To create some of these visualisations we trained the model on a
modified City dataset with birds-eye views..

| Reconstruction | Prediction
---|---|---
| Log-Likelihood $\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$
mvc-nerf | - | 28.63 | 0.910 | 91.92
Cond. nerf-vae | 3.51 | 23.94 | 0.709 | 50.98
nerf-vae | 3.50 | 23.92 | 0.708 | 52.12
Laser-NV (no geom.) | 3.82 | 27.13 | 0.870 | 24.19
Laser-NV | 3.83 | 29.61 | 0.923 | 22.54
Table 1: Results in City. The color likelihood is computed via importance sampling (10 samples); note that mvc-nerf is deterministic and thus we do not compare it on this metric. Reconstruction: We report performance of reconstruction of the 2 context views. Prediction: We evaluate the models on previously unobserved views using the FID score. mvc-nerf provides a single deterministic prediction whereas for the other models we average over 10 independent samples.  Figure 2:  Representative reconstructions of observed views and predictions of partially unobserved novel views. The models are conditioned on two context views (top left). The first two columns show reconstructions of those views. Note how nerf-vae struggles to reconstruct both views accurately, while the other models do well. We show predictions of two novel views (top right), where the first view is partially observed in the context (highlighted in cyan color) and the second view is fully unobserved. Laser-NV’s predictions are consistent with the observed part of the context, diverse, and plausibly so. nerf-vae’s predictions lack quality and diversity. mvc-nerf’s only produces a single prediction, which is not very plausible.  | DVR* | SRN* | SRT* | pixelNeRF* | nerf-vae | mvc-nerf | Laser-NV
---|---|---|---|---|---|---|---
PSNR $\uparrow$ | 22.70 | 23.28 | 27.87 | 26.80 | 25.36 | 28.03 | 27.92
SSIM $\uparrow$ | 0.860 | 0.849 | 0.912 | 0.910 | 0.875 | 0.925 | 0.923
Table 2: Category-agnostic ShapeNet evaluation by reconstructing 23 views from
one input view for all models. We further report numbers from the respective
papers for DVR* (Niemeyer et al., 2020), SRN* (Sitzmann et al., 2019), SRT*
(Sajjadi et al., 2022b), and pixelNeRF* (Yu et al., 2021).

Results on ShapeNet are shown in Table 2. Both mvc-nerf and Laser-NV
outperform existing published methods when predicting novel views of these
single-object scenes. We further observe low quality predictions of nerf-vae
in example predictions in Fig. 11. Finally, Fig. 12 shows a hand-selected
example where the input view has ambiguity and how Laser-NV can sample
plausible variations. As in the City experiment, Laser-NV produces plausibly
varied predictions of the object in contrast to the other methods.

Results on MSN-Hard are shown in Table 3. Laser-NV clearly outpeforms the
NeRF-based models, and improves over OSRT and the original SRT. Only a
modified SRT shows higher PSNR (25.93) over Laser-NV (24.45). Note that SRT
does not learn to estimate densities nor depth maps and is deterministic. Fig.
3 shows a prediction example using Laser-NV, and further examples are in
Appendix C which demonstrate high fidelity colours and densities.

| NeRF | Light field
---|---|---
| pixelNeRF | VOSRT | Laser-NV | OSRT | SRT | SRT++
PSNR $\uparrow$ | 21.97 | 21.38 | 24.45 | 23.33 | 23.54 | 25.93
Table 3: Novel view PSNR of one view from 5 input views on MSN-Hard. PixelNeRF
results are from Sajjadi et al. (2022b) and the remaining from Sajjadi et al.
(2022a). SRT++ is an improved SRT (Sajjadi et al., 2022a). Figure 3:  Novel
view syntesis with Laser-NV on MSN-Hard.

###  4.6 Train- and Test-time scaling & Data Efficiency

(a) Laser-NV

(b) Conditional nerf-vae

Figure 4:  Training reconstruction performance in City of Laser-NV with an
increasing latent set size (each with 128 dimensions) vs a cond. nerf-vae with
increasing latent dimensionality.

Latent set representations allow trading computation for increased model
capacity. We train Laser-NV on the City dataset with increasing latent set
sizes $K$, and compare it to a conditional nerf-vae also trained with
increasing latent dimensions444Note that this increases the number of nerf-
vae’s parameters.. Figure 4 shows reconstruction log-likelihoods as a function
of latent capacity. While Laser-NV consistently improves its reconstructions
with larger latent sets, cond. nerf-vae quickly saturates at 256 latent
dimensions, at worse reconstructions compared to Laser-NV. Further, in
Appendix D, we provide an analysis of test-time scaling—we show that
increasing the latent set size at test time does not result in clear change of
performance. Finally, in Figure 14 we show that Laser-NV performs better at
test time than nerf-vae and mvc-nerf for different training dataset sizes.

##  5 Related work

#### Neural scene representations

Inferring neural scene representations using a deep generative model of scenes
is an active research area, and we mention some of the most closely related
approaches below. nerf-vae learns a generative model of 3D scenes which is
only shown to work in relatively simple scenes. When applied to more
complicated data (see Fig. 2), it produces blurry reconstructions and novel
views. We posit that this is caused by insufficient capacity of its latent
representation $\bm{z}$. Generative Query Network (Eslami et al., 2018)
similarly learns a distribution over views of a scene but is inconsistent
between views. Subsequent work (Rosenbaum et al., 2018) improves over its
representational capacity by using an attentive mechanism when conditioning
the decoder from context in an analogous way to how Laser-NV improves over
nerf-vae. By implementing a NeRF in our decoder we gain the benefits of more
robust generalization to novel viewpoints compared to the convolutional
counterparts (Kosiorek et al., 2021). Simone (Kabra et al., 2021) is a
convolutional generative model cable of high quality novel view synthesis and
it also learns to segment input videos without supervision. Dupont et al.
(2022) propose directly modelling the underlying radiance fields as functions,
which they refer to as ‘functa’. Both our method and theirs learn a prior over
scene functions with a model that has a large representational capacity.
Bautista et al. (2022) propose a two-stage method to learn a generative model
of 3D scenes. As Laser-NV, their model can conditionally sample radiance
fields. However, they focus on scenes with dense views, whereas Laser-NV works
with sparse views.

#### Deterministic novel view synthesis

NeRF relies on optimizing the parameters of an MLP scene function, separately
for every scene. A number of methods in the literature propose amortizing the
mapping of input views to the scene function using multi-view geometry, see
e.g. IBRNet (Wang et al., 2021), General Radiance Fields (GRF, Trevithick &
Yang (2021)) and pixelNeRF (Yu et al., 2021). These methods use projective
geometry to select which context features to attend to. They are also shown to
generalize well and to carry out high-fidelity predictions. PixelNeRF is
designed to work well in a solely view-centered coordinate system. MVSNerf
(Chen et al., 2021) lifts context features to a 3D representation with strong
novel-view synthesis performance. Scene Representation Transformers (SRT)
(Sajjadi et al., 2022b) use global features obtained from context views using
a generalization of the Vision Transformer (Dosovitskiy et al., 2021b), which
bears similarities to Laser-NV. SRT bypasses the use of 3D geometry entirely,
yet shows remarkable performance on real world data.

#### Set Representations

Set representations became popular due to the success of the Transformer
architecture (Vaswani et al., 2017) even if initially they were not seen as
sets (Lee et al., 2019). In these works, the representation consists of a set
of feature vectors which are directly derived from the inputs: the size of the
set grows with the size or the number of the inputs, and often the set
elements retain positional information, see e. g. ViT (Dosovitskiy et al.,
2021a). This is in contrast to Perceiver (Jaegle et al., 2021), where the size
of the hidden set is a hyperparameter and is independent of the inputs.

In the context of scene representation, the attentive version of NeRF-VAE
(Kosiorek et al., 2021) and SRT (Sajjadi et al., 2022b) use a transformer-
style set-based representation. Unlike these models, Laser-NV follows
Perceiver which results in a representation devoid of any spatial meaning, and
whose size can be changed at test-time (see Fig. 4 for changing latent
capacity at test time). While OSRT (Sajjadi et al., 2022a) also uses a
Perceiver-like representation, it is with the purpose of assigning one token
for every object in the scene which creates a bottleneck.

#### NeRF for large-scale scenes

A number of NeRF methods have been developed to scale up to city-wide scenes.
They work by optimizing multiple separate scene functions for separate blocks
of a scene, which are then dynamically used when rendering (Tancik et al.,
2022; Turki et al., 2021). Alternatively they design a scene function that
represents the scene at multiple levels of scale (Martel et al., 2021; Xiangli
et al., 2021). Similarly, Mip-NeRF (Barron et al., 2021) allows prefiltering
the scene function inputs to better capture coarse and fine details. These
methods are orthogonal to ours since they do not learn a conditional NeRF
model sparse views nor address the uncertainty in predictions.

##  6 Conclusion

We propose Laser-NV, a conditional generative model of neural radiance fields
capable of efficient inference of large and complex scenes under partial
observability conditions. While recent advances in neural scene representation
and neural rendering lead to unprecedented novel view synthesis capabilities,
producing sharp and multi-modal completions of unobserved parts of the scene
remains a challenging problem. We experimentally show that Laser-NV can model
scenes of different scale and uncertainty structure, and isolate the
usefulness of each contribution through ablation studies. While NeRF serves as
a strong inductive bias for learning 3D structure from images, Laser-NV also
inherits some of its drawbacks. Volumetric rendering with neural radiance
fields is computationally costly and can limit the model from real-time
rendering. It remains to be seen whether fast NeRF implementations (Müller et
al., 2022) can be of help. Furthermore, accurate GT camera information is
required for learning and novel view synthesis; (Moreau et al., 2022) recently
made progress on localization methods using NeRF. While learning a generative
scene model of real scenes remains an open problem, we believe this work is an
important step in that direction. Finally, an interesting direction for future
research is incorporating object-centric structure and dynamics into Laser-NV,
which might be useful for for downstream reasoning tasks (Sajjadi et al.,
2022a; Ding et al., 2021).

### Acknowledgements

For the City environment, we would like to thank Nick Young, Tom Hudson, Alex
Platonov, Bethanie Brownfield, Sarah Chakera, Dario de Cesare, Marjorie
Limont, Benigno Uria, Borja Ibarz, Andrea Banino and Charles Blundell.
Moreover, for the City, we would like to extend our special thanks to Jayd
Matthias, Jason Sanmiya, Marcus Wainwright, Max Cant and the rest of the
Worlds Team.

We thank Mehdi S. M. Sajjadi and Klaus Greff for help with the MSN dataset. We
also thank Mehdi for clarifications of the SRT and OSRT models and for access
to detailed evaluations of these models, Alexander G. D. G. Matthews and
George Papamakarios for help with normalizing flows, and Tom Hudson for
pointers to graphics literature on background modelling.

## References

  * Barron et al. (2021) Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.  In _International Conference on Computer Vision_ , 2021.
  * Bautista et al. (2022) Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, and Josh Susskind.  GAUDI: A Neural Architect for Immersive 3D Scene Generation.  In _Advances in Neural Information Processing Systems_ , 2022 arXiv/2207.13751.
  * Bayer et al. (2021) Justin Bayer, Maximilian Soelch, Atanas Mirchev, Baris Kayalibay, and Patrick van der Smagt.  Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models.  In _International Conference on Representation Learning_ , 2021 arXiv/2101.07046.
  * Bender et al. (2020) Christopher M. Bender, Kevin O’Connor, Yang Li, Juan Jose Garcia, Junier Oliva, and Manzil Zaheer.  Exchangeable Generative Models with Flow Scans.  In _AAAI Conference on Artificial Intelligence_ , 2020.
  * Blinn (1982) James F. Blinn.  Light Reflection Functions for Simulation of Clouds and Dusty Surfaces.  In _SIGGRAPH_ , 1982.
  * Burgess et al. (2019) Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner.  MONET: Unsupervised scene decomposition and representation.  _arXiv/1901.11390_ , 2019.
  * Chen et al. (2021) Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo.  In _International Conference on Computer Vision_ , 2021 arXiv/2103.15595.
  * Ding et al. (2021) David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick.  Attention over learned object embeddings enables complex visual reasoning.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Ding et al. (2022) Yikang Ding, Wentao Yuan, Qingtian Zhu, Haotian Zhang, Xiangyue Liu, Yuanjiang Wang, and Xiao Liu.  TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformers.  In _Conference on Computer Vision and Pattern Recognition_ , 2022\.
  * Dinh et al. (2017) Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using Real NVP.  In _International Conference on Representation Learning_ , 2017 arXiv/1605.08803.
  * Dosovitskiy et al. (2021a) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _International Conference on Representation Learning_ , 2021a arXiv/2010.11929.
  * Dosovitskiy et al. (2021b) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Representation Learning_ , 2021b arXiv/2010.11929.
  * Dupont et al. (2022) Emilien Dupont, Hyunjik Kim, S. M. Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum.  From data to functa: Your data point is a function and you should treat it like one.  In _International Conference on Machine Learning_ , 2022 arXiv/2201.12204.
  * Eslami et al. (2018) SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al.  Neural scene representation and rendering.  _Science_ , (6394):1204–1210, 2018.  360\.
  * He et al. (2020) Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu.  Epipolar Transformers.  In _Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in Neural Information Processing Systems_ , 2017.
  * Jaegle et al. (2021) Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira.  Perceiver: General Perception with Iterative Attention.  In _International Conference on Machine Learning_ , 2021.
  * Kabra et al. (2021) Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerchner, and Chris Burgess.  Simone: View-invariant, temporally-abstracted object representations via unsupervised video decomposition.  _Advances in Neural Information Processing Systems_ , 2021.
  * Kato et al. (2018) Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada.  Neural 3D Mesh Renderer.  In _Conference on Computer Vision and Pattern Recognition_ , 2018\.
  * Kingma & Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference for Learning Representations_ , 2014 arXiv/1412.6980.
  * Kingma & Welling (2014) Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In _International Conference for Learning Representations_ , 2014\.
  * Kosiorek et al. (2021) Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosália G. Schneider, Soňa Mokrá, and Danilo Jimenez Rezende.  NeRF-VAE: A Geometry Aware 3D Scene Generative Model.  In _International Conference on Machine Learning_ , 2021 arXiv/2104.00587.
  * Lee et al. (2019) Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh.  Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks.  In _International Conference on Machine Learning_ , 2019.
  * Lin et al. (2021) Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  BARF: Bundle-Adjusting Neural Radiance Fields.  In _International Conference on Computer Vision_ , 2021.
  * Martel et al. (2021) Julien N. P. Martel, David B. Lindell, Connor Z. Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein.  ACORN: Adaptive Coordinate Networks for Neural Scene Representation.  _ACM Transactions on Graphics_ , 40:58:1–58:13, 2021.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _European Conference on Computer Vision_ , 2020.
  * Moreau et al. (2022) Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle.  LENS: Localization enhanced by NeRF synthesis.  In _Conference on Robot Learning_ , 2022.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  _ACM Transactions on Graphics_ , 41(4):102:1–102:15, July 2022.
  * Niemeyer et al. (2020) Michael Niemeyer, Lars M. Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , 2020.
  * Papamakarios et al. (2021) George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _Journal of Machine Learning Research_ , 22(57):1–64, 2021.
  * Pascanu et al. (2013) Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In _International Conference on Machine Learning_ , 2013 28.
  * Rezende & Mohamed (2015) Danilo Jimenez Rezende and Shakir Mohamed.  Variational Inference with Normalizing Flows.  In _International Conference on Machine Learning_ , 2015.
  * Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic Backpropagation and Approximate Inference in Deep Generative Models.  In _International Conference on Machine Learning_ , 2014.
  * Rosenbaum et al. (2018) Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J. Rezende, and S. M. Ali Eslami.  Learning models for visual 3D localization with implicit mapping.  _CoRR_ , arXiv/1807.03149, 2018.
  * Sajjadi et al. (2022a) Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Paveti’c, Mario Luvci’c, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf.  Object Scene Representation Transformer.  In _Advances in Neural Information Processing_ , 2022a arXiv/2206.06922.
  * Sajjadi et al. (2022b) Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs M. Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas A. Funkhouser, and Andrea Tagliasacchi.  Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations.  In _Conference on Computer Vision and Pattern Recognition_ , 2022b arXiv/2111.13152.
  * Sitzmann et al. (2019) Vincent Sitzmann, M. Zollhöfer, and G. Wetzstein.  Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations.  In _Advances in Neural Information Processing Systems_ , 2019.  arXiv/1906.01618.
  * Stelzner et al. (2021) Karl Stelzner, Kristian Kersting, and Adam R. Kosiorek.  Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation.  In _arXiv_ , 2021 arXiv/2104.01148.
  * Tancik et al. (2022) Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar.  Block-NeRF: Scalable Large Scene Neural View Synthesis.  _ArXiv_ , arXiv/2202.05263, 2022.
  * Trevithick & Yang (2021) Alex Trevithick and B. Yang.  GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering.  In _International Conference for Learning Representations_ , 2021\.  arXiv/2010.04595.
  * Turki et al. (2021) Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan.  Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs.  _CoRR_ , arXiv/2112.10703, 2021.
  * Vahdat & Kautz (2020) Arash Vahdat and Jan Kautz.  NVAE: A Deep Hierarchical Variational Autoencoder.  _arXiv/2007.03898_ , 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in Neural Information Processing Systems_ , 2017.  30\.
  * Wang et al. (2021) Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas A. Funkhouser.  IBRNet: Learning Multi-View Image-Based Rendering.  In _Conference on Computer Vision and Pattern Recognition_ , 2021\.
  * Wang et al. (2022) Xiaofeng Wang, Zheng Zhu, Fangbo Qin, Yun Ye, Guan Huang, Xu Chi, Yijia He, and Xingang Wang.  MVSTER: Epipolar Transformer for Efficient Multi-View Stereo.  _arXiv preprint arXiv:2204.07346_ , 2022.
  * Wang et al. (2004) Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * Wirnsberger et al. (2020) Peter Wirnsberger, Andrew J. Ballard, George Papamakarios, Stuart Abercrombie, Sébastien Racanière, Alexander Pritzel, Danilo Jimenez Rezende, and Charles Blundell.  Targeted free energy estimation via learned mappings.  _The Journal of Chemical Physics_ , 2020.
  * Wirnsberger et al. (2021) Peter Wirnsberger, George Papamakarios, Borja Ibarz, Sébastien Racanière, Andrew J Ballard, Alexander Pritzel, and Charles Blundell.  Normalizing flows for atomic solids.  _arXiv preprint arXiv:2111.08696_ , 2021.
  * Xi et al. (2022) Junhua Xi, Yifei Shi, Yijie Wang, Yulan Guo, and Kai Xu.  RayMVSNet: Learning Ray-based 1D Implicit Fields for Accurate Multi-View Stereo.  In _Conference on Computer Vision and Pattern Recognition_ , 2022\.
  * Xiangli et al. (2021) Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin.  CityNeRF: Building NeRF at City Scale.  _ArXiv_ , arXiv/2112.05504, 2021.
  * Yu et al. (2021) Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelNeRF: Neural Radiance Fields From One or Few Images.  In _Conference on Computer Vision and Pattern Recognition_ , 2021\.

##  Appendix A Model architectures

###  A.1 Normalizing flows for Laser

We wish to model our latent set representation
$\mathcal{Z}=\\{\bm{z}_{1},\ldots,\bm{z}_{K}\\}$ using a flexible distribution
that is efficient both to evaluate as well as sample from. Furthermore, the
distribution should ideally be invariant to re-ordering of the elements of the
set. Normalizing flows (Rezende & Mohamed, 2015) allow specifying a
distribution with precisely these properties. Formally, it is given by:

$\mathcal{Z}^{{(}0{)}}\sim p_{0}\left(\mathcal{Z}\right)\,,\
\mathcal{Z}^{{(}m{)}}=f_{m}\left(\mathcal{Z}^{{(}m-1{)}},\mathcal{V}\right)\,,\
m=1,\ldots,M\,,\ \text{with}\ \mathcal{Z}\coloneqq\mathcal{Z}^{{(}M{)}}\,,$
(4)

where $p_{0}\left(\cdot\right)$ is a base distribution, $f_{m}(\cdot)$ are
invertible transformations applied to an intermediate random variable, and
$\mathcal{Z}$ is the final laser after $M$ flow layers. $\mathcal{V}$ is an
additional set of context vectors that condition the flow transformation. The
resulting probability density of $\mathcal{Z}$ is

$p\left(\mathcal{Z}\mathrel{}\middle|\mathrel{}\mathcal{V}\right)=p_{0}\left(\mathcal{Z}^{{(}0{)}}\right)\prod_{m=1}^{M}\left|\det\frac{\delta
f_{m}\left(\mathcal{Z}^{{(}m-1{)}},\mathcal{V}\right)}{\delta\mathcal{Z}^{{(}m-1{)}}}\right|^{-1}\,.$
(5)

The need to compute the Jacobian of intermediate flow transformations puts
constraints on the design of those transformations, as they need to be
computationally efficient. We use a split-coupling flow similar to RealNVP
(Dinh et al., 2017), and we follow a similar architecture as the ones used in
Wirnsberger et al. (2020; 2021) in adapting it to sets. The method Flow Scans
(Bender et al., 2020) first introduced the general idea of adapting split-
coupling to work on sets, and Wirnsberger et al. (2020; 2021) use a
transformer for increased flexibility. A split-coupling transformation
partitions the set $\mathcal{Z}$ into two sets
$\mathcal{Z}_{1},\mathcal{Z}_{2}$ by splitting each element of the original
set across channels (we split in half). At a high level, the transformation of
one part is computed as a function of the other part. Finally, the original
set is recovered by elementwise concatenation of both transformed parts. More
formally, let $g$ be an invertible function, a split-coupling transform
$f^{\text{sc}}_{m}$ that maps
$\mathcal{Z}^{{(}m-1{)}}\rightarrow\mathcal{Z}^{{(}m{)}}$ is given by three
operations:

$\displaystyle\mathcal{Z}_{1}^{{(}m{)}}$
$\displaystyle=g\left(\mathcal{Z}_{1}^{{(}m-1{)}},t_{m}\left(\mathcal{Z}_{2}^{{(}m-1{)}},\mathcal{V}\right)\right)\,,$
(6) $\displaystyle\mathcal{Z}_{2}^{{(}m{)}}$
$\displaystyle=g\left(\mathcal{Z}_{2}^{{(}m-1{)}},t_{m}\left(\mathcal{Z}_{1}^{{(}m{)}},\mathcal{V}\right)\right)\,,$
$\displaystyle\mathcal{Z}^{{(}m{)}}$
$\displaystyle=\\{\text{concat}(\bm{z}_{k,1}^{{(}m{)}},\bm{z}_{k,2}^{{(}m{)}})\\}^{K}_{k=1}\,,$

where $\bm{z}_{k,1}$ indicates dimensions of split $1$ of element $k$, and
$\mathcal{V}$ is conditioning context. Note how the parameters of the
transformation $g$ of one partition are derived from the other partition. To
do so in a permutation-equivariant manner, the functions $t^{m}$ are a
composition of two transformers: a self-attention module is applied followed
by a cross-attention conditioned on context features, defined in Section A.2.
More specifically, when transforming split 1 as a function of split 2, we
have:

$\displaystyle\bm{\psi}^{{(}m-1{)}}_{2}=\text{CrossAttn}\left(\text{SelfAttn}\left(\mathcal{Z}^{{(}m-1{)}}_{2}\right),\mathcal{V}\right)\,.$
(7)

We finally apply the function $g$, in our case the affine function (Dinh et
al., 2017), from the parameters obtained for every element in
$\bm{\psi}_{2}^{m-1}$:

$\displaystyle\bm{z}_{k,1}^{{(}m{)}}$
$\displaystyle=\bm{z}_{k,1}^{{(}m-1{)}}\exp(\bm{W}^{m}_{scale}\bm{\psi}^{{(}m-1{)}}_{k,2})+\bm{W}^{m}_{scale}\bm{\psi}^{{(}m-1{)}}_{k,2}\
\text{for }k=1,\ldots K\,,$ (8)

where $\bm{W}^{m}_{scale}$ and $\bm{W}^{m}_{bias}$ are linear weights. When
composing our complete flow with $M$ layers of split-coupling flows, we find
that including an additional linear invertible transformation after each
$f_{m}$ to be helpful. To do so, we use the _Linear Permutation Equivariant_
($f^{\text{lpe}}_{m}$), proposed in the Flow Scans work, the purpose of which
is to allow capturing interdependencies in the dimensions of the set elements.
The only difference is that we parameterize each sub-block of the linear
transformation using a full unconstrained matrix (instead of a diagonal
matrix).

In summary, our flow chains together the sequence of mappings
$f^{\text{sc}}_{1},f^{\text{lpe}}_{1},\ldots,f^{\text{sc}}_{M},f^{\text{lpe}}_{M}$.
Thus, sampling from this flow involves sampling from the base distribution
followed by applying the chain of functions.

Our normalizing flow differs in a number of ways compared with Wirnsberger et
al. (2020; 2021). First, our flow layers are additionally parameterized by a
cross-attention module in order to condition the distribution on the context
set; second, we don’t use rational-quadratic splines, instead we found the
simpler affine coupling layer to work well. Notice that the size of the
resulting set depends only on the hyper-parameter $K$, that is the number of
elements sampled from the base distribution. It does not depend on the context
size, i.e. features extracted from input images.

In our experiments, we use a diagonal Gaussian base distribution, and set the
number of flow layers to be $M=8$ for the prior flow, and $M=4$ for the
posterior flows. Our self-attention and cross-attention are composed of $L=1$
layers, and 256 hidden units. Furthermore, we invert the chain of mappings
(which is possible due to invertibility) for the prior flow as it we found it
to be more numerically stable during training.

###  A.2 Attention modules

Figure 5:  Multi-head self-attention and cross-attention blocks. When
combining multiple blocks, the outputs $\bm{y}_{l}$ for layer $l$ become the
inputs for layer $l+1$.

We use transformer attention Vaswani et al. (2017) in various components of
Laser-NV. We distinguish two types of attention modules: self-attention, where
keys, queries and values are computed from the same input, and cross-
attention, where some context is used to compute keys and values, and queries
are computed from a different input). An attention module can be composed of
multiple attention blocks, where each attention block is shown in Fig. 5. We
use the standard multi-head dot-product attention with softmax weights. We
define $\text{SelfAttn}(\bm{y},L)$ to be a self-attention module with input
$\bm{y}$ and $L$ blocks555We often drop the $L$ input argument for notational
clarity. and, similarly, $\text{CrossAttn}(\bm{y},\bm{c},L)$ to be a cross-
attention module with context $\bm{c}$. Note how SelfAttn and CrossAttn do not
include positional encoding that are often used in transformers.

###  A.3 Image encoder

To encode the context images, we use a convolutional neural network based on
the ResNet architecture from Vahdat & Kautz (2020) (also described in Kosiorek
et al. (2021)) to independently encode context elements into $h\times w$
feature maps. The residual blocks that compose the encoder are shown in Fig.
6. For all models and datasets we use $L=4$. Each context view consists of an
image in RGB space with concatenated information of the camera position and
direction.

Figure 6:  ResNet architecture based on the encoder of NouveauVAE Vahdat &
Kautz (2020).

For $N$ context views, our last layer outputs $Nhw$ vectors that form the
conditioning set used to parameterize our distributions over $\mathcal{Z}$.
Furthermore, the feature maps at different layers are also combined into a
feature stack to use as the context local features as described in Section
A.4.

###  A.4 Scene function architectures

Fig. 7 provides a high-level overview of the information flows going into the
scene function of each model, and we describe implementation details of their
components below.

(a) Laser-NV scene function

(b) Laser-NV (no geom.) scene function

(c) mvc-nerf scene function

(d) nerf-vae scene function

Figure 7:  Scene function architecture for Laser-NV (a), its ablations (b) and
(c), and nerf-vae (d).

#### Latent attention

A query point $\bm{x}_{i}$ attends to $\mathcal{Z}$ as per Fig. 7 (a) using a
$\text{cross-attn}\left(\gamma(\bm{x}_{i}),\mathcal{Z},L\right)$, where
$\gamma(\bm{x}_{i})$ is the circular encoding function of positions (see
Section A.4). We use the same architecture across datasets, setting $L=3$
layers, 512 hidden units, and 4 attention heads.

#### Local features from multi-view geometry

We follow PixelNERF’s design of stacking the features of the encoded images at
different levels of resolution, which allows the conditioning features to
capture context structure at different scales. For input images of resolution
$W\times H$, we stack the features of the outputs from all layers after the
first layer:

$\mathbf{H}^{n}=\text{stack}(\\{\mathbf{H}^{n}_{l}\in\mathbb{R}^{\frac{W}{i}\times\frac{H}{i}\times
16i}\\}^{L}_{l=1}),$ (9)

where $i=2^{l}$. When stacking, we upscale the feature maps of all the feature
maps of layers $l\in\\{2\ldots L\\}$ to the size of the feature map with
largest resolution ($l=1)$ using bilinear interpolation. The resulting set of
stacked feature maps (one for each context view) is then used to condition the
scene function.

As we described in Section 3.2, our multi-view geometry module is in part
based on pixelNeRF’s processing of context views. Particularly, we process
local features with exactly the $f_{1}$ ResNet detailed in Yu et al. (2021):
positions and directions are projected to the view-space using the cameras
intrinsic parameters, and then processed with 3 residual blocks of 512 hidden
units and ReLu activation. When computing the projection, we also define the
binary variable $v^{n}_{i}$ based on whether the projected coordinates are
within the image plane.

The output features are integrated with our latent features via $M_{\theta}$
which is a cross-attention module. Note that context features with
$v^{n}_{i}=0$ are not attended to (we modify the pre-softmax logits of those
context features by adding a large negative number to the logits). In our
experiments $M_{\theta}$ has 3 layers and 512 hidden units.

#### Scene function output MLP

We now describe the final component of the scene functions of all models, as
shown in Fig. 7. This component follows the residual MLP architecture of the
scene function of nerf-vae. Namely, it takes as input the circular encoded
positions $\gamma(\bm{x}_{i})$ and directions $\gamma(\bm{d}_{i})$, and at
every layer it linearly projects the conditioning features (i.e. $\bm{z}$ for
nerf-vae, or $\bm{f}_{i}$ for Laser-NV) and sums it to the activations. The
MLP is composed of two parts: the first has 4 layer of 256 hidden units each
and outputs the densities, and the second part has an additional 4 layers and
outputs the colours. All layers use a swish activation function.

The circular encoding function $\gamma$ is given by

$\gamma(p)=(\sin(2^{L_{\text{min}}}\pi p),\cos(2^{L_{\text{min}}}\pi
p),\ldots,\sin(2^{L_{\text{max}}}\pi p)\,,\cos(2^{L_{\text{max}}}))$ (10)

where we use different constants $L_{\text{min}}$ and $L_{\text{max}}$ for
positions and directions and for each dataset (listed in Appendix E).

###  A.5 nerf-vae and conditional nerf-vae

Our implementation nerf-vae follows the same overall architecture as that of
Kosiorek et al. (2021), but we use the image encoder described in Fig. 6
instead. The approximate posterior is computed as a diagonal Gaussian
distribution, the means and log-variances of which are given by an MLP $e$ as
a function of the pooled feature maps $\mathcal{H}$ (pooled with a simple
averaging over feature maps). The MLP $e$ has two layers of 256 hidden units
and ReLu activation function. The scene function is composed of an MLP that
processes the latent vector $\bm{z}$ (one linear layer of 256 units, followed
by a ReLu non-linearity), followed by the output MLP described in Section A.4.

###  A.6 Background scene function

The volumetric rendering integral of a ray $\bm{r}$ is given by

$\bm{c}(\bm{r})_{\lx@glossaries@gls@link{acronym}{NERF}{\leavevmode\textsc{n{e}rf}}}=\int_{t_{n}}^{t_{f}}T(t)\sigma(\bm{r}(t))\bm{c}(\bm{r}(t),\bm{d})\,\mathrm{d}t\,,\
\text{with}\
T(t)=\exp\left(-\int_{t_{n}}^{t_{f}}\sigma(\bm{r}(s))\,\mathrm{d}s\right)\,.$
(11)

If the ray is infinite (camera’s far plane $t_{f}\to\infty$) the colour
weights $T(t)\sigma(\bm{r}(t))$ would always sum to one. For a truncated ray
(finite $t_{f}$), this happens only if the ray passes through a solid surface.
This heuristic allows to model distant backgrounds by putting them at the end
of the integration range, but it doesn’t work if the camera position changes
significantly compared to the ray range $t_{f}-t_{n}$. If not normalized, the
final weight $T(t_{f})$ tell us about the probability of light hitting a
particle beyond the ray.

In that case, we can model the background as an infinite dome around the scene
and model it with a light-field that depends only on the camera direction.
That is, we compute the colour as

$\bm{c}(\bm{r})=\bm{c}(\bm{r})_{\lx@glossaries@gls@link{acronym}{NERF}{\leavevmode\textsc{n{e}rf}}}+T(t_{f})f_{bg}(\bm{d})\,.$
(12)

For the City dataset, in order to model the sky and sun,
$f_{bg}(\mathcal{Z},\bm{d_{i}})$ is a learned neural network. We use a cross-
attention module with 2 layers to attend to $\mathcal{Z}$. For nerf-vae and
cond. nerf-vae we instead use a 2 layered MLP with ReLu activations. For the
ShapeNet $f_{bg}$ is a constant colour (white).

We address the challenging backgrounds of MSN-Hard with a more expressive
attentive background scene function that conditions on both the latents
$\mathcal{Z}$ as well as context image features – this is in a similar fashion
to how the NeRF scene function of Laser-NV integrates its latents with
attention to local context features. Given an encoded ray direction
$\gamma(\bm{d}_{i})$ and context feature maps $\bigcup_{n=1}^{N}{\bm{H}^{n}}$,
a cross-attention module attends to the latents:
$\bm{\hat{h}}^{bg}_{i}=f^{1}_{bg}(\gamma({\bm{d}_{i}}),\mathcal{Z})$, and the
output is used to attend to the set of context features
$\bigcup_{n=1}^{N}{\bm{H}^{n}}$ with a second cross-attention module
$\bm{h}^{bg}_{i}=f^{2}_{bg}(\bm{\hat{h}}^{bg}_{i},\bigcup_{n=1}^{N}{\bm{H}^{n}})$.
Finally, an MLP $f^{mlp}_{bg}$ computes the background colour by taking as
input the obtained background features and additionally conditions on the
encoded directions by adding a linear projection of the directions to the
activations of every layer. This MLP is composed of 2 layers of 512 hidden
units each with a ReLu nonlinearity, and outputs the RGB background colours
using a sigmoid nonlinearity.

##  Appendix B City dataset

Figure 8:  Examples of generated cities.  Figure 9:  Examples of different
viewpoints (column) for each scene (row).

The aim of the City dataset is to allow generating visually complex scenes
with large variability in its structure and appearance. Each generated city
consists of a large area spanning approximately $270m\times 270m$, which
consists of a 2x2 grid of blocks, where each block is composed of 4 lots. A
lot can be a building or a green patch (park). The city 2x2 grid is surrounded
by an outer ring of buildings. Roads separate all blocks in the scene. Each
building is randomly generated with variation in size, appearance (textures,
materials, colours), architectural styles (residential, industrial,
commercial). Fig. 8 shows examples of generated city scenes. Once a city is
built, we specify a range of points of interest, i.e. road intersections and
parks and use each of these points as the starting placement for a _scene_
(data point) of our dataset. Each scene is comprised of posed images rendered
from randomly chosen viewpoints near the point of interest. More specifically,
we render 8 randomly chosen nearby points, and 4 random viewpoints for each
point, making a total of 32 generated views per scene. Each generated view is
annotated with the ground-truth render, camera parameters and depth maps. Our
generated training dataset contains $100{,}000$ scenes generated in total from
approximately $2{,}600$ uniquely generated cities. We test on $3{,}200$ scenes
generated in total from $10$ uniquely generated cities.

##  Appendix C Additional visualizations

#### Capturing detail at different scale

In Fig. 10 we can see at a higher resolution how our model is expressive
enough to capture texture and geometry details at different scales (e.g. far-
away buildings and nearby objects such as the lamp post and the colonnade).

Figure 10:  Higher resolution reconstructions of Laser-NV reveal detail
captured at different scales (e.g. lamp posts, and nearby buildings vs
buildings at a large distance).

#### Novel view synthesis with ShapeNet NMR

In Fig. 11 we compare predictions for a number of objects that reveal how
nerf-vae produces blurry reconstructions compared to mvc-nerf and Laser-NV.
When an image does not reveal the full shape of an object, as shown in Fig.
12, Laser-NV is able to generate multiple plausible variations (in contrast to
a deterministic which generates a single blurry prediction).

Figure 11:  Samples of category-agnostic ShapeNet for ablations of Laser-NV.
mvc-nerf and Laser-NV both result in high-fidelity predictions, while the
nerf-vae’s outputs are blurrier.  Figure 12:  When predicting novel view from
an ambiguous input, Laser-NV generates plausible variations, whereas mvc-nerf
results in a single blurry prediction.

#### MSN-Hard predictions

Figure 13 shows Laser-NV’s predictions on MSN-Hard including depth maps
estimated from nerf’s densities.

Figure 13:  Representative examples of Laser-NV predictions on MSN-Hard.

##  Appendix D Additional results

#### Data efficiency

We evaluate how data hungry Laser-NV is compared to nerf-vae and mvc-nerf. In
Figure 14 we report the reconstruction colour log-probability estimated on a
validation set of the City dataset. For Laser-NV, we observe little
degradation between using 50K scenes and 12.5K unique scenes. For nerf-vae we
notice a larger drop when reducing the size from 25K to 12.5 scenes, showing
clearer signs of overfitting. Finally, since mvc-nerf severely underfits the
City dataset for all training set sizes, we do not see any clear performance
drop.

Figure 14: Validation reconstruction performance of different method as a
function of number of scenes throughout training.

#### Test-Time Scaling of the Latent Set

We further want to understand how different latent set sizes at _test time_
impact performance. Figure 15 shows reconstruction performance for different
values of $K$ at test time, for various values of used $K$ during training. As
we can see, Laser-NV does not significantly leverage a larger latent set size
than trained with. As expected, we do see performance degrading when using
_smaller_ sets at test time, and this effect becomes stronger for models
trained with larger capacity.

Figure 15: Reconstruction performance of Laser-NV as a function of the latent
set size used at test time for different models trained with increasing sizes
$K$.

#### Training Laser-NV without ground-truth depth supervision

We train all our models using ground-truth depth on the City dataset. This
reduces memory and computation requirements during training. We note, however,
that it is also possible to train Laser-NV without using ground truth depths.
Once trained, we find that both the model trained with depths and the model
trained without achieves slightly higher average colour log-likelihood (3.87)
compared to Laser-NV (of 3.83) (importance-weighted estimate using 10
samples). We hypothesize that this is because the model trained without depths
is trained solely using the rendering method used at test time.

As any NeRF-related method, Laser-NV can output estimated depth maps easily.
We show example reconstructions and _inferred_ depth maps of a trained Laser-
NV on the City in Fig. 16. Note how the model is capable of capturing fine
detail in textures and shapes (e.g. lamp post).

Figure 16:  Reconstructions of Laser-NV trained without the use of ground-
truth depths.

#### Computational efficiency of Laser-NV vs nerf-vae

For a model conditioned on 10 input views, inference (inferring the latent
distribution) and rendering of a single image for Laser-NV (with 24 slots)
take 0.39 GFLOPs and 1.80 GFLOPs respectively. nerf-vae instead takes 0.37
GFLOPs and 0.15 GFLOPs. The main reason for more expensive rendering in Laser-
NV is the introduction of PixelNeRF-like conditioning. A model with a flow
posterior but without such conditioning uses 0.47 GFLOPs to render a single
image. Attention, however expensive, is evaluated only once per scene (to
compute the prior/posterior). The PixelNeRF features are cheap per point, but
they are evaluated for every 3D point used in rendering, therefore dominating
the overall cost. Note that the above numbers do not reflect training latency.
At training time we can guide the rendering process by using depths, which
allows us to evaluate the scene function only twice for every pixel. We also
subsample images and we never reconstruct the whole target view(s). This makes
training faster, and means that Laser-NV is only about 4x slower per training
iteration than nerf-vae. Having said that, we noticed that nerf-vae does not
benefit from longer training (its performance saturates), while the
performance increase from increasing the model size (to equalize the FLOPs) is
negligible.

#### Robustness to camera noise

We carry out an experiment on MSN-Hard in which we train and test with camera
poses with incremental levels of Gaussian noise. We replicate the set up in
Sajjadi et al. (2022b) and compare their reported results using SRT and
PixelNeRF to Laser-NV. Results in Fig. 17 show that while Laser-NV outperforms
the other models when there is no noise, its performance drops more quickly
compared to SRT, in a similar fashion to PixelNeRF. However, Laser-NV is more
robust than PixelNeRF for all levels of noise. We believe that the use of NeRF
and local features makes these models less robust to noisy cameras. That said,
there are recent methods that allow mitigating noise sensitivity with NeRF
(Lin et al., 2021) that can be incorporated.

Figure 17:  PSNR of models trained and evaluated under cameras with different
levels of noise. The PSNR values of Laser-NV * are preliminary training values
models that have not trained to convergence (340K steps out of 1000K).

##  Appendix E Training details

#### Hierarchical rendering

Instead of optimizing separate sets of parameters for the coarse and fine
rendering passes, we find that it suffices to only keep separate parameters of
$F_{\theta}$. This allows sharing parameters (and reduce computation) when
querying the scene function on the coarse sampled points with respect to
latent attention and multi-view geometry components of models.

#### Training with depth (City)

Because we have RGB-D data in the City dataset, we can use depth as a
supervised signal following the method in Stelzner et al. (2021). This method
involves replacing the pixel colour likelihood, for a given ray $\bm{r}$, with
the ray colour and depth likelihood $p\left(\mathbf{t}(\bm{r}),\bf
c(\bm{r})\mathrel{}\middle|\mathrel{}\mathcal{V},\mathcal{Z}\right)$, where
$\bf t(\bm{r})$ is the ray’s ground-truth depth. This alternative likelihood
requires only two scene function evaluations per ray. In practice, we find
that we get the best quality renders when combining both log-likelihood terms
to the loss (each weighted by 0.5). At every training set, similar to how
nerf-vae is trained, the likelihood terms are estimated by only reconstructing
a subset of rays (i.e. pixels) of all the target images. We find that using 64
rays for estimating the (expensive) image log-likelihood, and 512 to estimate
the ray colour and depth likelihood term, works well in the City.
Additionally, we restrict output densities to be between 0 and 10 by using a
weighted sigmoid in the output of the scene function.

#### Additional regularization

The loss function for ShapeNet experiments includes a density regularization
term based on the L1-norm of the densities, with a weight of $0.01$. For all
experiments, we scale the model parameter updates (Pascanu et al., 2013, Sec.
3.2), such that loss gradient norms have a maximum value of 10 for improved
stability.

#### Hyper-parameters

We use the following hyper-parameters across all models. Some of which were
selected for each dataset based on a preliminary exploration using the
training dataset and a validation split.

Hyperparameter | City | Shapenet NMR | MSN-Hard
---|---|---|---
Learning rate | 0.0003 | 0.0002 | 0.0001
Batch size | 96 | 96 | 32
# of rays per scene | 512 | 512 | 384
# of context views | 2 | 1 | 5
# of target views | 8 | 23 | 3
# of posterior context views | 10 | 4 | 8
# of coarse points | 256 | 32 | 64
# of fine points | 64 | 64 | 32
Camera $t_{near}$ | 0.05 | 0.1 | 0.01
Camera $t_{far}$ | 270 | 3.7 | 19
Positions circular encoding $L_{\text{min}}$ | -8 | -2 | -5
Positions circular encoding $L_{\text{max}}$ | 8 | 8 | 10
Directions circular encoding $L_{\text{min}}$ | 0 | 0 | 0
Directions circular encoding $L_{\text{max}}$ | 4 | 4 | 8
Laser-NV set size K | 24 | 8 | 32
Laser-NV latent vector dimensionality | 128 | 128 | 128
Table 4: Common hyper-parameters