small $S$-path $Q$ in terms of these new vectors. By the above discussion, we
conclude that this problem is also decidable.

    22. fnum@enumiiitem 1.11.1

Appendix

    23. fnum@enumiiitem 1.21.2

Decision Problems

A decision problem is a problem that can be stated as a “True”/”False”
question for some set of input values. A deterministic algorithm is an
algorithm which performs a finite amount of steps that only depend on its
input. If there is a deterministic algorithm that solves a decision problem,
then this problem is said to be solvable or decidable. Given two decision
problems $P_{1}$ and $P_{2}$, if a deterministic algorithm that solves $P_{1}$
can also be used as a subroutine to deterministically solve $P_{2}$, then we
say that $P_{2}$ is reducible to $P_{1}$.

In many cases, the question of decidability of a problem is reducible to the
problem of calculating a value (or a function) which depends on the input. For
example, this value can be a function representing the amount of steps needed
by an optimal Turing Machine to write the correct output on a tape. If a well
defined function can be calculated by a deterministic algorithm, it is said to
be computable. If no such algorithm exists, the function is said to be
uncomputable.

A very fundamental decision problem in computation theory which is known to be
undecidable was introduced by Post [PCP]. It is known as the “Post
Correspondence Problem”, commonly abbreviated as PCP.

######  Problem 1.2.1.

(Post Correspondence Problem)

Let $\Sigma^{*}$ be the set of finite strings over a finite alphabet $\Sigma$.
For $s_{1},s_{2}\in\Sigma^{*}$ we denote the concatenation of $s_{1}$ and
$s_{2}$ by $s_{1}s_{2}$. A Post correspondence system (PCS) is a finite set

$P=\left\\{(a_{1},b_{1}),...,(a_{n},b_{n})\right\\}$

of pairs of elements in $S$. A solution of $P$ consists of an integer $1\leq
m$ and a sequence $i_{1},...,i_{m}$ such that

$a_{i_{1}}a_{i_{2}}...a_{i_{m}}=b_{i_{1}}b_{i_{2}}...b_{i_{m}}$

The Post Correspondence Problem for $P$ is to deterministically determine
whether a given set $P$ has a solution.

The classical proof of the undecidability of PCP is by reduction from the
“Halting Problem” (see [MS] section $5.2$). It was also shown that the problem
is undecidable even when the size of $P$ is bounded [P5],[P7].

######  Fact 1.2.2.

PCP is undecidable.

If we define the value $f(P)$ as the maximal value which needs to be
“considered” when searching for a solution for $P$, then this fact is
equivalent to saying that the function $f$ is not computable. For otherwise a
deterministic algorithm that computes $f(P)$ (in a finite amount of steps) and
then tries all possible sequences $i_{1},...,i_{m}$ where $m\leq f(P)$ solves
PCP, in contradiction to the problem being undecidable.

    24. fnum@enumiiitem 1.31.3

Vectors and Norms

Suppose $v,w\in\mathbb{R}^{n}$ are two vectors, whose coordinate
representation is $v=(v_{1},...,v_{n})$ and $w=(w_{1},...,w_{n})$ accordingly.
The $\ell_{1}$ norm of $v$ is defined by $||v||_{1}=\sum_{i=1}^{n}|v_{i}|$,
and the distance between $v,w$ is defined by
$\text{dist}(v,w)\coloneqq||v-w||_{1}$. For a set $W\subseteq\mathbb{R}^{n}$,
the distance between $v$ and $W$ is defined by

$\text{dist}(v,W)\coloneqq||v-W||_{1}\coloneqq\inf_{w\in W}||v-w||_{1}$

If $W$ is finite then $\text{dist}(v,W)=\min_{w\in W}||v-w||_{1}$. Otherwise a
minimum does not necessarily exist.

    25. fnum@enumiiitem 1.41.4

Technical Lemmas

######  Lemma 1.4.1.

Let $t,q>0$ and $\epsilon\in(0,1)$. Let $\epsilon_{1}$ be

$\epsilon_{1}=\frac{\epsilon}{4(2t+2)^{2}\left(1+2(2t+1)^{q}\right)}$

Then

$(2t+2)\cdot\left(\epsilon_{1}+\left(1+2(2t+1)^{q}\right)\left(\frac{1}{1-(2t+2)\epsilon_{1}}-1\right)\right)\leq\epsilon$

Proof

By the definition of $\epsilon_{1}$, we have

$\epsilon_{1}\leq\min\\{\frac{1}{4t+4},\frac{\epsilon}{4t+4}\\}$

And also

$2(2t+2)^{2}\cdot\left(1+2(2t+1)^{q}\right)\epsilon_{1}=\frac{\epsilon}{2}$

Therefore

$\displaystyle(2t+2)\cdot\left(\epsilon_{1}+\left(1+2(2t+1)^{q}\right)\left(\frac{1}{1-(2t+2)\epsilon_{1}}-1\right)\right)=$
$\displaystyle=(2t+2)\cdot\left(\epsilon_{1}+\left(1+2(2t+1)^{q}\right)\left(\frac{(2t+2)\epsilon_{1}}{1-(2t+2)\epsilon_{1}}\right)\right)\leq$
$\displaystyle\leq(2t+2)\cdot\left(\frac{\epsilon}{4t+4}+\left(1+2(2t+1)^{q}\right)\left(\frac{(2t+2)\epsilon_{1}}{1-(2t+2)\frac{1}{4t+4}}\right)\right)\leq$
$\displaystyle\leq(2t+2)\cdot\left(\frac{\epsilon}{4t+4}+\left(1+2(2t+1)^{q}\right)\left(2(2t+2)\epsilon_{1}\right)\right)$
$\displaystyle=\frac{\epsilon}{2}+2(2t+2)^{2}\cdot\left(1+2(2t+1)^{q}\right)\epsilon_{1}=$
$\displaystyle=\frac{\epsilon}{2}+\frac{\epsilon}{2}=\epsilon$

Which is what we had to prove.∎

References

       * [AL]

L. Lovasz. Large Networks and Graph Limits. American Mathematical Society,
2012.

       * [BER]

P. Indyk, A. McGregor, I. Newman and K. Onak. Bertinoro workshop on sublinear
algorithms 2011. http://sublinear.info/42 In Open Problems in Data Streams,
Property Testing, and Related Topics, 2011.

       * [BSS]

I. Benjamini, O. Schramm, and A. Shapira. Every Minor-Close Property of Sparse
Graphs Is Testable. Advances in Mathematics, 223(6):2200-2218, 2010.

       * [BU]

V. K. Bulitko, On graphs with given environments of vertices. Proc. Steklov
Inst. Math. 133 (1973), 77-94

       * [FPS]

H. Fichtenberger, P. Peng and C. Sohler. On Constant-Size Graphs That Preserve
the Local Structure of High-Girth Graphs. In proceedings of the 19th
International Workshop on Randomization and Computation (RANDOM), 2015.

       * [GR]

O. Goldreich and D. Ron. Property Testing in Bounded Degree Graphs.
Algorithmica, 32:302-343, 2002.

       * [HKN]

A. Hassidim, J. Kelner, H. Nguyen, and K. Onak. Local Graph Partitions for
Approximation and Testing. In Proceedings of the 50th annual IEEE sumposium on
Foundations of Computer Science, pages 22-31. IEEE, 2009

       * [J]

D. P. Jacobs. Undecidability of Winkler’s r-Neighborhood Problem for Covering
Digraphs. J. Combin. Theory Ser.B, 60: 254-267, 1994

       * [MS]

M. Sipser Introduction to the Theory of Computation. Cengage Learning, 2013.

       * [PCP]

E. L. Post. A variant of a recursively unsolvale problem. Bull. Amer. Math.
soc. 52 (1946), 264-268

       * [P5]

T. Neary Undecidability in Binary Tag Systems and the Post Correspondece
Problem for Five Pair of Words. Ins. of Neuro., Uni. of Zurich, 1998

       * [P7]

Y. Matiyasevich. and G. Senizergues. Decision Problem For Semi-Thue Systems
With A Few Rules. Steklov Inst. Math. 1996

       * [REG]

E. Szemeredi. Regular Partitions of Graphs. Technical report, DTIC Document,
1975.

       * [W]

Peter M. Winkler. Existence of graphs with a given set of r-neighborhoods. J.
Combin. Theory Ser.B, 34(2): 165-176, 1983


# SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input
Images

Abdullah Hamdi 1,2,3 Bernard Ghanem 2 Matthias Nießner 1
1Technical University of Munich (TUM) 2King Abdullah University of Science and
Technology (KAUST)
3Visual Geometry Group, University of Oxford
<EMAIL_ADDRESS>
###### Abstract

Recent advances in Neural Radiance Fields (NeRFs) treat the problem of novel
view synthesis as Sparse Radiance Field (SRF) optimization using sparse voxels
for efficient and fast rendering [14, 46]. In order to leverage machine
learning and adoption of SRFs as a 3D representation, we present SPARF, a
large-scale ShapeNet-based synthetic dataset for novel view synthesis
consisting of $\sim$ 17 million images rendered from nearly 40,000 shapes at
high resolution (400 $\times$ 400 pixels). The dataset is orders of magnitude
larger than existing synthetic datasets for novel view synthesis and includes
more than one million 3D-optimized radiance fields with multiple voxel
resolutions. Furthermore, we propose a novel pipeline (SuRFNet) that learns to
generate sparse voxel radiance fields from only few views. This is done by
using the densely collected SPARF dataset and 3D sparse convolutions. SuRFNet
employs partial SRFs from few/one images and a specialized SRF loss to learn
to generate high-quality sparse voxel radiance fields that can be rendered
from novel views. Our approach achieves state-of-the-art results in the task
of unconstrained novel view synthesis based on few views on ShapeNet as
compared to recent baselines. The SPARF dataset is made public with the code
and models on the project website abdullahamdi.com/sparf.

##  1 Introduction

Although we observe the surrounding world only as a stream of 2D images, it is
undeniably 3D. The goal of recovering this underlying 3D from 2D observations
has been a longstanding goal of computer vision. The task of inverting the
rendering process that creates the 2D projections we observe by trying to
construct the 3D world is known as Vision as Inverse Graphics (VIG) [7, 30,
76, 28]. With the emergence of deep learning applications in computer graphics
and the availability of 3D datasets, several approaches address the 3D
generation task directly from 3D data, without relying on appearance [47, 53,
24, 1, 71, 31]. However, recent developments in differentiable rendering have
refueled the VIG direction, which facilitates using gradients of the rendering
process to optimize for the underlying 3D setup based on image observations
[54, 61, 16, 77, 21, 22, 17, 42, 44, 36, 40, 58, 35]. More specifically,
Neural Radiance Fields (NeRFs) [44, 51, 75, 6] show impressive performance on
novel view synthesis by optimizing volumetric radiance fields on a large
number of posed multi-view images.

Figure 1: Distribution of Radiance Fields. We treat Sparse Radiance Field
(SRF) as a 3D data structure and learn the conditional generation of SRFs from
few input images for the task of novel view synthesis. In order to do this, we
build SPARF, a large-scale dataset of SRFs.

| Posed Multi-View Datasets
---|---
Attribute |  SRN [59] | DTU[26] |  NMR [49] |  RTMV [60] |  SPARF (ours)
Number of Classes | 2 | N/A | 13 | N/A | 13
[HTML]EFEFEF Number of Scenes/Objects | 3,511 | 124 | 43,756 | 2,000 | 39,705
Image Resolution | 128 | 1,200 | 64 | 1,600 | 400
[HTML]EFEFEF Number of Radiance Fields | 0 | 0 | 0 | 2,000 | 1,072,008
Real/Synthetic | synthetic | real | synthetic | synthetic | synthetic
[HTML]EFEFEF View Setup | sphere | random | circle | hemisphere | sphere
Total Number of Images | 265,550 | 4,235 | 1,050,144 | 300,000 | 17,073,150
[HTML]EFEFEF Views per Model | 50 | N/A | 24 | N/A | 430
Dataset Size (GB) | 5.8 | 1 | 33 | 2,520 | 3,432

Table 1: Comparison of Different Posed Multi-View Datasets. We compare posed
multi-view datasets to our large-scale SPARF dataset.

Various subsequent works addressed NeRFs’ shortcomings, as rendering speed
[46, 14, 74], training size requirement [75, 6, 45], or pose requirements [51,
39]. The seminal work of Plenoxels [14] showed that the MLP network is not
necessary for quick optimization and volumetric rendering of the radiance
fields. However, these recent methods are still optimization-based, where a
single scene representation is optimized without any generalization to new
scenes/objects [46, 14]. In this work, we treat Sparse Radiance Fields (SRFs)
as a 3D data structure and try to learn a generative model (dubbed SuRFNet) on
the distribution of sparse-voxel radiance fields conditioned on a few images
to generalize to unseen 3D shapes (see Figure 1).

In order to train deep learning models on 3D data to generalize to unseen
examples, the dataset size should be in tens of thousands of samples [68, 5].
However, current posed multi-view datasets are not suitable for leveraging the
power of deep networks as can be seen in Table 1. The image resolution is
either too low (_e.g_. $64\times 64$ in [49]), the samples don’t fall into
distinct shape categories with similar numbers of views [26, 55, 27], or lack
diversity in the samples and classes [59]. For these reasons, we construct a
large and high-resolution dataset (SPARF) of posed multi-view images from
ShapeNet [5] that correspond to the same 13 classes originally used in the NMR
dataset [49], but with an order of magnitude more images and pixels (17M _vs_.
1M images and 400$\times$400 _vs_. 64 $\times$ 64 pixels). We also provide
more than one million optimized sparse radiance fields of spherical harmonics
and densities that allow for the novel view synthesis of the 40K models using
Plenoxels [14].

The idea of learning a prior (2D CNN/ViT) on radiance fields in order to
enhance the few-view setup of novel view synthesis is previously investigated
by several works [75, 56, 34]. However, we propose SuRFNet to directly learn
from the 3D sparse radiance fields, by optimizing partial SRFs from the few
images and training a generalizable network that converts these partial SRFs
to whole SRFs in a supervised fashion. Such a 3D setup benefits from grid-
based 3D learning, creating a 3D prior that ensures multi-view consistency,
especially when rendering from out-of-distribution views. Also, this 3D sparse
voxel setup benefits from the advancements in fast volume rendering [46, 14],
allowing for end-to-end deep learning pipelines that harness volume rendering.
To the best of our knowledge, our SURFNet is the first model that learns to
generate 3D radiance fields for unseen objects at test time with only a
few/single views by learning from the distribution of radiance fields in 3D.

Contributions: (i) To facilitate the application of deep learning on radiance
fields, we provide a new Posed Multi-view dataset (SPARF) that is an order of
magnitude larger than others (around 40K 3D models). The dataset includes a
total of one million optimized Sparse Radiance Fields (SRFs) with multiple
voxel resolutions, which allows for high-quality novel view synthesis and will
be made publicly available. (ii) We propose a novel architecture and a
pipeline (SuRFNet) equipped with a specialized SRF-loss to generate voxel-
based radiance fields from a few images based on learning to complete partial
radiance fields. SuRFNet improves the performance of unconstrained novel view
synthesis based on few views compared to state-of-the-art methods.

##  2 Related Work

Learning 3D Shapes. Several works aim to predict the geometry of 3D shapes
given several input images, by directly optimizing the vertices of a template
mesh through differentiable projections or through fitting a network [61, 16,
77, 17, 42, 15, 52]. Other works use MLPs as a deep prior to the optimized
mesh [23, 66]. Alternately, some methods try to learn the distribution of 3D
meshes by optimizing 3D generators independent of how the meshes look when
rendered, solely based on the available 3D data and heuristic regularizers
[47, 53, 9]. Point cloud methods offer an alternative to the mesh complex
topology by learning generative models on the point clouds themselves, _e.g_.
by using an Auto Encoder [1, 71] or a GAN framework [1, 31]. The implicit
representation paradigm offers an alternative to meshes for smooth and
detailed shape representation. These methods learn a continuous implicit
representation of shapes by learning the Signed Distance Functions or
occupancy of the object through MLPs [50, 41, 73, 48, 4, 3, 57, 36]. In this
work, the scope focuses on the quality of the rendering from novel views and
not on 3D reconstruction.

Neural Radiance Fields (NeRFs). NeRFs [44] proved to be a successful
popularizing in implicit volume representation and novel view synthesis. They
define an implicit field and learn an MLP that predicts the RGB and density
value of that 3D field given a set of posed images. NeRFs shoot rays on the
volume and integrate the predictions to obtain individual pixel values. This
formulation, however, has many drawbacks including large memory and compute
requirements, inability to model dynamic scenes, posed image requirements, and
the limitation to small 3D objects or rooms [39, 51, 74, 48, 75, 32]. To
address the speed limitation, PlenOctreeNeRF [74] stores the precomputed RGB,
density, and spherical harmonics in the 3D volume as an Octree data structure
for fast inference. Plenoxels [14] optimize the density and spherical
harmonics on sparse voxels with a TV loss and perform ray marching for
rendering from novel views. Similarly, INGP [46] uses multi-resolution voxel
hashing to perform a real-time rendering of radiance fields, demonstrating
that the redundancy of the MLP in NeRFs. We build on these observations and
build the SPARF dataset of sparse voxel radiance fields in order to facilitate
learning on these SRFs as 3D data structures instead of as just side outcomes
of a volumetric optimization.

Figure 2: SPARF: a Large Dataset for 3D Shapes Radiance Fields and Novel Views
Synthesis.

SPARF (ours) | SRN [59] | NMR [49]
---|---|---
|  |

Figure 3: SPARF _vs_. other Datasets. SPARF offers a large-scale high-
resolution dataset compared to other posed multi-view datasets. We show the
same chair here on SPARF, SRN, and NMR (please zoom in for differences). This
highlights the huge quality gap between SPARF and other ShapeNet-based
datasets.

Few-Image NeRFs. To address the original NeRF’s requirement of many posed
images, several methods were proposed. The seminal work PixelNerf [75] is the
first to reduce the image data requirements in order to learn a NeRF by using
a trained CNN prior that can allow for transferable representation between
scenes. Similarly, MVSNerf [6, 18], AutoRF [45], and ShaRF [56] learn a CNN
prior to generalize across scenes. IBRNet [62] learns to render novel views
based on neighboring views and optimized neural volume representation. More
recently, VisionNerf [34] proposes to use a ViT [12] to extract global
features from the input images to enhance the capability of the NeRF MLP to
predict the radiance field when one image is used as input. Unlike these
works, we propose SuRFNet to directly learn from the 3D sparse radiance
fields. Such a 3D setup benefits from structured 3D learning, creating a 3D
prior that guarantees multi-view consistency while benefiting from the speed
of recent voxel-based methods. A concurrent work by Guo _et al_. [18] learns a
3D prior based on a perceptual loss, but does not use 3D supervision and only
uses dense voxels ( limiting the pipeline to the low resolution of $64^{3}$ ).

Datasets for novel view synthesis. Several datasets were proposed to support
the task of novel view synthesis. NeRF [44] introduced 8 synthetic scenes with
360-degree views. Wang _et al_. [63] introduced Google Scanned Objects. Other
datasets for training multi-view algorithms include DTU [26], LLFF [43], Tanks
and Temples [29], Spaces [13], RealEstate10K [79], SRN [59], Transparent
Objects [25], ROBI [70], CO3D [55], SAPIEN [69], and BlendedMVS [72].
Recently, RTMV [60] introduced a ray-traced posed multi-view dataset with 2000
scenes and high-resolution images. Unfortunately, the current posed multi-view
datasets commonly used in NeRF research are either small in image resolution
or the number of posed images (SRN [59] and NMR [49]), small in the number of
scenes/shapes and classes (Synthetic NeRFs [44]), or lack structure (DTU [26],
RTMV [60]) and Objeverse [11]. A detailed multi-attribute comparison is
provided in Table 1.

##  3 SPARF: a Large Dataset of 3D Shapes Radiance Fields

###  3.1 Motivations for SPARF

One of the goals of this work is to learn to generate high-quality SRFs in one
forward pass of a deep network to enable fast novel view synthesis. In order
to do this, harnessing the power of deep networks would require a large
dataset of SRFs that follow a distinct distribution (_e.g_. different class
categories). Datasets similar to SPARF (SRN [59], NMR [49]) form the basis of
many advancements that pushed the community of neural radiance fields forward
[75, 34]. As can be seen in Table 1 and Figures 2 and 3, SPARF naturally
extends both datasets in orders of magnitude in resolution, scale, and
diversity, benefiting the entire research community. Furthermore, the density
of views ( $\sim$ 17M ) is higher than in previous datasets. This setup helps
in studying out-of-distribution generalization of the views as a new benchmark
that was neglected in previous works (see Section 5.3).

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |

Figure 4: SRFs: The optimized Sparse Radiance Fields in SPARF. A total of one
million SRFs have been collected in SPARF, including on multiple voxel
resolutions: 32 (top), 128 (middle), and 512 (bottom).

###  3.2 Dense Posed Multi-view Image Dataset

The first step in collecting the desired large high-quality radiance field
dataset is to collect a synthetic posed multi-view dataset. We use ShapeNet
Core 55 [5] as the data of choice for 3D shapes. For rendering, we used Pyglet
[2] API through Trimesh library [10]. The renderer is based on OpenGL [67]
rasterizer to render over 17 million images of around 40,000 shapes from 13
different classes at a high resolution of $400\times 400$. Every shape is
rendered equidistantly from 400 views distributed in a spherical configuration
surrounding the object, including from the bottom (see Figure 2 for examples).
An additional 20 views are rendered from random views from the same distance
as test views for novel view synthesis tasks. Furthermore, an additional 10
views are rendered randomly from random distances bounded by a reasonable
range, such that at least a part of the object is guaranteed to be visible.
This last set is aimed at robustness purposes to test whether novel view
synthesis methods can generalize to out-of-distribution posed views.

SRF Renderings | Extracted Mesh
---|---
|  |  |

Figure 5: Extracting 3D Meshes from SRFs. Since SPARF and SuRFNet lie on the
3D voxel’s space, extracting the mesh is straightforward with one pass of
MarchingCubes [37].

###  3.3 Multi-Resolution 3D SRFs

Sparse Radiance Field (SRF) can be defined as a voxel grid of dimension $1+d$,
where $d$ is the dimension of radiance colors
$\boldsymbol{\rho}_{i,j,k}\in\mathbb{R}^{d}$ at that specific $(i,j,k)$
indexed voxel in addition to one dimension for density
$\alpha_{i,j,k}\in\mathbb{R}$. We assume that the grid is of size $H$ in each
of the three dimensions: $\mathcal{X}\in\mathbb{R}^{H^{3}\times(1+d)}$. Since
the SRF is sparse, it can be represented with the COO format [8] as a set of
$M$ tuples of positive integer coordinates $\mathbf{c}\in\mathbb{Z}^{+}$ and
features $\mathbf{f}\in\mathbb{R}^{d+1}$ with the sparsity of
$1-\frac{M}{H^{3}}$ as follows:

$\displaystyle\mathcal{X}_{\text{non-
empty}}=\\{(\mathbf{c_{m}},\mathbf{f_{m}})\\}_{m=1}^{M}$ (1)

The ordering of the set of tuples is arbitrary, but the features
$\mathbf{f}_{m}$ consist of the density $\alpha_{i,j,k}$ and radiance colors
$\boldsymbol{\rho}_{i,j,k}$ at that location $\mathbf{c}_{m}=(i,j,k)$. For the
radiance field colors, we use the spherical harmonics proposed in Plenoxels
[14] for view-dependent learning of radiance common in NeRFs [44]. The SRF can
be viewed as a encoding of the NeRF MLP into sparse voxels for efficient
optimization and volume rendering. In many 3D object tasks, a coarse-to-fine
approach is followed [33], demanding multiple resolutions. Hence, we collect
the SPARF with multiple resolutions $H\in\\{32,128,512\\}$, as shown in Figure
4. We used an adaptation of Plenoxels [14] to collect the dataset of a total
of one million SRFs as we detail next. In order to scale up the Plenoxels
optimization for this huge number of shapes and variants, we utilize a large
number of images in the SPARF dataset to reduce the iterations to a minimum
number while maintaining a high average PSNR across the dataset for the
collected SRFs across the multiple resolutions. highly detailed 3D meshes can
be extracted easily from the collected SRFs as can be seen in Figure 5.

###  3.4 Representing Images with Partial SRFs

In addition to collecting the “whole" part of SPARF that utilized all 400
images for every shape in optimizing the SRFs, we collect “partial" SRFs.
These are SRFs that are quickly optimized on only a small number of images (1
or 3) randomly sampled from all 400 images, resulting in multiple partial SRF
variants of that shape (see Figure 6). The Million SRFs dataset consists of
two types of SRFs, either optimized using the entire set of 400 views to
generate one Plenoxel [14] per object, or optimized using multiple partial
views from each object to generate multiple Plenoxel variants (see Figure 7).
The partial SRFs were utilized as input to our SuRFNet pipeline (refer to
Figure 8). The release of these partial SRFs should assist in 3D pipelines
that consider radiance fields as a 3D data structure [18], as we demonstrate
later in Section 6.1.

##  4 Learning to Generate SRFs

###  4.1 SuRFNet: Sparse Radiance Fields Network

3D Pipeline. Previous methods in few-views novel view synthesis (_e.g_.
PixelNerf [75] and VisionNerf [34]) learn 2D priors to generalize across
different shapes. On the other hand, we propose SuRFNet to distill the image
views into partial 3D SRFs and then perform the learning in the 3D sparse
voxel space to generate full SRFs based on the partial SRFs. Such a 3D
Learning setup ensures multi-view consistency, especially when rendering from
out-of-distribution views (as we show in Section 5.3). The input to the
pipeline is the input partial SRFs from Section 3.4, where the goal is
learning a generalizable network that converts partial SRFs to whole SRFs as
can be seen in Figure 8. We leverage the Minkowski Net [8] as the 3D sparse
convolution network of choice. However, As this novel setup of learning SRFs
poses new challenges, we propose several modifications to the typical pipeline
of training MinkoskiNet (the type of losses and where to define them).

Whole SRF | Partial (3 Views) | Partial (1 View)
---|---|---
|  |

Figure 6: Whole _vs_. Partial SRFs. The partial SRFs are used instead of the
few images that generated them as input to the learning pipeline to generate
the whole SRFs

Challenges of Learning SRFs. As a 3D data structure, SRF is an irregular
volumetric representation that does not necessarily reflect the underlying 3D
shape/scene, as it results from the optimization of posed images into volume.
Many of the non-empty voxels have low densities and do not affect the volume
rendering, but include color information that can confuse the network. Also,
small errors in predicting the densities or radiance colors can result in
large distortions in the rendered images, hurting the overall performance of
novel view synthesis. Another challenge in learning SRFs is vanishing
gradients. The typical sparsity in our setup is $\sim 99\%$, and misalignment
between the input SRF coordinates and output SRF coordinates can further harm
the gradients and affect the learning process. In order to tackle these
issues, we propose three specialized losses detailed next.

Figure 7: SPARF Distribution. We show the distribution of classes in SPARF and
how the one million partial and whole SRFs are distributed. The numbers are
equally distributed on three voxel resolutions: 512,128. and 32.

###  4.2 SRF-Loss

Density loss. The goal of the density loss is to create a dense surface. We
propose the following binary cross-entropy loss on the predicted densities
$\alpha$ as follows:

$\displaystyle L_{\alpha}\leavevmode\nobreak\
\big{(}\mathcal{X}\leavevmode\nobreak\ ,\leavevmode\nobreak\
\hat{\mathcal{X}}\big{)}\leavevmode\nobreak\
=-{(\mathbf{\hat{y}}\log(\mathbf{y})+(\mathbf{1}-\mathbf{\hat{y}})\log(\mathbf{1}-\mathbf{y}))}$
(2) $\displaystyle\text{s. t.}\quad\mathbf{\hat{y}}=\leavevmode\nobreak\
\mathbbm{1}\left(\mathcal{S}(\hat{\mathcal{X}}_{\alpha})>\alpha_{\text{dense}}\right)\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\
\mathbf{y}=\mathcal{S}\left(\mathbf{F}(\mathcal{X})\right)_{\alpha}$

where $\mathcal{X},\hat{\mathcal{X}}$ are the input partial SRF and the ground
truth whole SRFs respectively (as defined in Eq (1) ), and
$\alpha_{\text{dense}}$ is the density threshold distinguishing dense voxels
from the air (usually set to 0). The sampling function $\mathcal{S}$ samples
points in the grid space where the loss is defined on the outputs $\mathbf{y}$
and the ground truth densities’ labels $\mathbf{\hat{y}}$. One of the
challenges in working with sparse voxels of high resolution is that training
the pipeline can not involve densifying the voxels to the original resolution
due to prohibitive memory requirements. The input/output topologies are not
necessarily the same, as the sparse convolutional strides and pruning can
alter the sparse voxels’ coordinates. This is why the sampling function
$\mathcal{S}$ in Eq (2) is of utmost importance in guiding the training of
SuRFNet. We sample the loss according to Quantized Gaussian sampling
(Q-Gaussian) with random coordinates centered at the middle of the voxel grid.
Simply put, the Q-Gaussian is 3D normal distribution quantized to integer
coordinates to give a prior about where the output is expected and where the
loss is defined. Further details and alternative configurations are provided
in Section 6.1.

Radiance color loss. To ensure the output SRFs follow the ground truth
optimized SRFs in radiance color, we follow the simple L1 loss on the radiance
colors $\boldsymbol{\rho}$. However, as mentioned earlier, some of the non-
empty voxels contain low density and will not be seen in the rendering, and
can contain any random colors. Therefore, we mask these non-empty low-density
voxels out of the L1 loss as follows:

$\displaystyle L_{\rho}\leavevmode\nobreak$
$\displaystyle\big{(}\mathcal{X}\leavevmode\nobreak\ ,\leavevmode\nobreak\
\hat{\mathcal{X}}\big{)}\leavevmode\nobreak\
=\|\mathbf{M}_{\alpha}\mathbf{F}(\mathcal{X})_{\rho}-\mathbf{M}_{\alpha}\hat{\mathcal{X}}_{\rho}\|_{1}$
(3) $\displaystyle\text{s. t.}\quad\mathbf{M}_{\alpha}=\leavevmode\nobreak\
\mathbbm{1}(\hat{\mathcal{X}}_{\alpha}>\alpha_{\text{dense}})$

Figure 8: SuRFNet: Learning to Generate Whole Radiance Fields from Partial
Views. We process the input images into partial SRFs $\mathcal{X}$ before
learning a sparse convolutional network to generate the whole SRF. A
perceptual loss is employed on the rendered images from poses $\phi$ to
enhance the perceptual quality of the generated SRF. The whole SRF
$\hat{\mathcal{X}}$ is used to 3D-supervise the SRF generation with density
and radiance color losses.

Perceptual loss. Using only the 3D radiance color loss in Eq (3) ignores the
rendering quality of the generated SRF, and would make it sensitive to
hyperparameters (see Figure 9). Hence, we introduce an online perceptual loss
that would volume render the generated SRF during training from $M$ random
views that come from the same ground truth image poses $\phi$, and an L1 loss
is defined between the generated images and the ground truth posed images
$\mathbf{I}_{\phi}$.

$\displaystyle L_{R}\leavevmode\nobreak$
$\displaystyle\big{(}\mathcal{X}\big{)}\leavevmode\nobreak\
=\|\mathcal{R}_{\phi}\left(\mathbf{F}(\mathcal{X})\right)-\mathbf{I}_{\phi}\|_{1},$
(4)

where $\mathcal{R}_{\phi}$ is the fast volume rendering function that renders
SRFs from poses $\phi$ using the trilinear interpolation between voxels
proposed in Plenoxels[14].

The final loss to train the network $\mathbf{F}$ would be combining the three
losses in Eq (2,3,4) as follows:

$\displaystyle\text{Loss}_{\mathbf{F}}\leavevmode\nobreak\
=\leavevmode\nobreak\ L_{\alpha}\leavevmode\nobreak\ +\leavevmode\nobreak\
\lambda_{\rho}L_{\rho}\leavevmode\nobreak\ +\leavevmode\nobreak\
\lambda_{R}L_{R},$ (5)

where $\lambda_{\rho},\lambda_{R}$ are hyperparameters to control the radiance
colors compared to the density predictions. The network is trained on all $N$
whole SRFs $\hat{\mathcal{X}}$ in the dataset, while the input SRFs
$\mathcal{X}$ are randomly chosen from several partial SRFs created by the
same number of images from those shapes.

| SPARF Validation PSNR |
---|---|---
Baselines | chair | watercraft | rifle | display | lamp | speaker | cabinet | bench | car | airplane | sofa | table | phone | mean
Plenoxels [14] (1V) | 9.2 | 11.1 | 11.7 | 8.0 | 13.6 | 8.2 | 10.4 | 10.5 | 7.1 | 12.8 | 9.3 | 9.9 | 8.3 | 10.0
[HTML]EFEFEF Plenoxels [14] (3V) | 10.7 | 13.3 | 14.9 | 9.7 | 15.8 | 10.4 | 12.4 | 11.6 | 7.1 | 14.6 | 11.6 | 10.8 | 9.7 | 11.7
PixelNerf [75] (1V) | 13.3 | 16.3 | 16.7 | 11.9 | 17.6 | 11.3 | 14.5 | 14.6 | 13.2 | 19.2 | 13.3 | 13.2 | 13.2 | 14.5
[HTML]EFEFEF PixelNerf [75] (3V) | 13.5 | 16.6 | 16.9 | 12.2 | 17.9 | 11.9 | 14.9 | 14.8 | 13.4 | 19.4 | 13.4 | 13.3 | 13.3 | 14.7
VisionNeRF [34] (1V) | 13.0 | 15.6 | 15.8 | 11.7 | 16.7 | 11.2 | 14.0 | 14.3 | 12.7 | 17.8 | 13.3 | 13.0 | 12.6 | 14.0
[HTML]EFEFEF SuRFNet (ours) (1V) | 11.6 | 16.2 | 17.0 | 12.0 | 16.2 | 12.6 | 17.0 | 13.5 | 16.6 | 17.5 | 14.1 | 10.1 | 15.3 | 14.6
SuRFNet (ours) (3V) | 15.3 | 18.3 | 18.8 | 15.0 | 19.0 | 16.6 | 20.0 | 15.6 | 16.6 | 18.5 | 18.1 | 14.9 | 17.8 | 17.3

Table 2: SPARF Benchmark on Out-Of-Distribution View Synthesis. We compare the
validation PSNR of some of the widely used novel view synthesis techniques on
the SPARF dataset for the generalization of novel view synthesis beyond a
single example and on view tracks completely different from the ones seen in
training views. One view (1V) and three views (3V) inputs are reported.

##  5 Experiments

###  5.1 Collecting SPARF

The engineering aspect of collecting, storing, and organizing the one million
SRFs with multiple resolutions is as challenging as training properly on SRFs.
In order to do that in manageable time and memory, while maintaining high
quality in the optimized samples, a set of strategies is employed. The
dimension of the radiance color $d$ is chosen to be $d=3\times 4=12$ of 4
spherical harmonics factors of RGB channels for view-dependent SRF and
$d=3\times 1=3$ for fixed RGB colors of the SRF. Since the input partial SRFs
are noisy, we use $d=3$ while the final output SRFs use $d=12$ for high-
quality image generation. Using fewer Spherical Harmonics components (from 9
to 4 per RGB channel) reduces the optimization space by 40% and time by 10%,
while maintaining the same PSNR. Using RGB as colors instead of SH factors
reduces PSNR by $\sim 1$ dB, space by 80%, and time by 20%. Running Plenoxels
[14] for fewer iterations (3$\times$12K) reduces the time by 30% while
maintaining the same PSNR. Using 400 views/shapes in SPARF to optimize the
SRFs keep the time manageable in optimization ($\sim 4$ minutes for the 512
resolution whole SRFs) while maintaining high PSNR ($\sim 30$dB). It takes way
less time than that for the rest of the setups (lower resolution or partial
SRFs). Other similar sparse voxels optimizations (_e.g_. InstantNGP [46]) can
be used to obtain SRFs similarly as fast with similar quality. A total of four
variants of the partial SRFs are collected for all the resolutions and the
partials use 1 and 3 images. The anatomy of the distribution of classes and
SRFs in SPARF is presented in Figure 7. More details about SPARF and
visualizations of some of its samples are available in the supplementary
material.

Output w/o $L_{R}$ | Output w/ $L_{R}$ | Whole SRF
---|---|---
|  |

Figure 9: Effect of the Perceptual Loss $L_{R}$. Adding a perceptual loss on
volume-rendered images during training SuRFNet insures the rendered images
remain closer to how they should be rendered, as the 3D radiance colors
supervision won’t guarantee the rendering quality. (left): without perceptual
loss , (middle): with the loss.

###  5.2 Training Setup

Dataset. We pick our SPARF for the task of predicting whole SRFs for the
purpose of novel view synthesis. The other datasets (SRN [59] and NMR [49])
are too small or low in resolution, which prevents optimizing high-quality
radiance fields (see Figure 3).

Evaluation metrics. Following the previous novel view synthesis works [75, 14,
34], we use PSNR, SSIM, and LPIPS [78] as metrics to evaluate the synthesis.
However, one key difference between our work and previous ones is that our
setup is a learning setup (with training and validation), while previous works
treat it as an optimization problem. Most previous works on novel view
synthesis try to only generalize the generated views on the same shape, while
we aim to generalize _across shapes of the same category_ and across views. We
treat the collected whole SRFs as ground truth labels for the input few images
from the training set. We consider the validation PSNR, SSIM, and LPIPS of the
input images at the validation SRF set of shapes (on the test images of those
shapes) as the main evaluation metrics. Also, we report validation accuracy =
$\frac{\text{validation PSNR with test few images}}{\text{whole SRF
optimization's PSNR}}$ and propose it as a new metric to evaluate such a
learning setup of SRFs. Furthermore, as we describe in Section 3, SPARF has 10
posed Out-Of-Distribution (OOD) images for every 3D shape to evaluate the
robustness of novel view synthesis methods in the unconstrained setup. We
report these OOD PSNR, SSIM, LPIPS, and Accuracies as well.

Basleines. We use PixelNeRF [75] (ResNet34 backbone), Plenoxels [14], and
VisionNerf [34] (ViT-B backbone) as the main baselines for our work. Our
SuRFNet network has two sizes: large (87 million parameters) and small (13
million parameters).

|  |  |
---|---|---|---

Figure 10: SuRFNet: Generating High-Resolution Radiance Fields. We show some
volume-rendered sequences based on our SuRFNet voxel radiance field outputs ,
given only 3 input images.

###  5.3 Results

We show qualitative results of generating novel views from few input images on
unseen shapes in Figure 10. We also show qualitative comparisons in Figure 12.
We present a summary of the quantitive evaluations next, where SuRFNet
achieves state-of-the-art results on unconstrained novel view synthesis from
one or few images on unseen shapes.

Figure 11: Scaling-Up Training on SRFs. As the training data (partial SRFs) of
radiance fields increase, the generalization improves, as can be seen in the
car class here. The 3-view and 1-view metrics are reported with test and OOD
metrics.

SPARF View-Generalization benchmark. In Table 2, we report the average PSNR
results on the validation set of SPARF for different methods on unseen shapes
during training on all 13 different object classes and on out-of-distribution
views. It shows that our SuRFNet can generalize to out-of-distribution views
on unseen shapes during test time, surpassing state-of-the-art PixelNeRF [75]
and VisionNeRF [34]. Visual comparisons can be found in Figure 12. As can be
seen from those results, the learned 3D prior results in multi-view
consistency, especially when rendering from out-of-distribution views.

Input View | Target Ground Truth | SuRFNet (ours) | PixelNeRF [75] | VisionNeRF [34]
---|---|---|---|---
|  |  |  |  |  |  |  |

Figure 12: Qualititve Comparisons. We show different renderings from our
SuRFNet outputs generated from a single image compared to other methods
(pixel-NeRF [75], and VisionNeRF [34] ) and whole SRF ”GT" renderings. Note
that the predicted two views lay outside the training views distribution
(zoomed in randomly). This test highlights the weakness of the 2D-based
baselines [75, 34] outside the training track, while our 3D approach maintains
multi-view consistency everywhere.

Real | Generated
---|---
|  |  |  |
|  |  |  |

Figure 13: Real images We show real images of Co3D [55] and the corresponding
generated views from our SRFs (top row) and pretrained PixelNeRF [75] (bottom
row).

##  6 Analysis and Insights

###  6.1 Ablation Study

Effect of Dataset Size. We study the effect of increasing the dataset size
(Partial SRFs) on the generalization performance of SuRFNet in Figure 11. It
shows that as the size increase ( normalized by the total number of shapes in
car class ), the generalization performance increase. This scalability effect
underlines the importance of SPARF. This justifies collecting 4 variants per
resolution ( as detailed in Figure 7).

Training SuRFNet. We ablate different components of SuRFNet’s architectures
and the loss configuration choices and report the results in Table 3. The
Results show that increasing the size of the network (from 13M to 87M
parameters ) helps improve generalization accuracy. Also, they show the
importance of the loss components proposed in Eq (5). The use of only density
loss creates a reasonably dense shape but without colors. While combining the
density loss with the 3D radiance color loss creates colorful objects, it does
not perform well in the task of novel view synthesis as it does not respect
how the object renders. Small radiance distortions lead to large image errors
(see Figure 9 for the importance of the perceptual loss). More ablations on
the network, loss sampling, and hyperparameters of training SuRFNet can be
found in supplementary material.

3D Backbone | Loss components | Results
---|---|---
Small | Large |  ($L_{\alpha}$) |  ($L_{\rho}$) |  ($L_{R}$) | Val. Acc.
✓ | - | ✓ | ✓ | - | 65.2
✓ | - | ✓ | ✓ | ✓ | 65.7
- | ✓ | ✓ | ✓ | - | 66.4
- | ✓ | ✓ | ✓ | ✓ | 68.2

Table 3: Ablation Study. We ablate different components of in SuRFNet (3D
backbone and SRF-Loss) and report validation accuracy of car class.

###  6.2 Testing on Real Images

Previous works on similar large-scale learning setups report only the ShapeNet
benchmark [34, 56, 64] for standardized evaluations. Testing on real images
can provide additional value. Hence, In Fig. 13, we show renderings of whole
SRFs (used in our SuRFNet pipeline) trained on real Co3D [55] images _vs_.
pretrained PixelNeRF renderings. This demonstrates the potential for the
proposed method on real images. The small noise is due to background
corruption resulting from imperfect masks provided in Co3D.

###  6.3 Speed and Compute Cost

To assess the contributions of the SuRFNet pipeline, we study the time and
memory requirements of each element in the pipeline. We record in Table 4 the
number of floating-point operations (GFLOPs) and the runtime of a forward pass
(including rendering) for a single output image from one input image on Titan
RTX GPU. Speed and compute details about collecting the SRFs are included in
Section 5.1.

Network | Network | Network | Parameters | Rendering
---|---|---|---|---
FLOP (G) | Infer. (ms) | Number (M) | Speed (FPS)
PixelNeRF [75] | 7.3 | 5.33 | 21.8 | 1.2
VisionNerf [34] | 33.7 | 12.5 | 68.6 | 1.2
SuRFNet (small) |  $\sim$15 | 14.4 | 13.4 | 15
SuRFNet (large) |  $\sim$100 | 90.0 | 87.3 | 15

Table 4: Time and Memory Requirements. We assess the computational cost of the
main components studied

##  7 Conclusions and Future Work

We propose a large-scale dataset SPARF of sparse radiance fields that include
around one million SRFs and 17 million posed images of 3D shapes. The dataset
aims to move the community in the direction of treating radiance fields as a
3D data structure, instead of optimization results and MLP fitting. We
domnstrated SPARF’s utility using our proposed SuRFNet pipeline.

One crucial limitation in this work is the large amount of compute and memory
necessary to create, store, and process SRFs, especially at high-resolution
voxel grids. This creates a bottleneck in training, developing, and building
on SRFs. Developing efficient methods to work and learn from sparse voxel
grids would be a viable plan moving forward in order to develop deep and large
models in this space.

Acknowledgement. This work was supported by the ERC Starting Grant Scan2CAD
(804724) and the German Research Foundation (DFG) Research Unit “Learning and
Simulation in Visual Computing". It was also supported by the KAUST Office of
Sponsored Research through the Visual Computing Center funding. Part of the
support is also coming from the KAUST Ibn Rushd Postdoc Fellowship program.

## References

  * [1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  International Conference on Machine Learning (ICML), 2018.
  * [2] Alex Holkner et al.  Pyglet.
  * [3] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3981–3990, June 2022.
  * [4] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5799–5809, 2021.
  * [5] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [6] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124–14133, 2021.
  * [7] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  In Advances in Neural Information Processing Systems, pages 9609–9619, 2019.
  * [8] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
  * [9] Angela Dai and Matthias Nießner.  Scan2mesh: From unstructured range scans to 3d meshes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5574–5583, 2019.
  * [10] Dawson-Haggerty et al.  trimesh.
  * [11] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 13142–13153, 2023\.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [13] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2367–2376, 2019.
  * [14] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5501–5510, June 2022.
  * [15] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  Get3d: A generative model of high quality 3d textured shapes learned from images.  In Advances In Neural Information Processing Systems, 2022.
  * [16] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh r-cnn.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9785–9795, 2019.
  * [17] Shubham Goel, Georgia Gkioxari, and Jitendra Malik.  Differentiable stereopsis: Meshes from multiple views using differentiable rendering.  arXiv preprint arXiv:2110.05472, 2021.
  * [18] Pengsheng Guo, Miguel Angel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M Susskind, and Qi Shan.  Fast and explicit neural view synthesis.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3791–3800, 2022.
  * [19] Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu.  Deligan : Generative adversarial networks for diverse and limited data.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [20] Abdullah Hamdi and Bernard Ghanem.  Ian: Combining generative adversarial networks for imaginative face generation.  arXiv preprint arXiv:1904.07916, 2019.
  * [21] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem.  Mvtn: Multi-view transformation network for 3d shape recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, October 2021.
  * [22] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem.  Voint cloud: Multi-view point cloud representation for 3d understanding.  In The Eleventh International Conference on Learning Representations, 2023.
  * [23] Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or.  Point2mesh: A self-prior for deformable meshes.  arXiv preprint arXiv:2005.11084, 2020.
  * [24] Qixing Huang, Xiangru Huang, Bo Sun, Zaiwei Zhang, Junfeng Jiang, and Chandrajit Bajaj.  Arapreg: An as-rigid-as possible regularization loss for learning deformable shape generators.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5815–5825, 2021.
  * [25] Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg.  Dex-nerf: Using a neural radiance field to grasp transparent objects.  arXiv preprint arXiv:2110.14217, 2021.
  * [26] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engin Tola, and Henrik Aanæs.  Large scale multi-view stereopsis evaluation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 406–413, 2014.
  * [27] Yoonwoo Jeong, Seungjoo Shin, Junha Lee, Chris Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park.  PeRFception: Perception using radiance fields.  In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [28] Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess.  Unsupervised learning of 3d structure from images.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4996–5004. Curran Associates, Inc., 2016.
  * [29] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun.  Tanks and temples: Benchmarking large-scale scene reconstruction.  ACM Transactions on Graphics (ToG), 36(4):1–13, 2017.
  * [30] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum.  Deep convolutional inverse graphics network.  In Advances in neural information processing systems (NIPS), pages 2539–2547, 2015.
  * [31] Ruihui Li, Xianzhi Li, Ka-Hei Hui, and Chi-Wing Fu.  Sp-gan: Sphere-guided 3d shape generation and manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–12, 2021.
  * [32] Zuoyue Li, Tianxing Fan, Zhenqiang Li, Zhaopeng Cui, Yoichi Sato, Marc Pollefeys, and Martin R Oswald.  Compnvs: Novel view synthesis with scene completion.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I, pages 447–463. Springer, 2022.
  * [33] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
  * [34] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In WACV, 2023.
  * [35] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  arXiv preprint arXiv:2303.11328, 2023.
  * [36] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  ACM Trans. Graph., 38(4):65:1–65:14, July 2019.
  * [37] William E. Lorensen and Harvey E. Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  SIGGRAPH Comput. Graph., 21(4):163–169, aug 1987.
  * [38] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [39] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  * [40] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  Realfusion: 360$\\{$$\backslash$deg$\\}$ reconstruction of any object from a single image.  In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
  * [41] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [42] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  arXiv preprint arXiv:2112.03221, 2021.
  * [43] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  * [44] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [45] Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder.  Autorf: Learning 3d object radiance fields from single view observations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3971–3980, June 2022.
  * [46] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  * [47] Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia.  Polygen: An autoregressive generative model of 3d meshes.  In International Conference on Machine Learning, pages 7220–7229. PMLR, 2020.
  * [48] Michael Niemeyer and Andreas Geiger.  Giraffe: Representing scenes as compositional generative neural feature fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453–11464, 2021.
  * [49] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3504–3515, 2020.
  * [50] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  * [51] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [52] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  arXiv preprint arXiv:2306.17843, 2023.
  * [53] Anurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black.  Generating 3d faces using convolutional mesh autoencoders.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 704–720, 2018.
  * [54] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [55] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10901–10911, October 2021.
  * [56] Konstantinos Rematas, Ricardo Martin-Brualla, and Vittorio Ferrari.  Sharf: Shape-conditioned radiance fields from a single view.  arXiv preprint arXiv:2102.08860, 2021.
  * [57] Katja Schwarz, Axel Sauer, Michael Niemeyer, Yiyi Liao, and Andreas Geiger.  Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids.  In Advances in Neural Information Processing Systems (NeurIPS), 2022\.
  * [58] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim.  Let 2d diffusion model know 3d-consistency for robust text-to-3d generation.  In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
  * [59] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  Advances in Neural Information Processing Systems, 32, 2019.
  * [60] Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, et al.  Rtmv: A ray-traced multi-view synthetic dataset for novel view synthesis.  arXiv preprint arXiv:2205.07058, 2022.
  * [61] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  * [62] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [63] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2021.
  * [64] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  In The Eleventh International Conference on Learning Representations, 2023.
  * [65] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  In The Eleventh International Conference on Learning Representations, 2023.
  * [66] Xingkui Wei, Zhengqing Chen, Yanwei Fu, Zhaopeng Cui, and Yinda Zhang.  Deep hybrid self-prior for full 3d mesh generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5805–5814, 2021.
  * [67] Mason Woo, Jackie Neider, Tom Davis, and Dave Shreiner.  OpenGL Programming Guide: The Official Guide to Learning OpenGL, Release 1.  Addison-wesley, 1998.
  * [68] Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao.  3d shapenets: A deep representation for volumetric shapes.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1912–1920, 2015.
  * [69] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al.  Sapien: A simulated part-based interactive environment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11097–11107, 2020.
  * [70] Jun Yang, Yizhou Gao, Dong Li, and Steven L Waslander.  Robi: A multi-view dataset for reflective objects in robotic bin-picking.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9788–9795. IEEE, 2021.
  * [71] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 206–215, 2018.
  * [72] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan.  Blendedmvs: A large-scale dataset for generalized multi-view stereo networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1790–1799, 2020.
  * [73] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman.  Volume rendering of neural implicit surfaces.  Advances in Neural Information Processing Systems, 34:4805–4815, 2021.
  * [74] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [75] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578–4587, 2021.
  * [76] Ye Yu and William AP Smith.  Inverserendernet: Learning single image inverse rendering.  arXiv preprint arXiv:1811.12328, 2018.
  * [77] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan.  Ners: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild.  Advances in Neural Information Processing Systems, 34, 2021.
  * [78] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [79] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.

##  Appendix A Detailed Formulations

###  A.1 Sparse Convolutions

Sparse convolutions are a variant of standard convolutions that are used in
deep learning. In a sparse convolution, only a subset of the input elements is
used in the computation, which allows for more efficient use of computation
resources and can improve the performance of the convolutional neural network.
To perform a sparse convolution, we first define a set of indices that specify
which input elements should be used in the convolution. This set of indices is
called the "support" of the convolution. We then use these indices to select
the relevant input elements and compute the convolution using these elements.
This is typically done by applying a filter to the selected input elements and
summing the results to produce the output of the convolution.

In the simplest 1 D case, let $\mathbf{x}$ be the input tensor, $\mathbf{w}$
be the convolutional filter, and $c$ be the support of the convolution (i.e.
the set of indices specifying which elements of $x$ should be used in the
convolution). The output of the sparse convolution, $\mathbf{y}$, can be
computed as: $\mathbf{y}=\mathbf{x}[c]*\mathbf{w}$ where $*$ denotes the
convolution operation, and $\mathbf{x}[c]$ is the subset of elements from
$\mathbf{x}$ specified by the support $c$. This equation applies the
convolutional filter $\mathbf{w}$ to the selected input elements and sums the
results to produce the output of the convolution. For more detailed
formulation and implementation of the Sparse convolutions we used in our work,
please refer to MinkowskiNetwork [8].

###  A.2 Q-Gaussian loss sampling

In the space of sparse voxels of high resolution, defining where the loss is
sampled is difficult, especially if the output topology is unknown. One of the
challenges in working with sparse voxels of high resolution (_e.g_. 512) is
that training can not involve densifying the voxels to the original resolution
(_i.e_. 5123) due to prohibitive memory requirements. The input/output
topologies are not necessarily the same, as the sparse convolutional strides
and pruning can alter the sparse voxels’ coordinates. This is why the sampling
function $\mathcal{S}$ in Eq (2) is of utmost importance in guiding the
training of SuRFNet. We sample at random coordinates centered at the center of
the voxel grid $\mathcal{S}:\leavevmode\nobreak\
\mathbf{c}\sim\mathcal{Q}\left(\mathcal{N}(\frac{\mathbf{H}}{2},\,\frac{\mathbf{H}\sigma^{2}}{2}\mathbb{I})\right)$,
where $\mathbf{H}=(H,H,H)$ is the voxel grid resolution vector, $\mathbb{I}$
is the identity matrix, $\sigma$ is a hyperparameter determining the spread of
the loss, and $\mathcal{Q}:\mathbb{R}^{3}\rightarrow{\mathbb{Z}^{+}}^{3}$ is
the quantization-and-cropping function of coordinates that ensure the output
coordinates are integers within bounds $\mathbf{c}\in[0,1,...,H-1]^{3}$. We
discuss more details about $\mathcal{S}$ and alternative configurations in
Section C.3. Simply put, the Q-Gaussian is 3D normal distribution quantized to
integer coordinates to give prior about where the output is expected and where
the loss is defined.

##  Appendix B Detailed Setup

###  B.1 SPARF Dataset

All the rendered images are of $400\times 400$ resolution with 4 channels (RGB
+ alpha channel for background). SPARF has three main splits for every 3D
shape: training views (400 views), test views (20 views), and an OOD “hard"
views (10 views) as can be shown in Figure 26. Regarding the collected SRFs,
Plenoxels [14] is used as the base module. The spherical harmonics dimension
of the whole SRFs is $d=4\times 3=12$, while for partial SRFs, it is
$d=1\times 3=3$. Most of the hyperparameters used in optimizing the SRFs are
the default ones proposed in the Plenoxels paper [14] (as can be seen in the
attached code under Svox2/opt/opt-py). However, the following hyperparameters
were engineered in order to scale up the optimization and maintain the quality
of the SRFs ( as can be seen in Figure 27, and 28). The flickering temporal
noise introduced in the $32^{3}$ resolution SRFs is due to the extremely low
number of voxels representing the radiance fields while the views are rendered
densely from the spiral sequence, hence aliasing occurs.

Running Plenoxels [14] for fewer iterations (3$\times$12K) reduces the time by
30% while maintaining the same PSNR. Using 400 views/shapes in SPARF to
optimize the SRFs keep the time manageable in optimization ($\sim 4$ minutes
for the 512 resolution) while maintaining high PSNR ($\sim 30$dB). When saving
the SRFs, we only save the set of coordinates (integers) and float features
(densities and radiance components). The upsampling iteration of Plenoxels is
set to 1$\times$12K for faster convergence. The distribution of the collected
dataset is detailed in Table 5. More examples of the whole _vs_. partial SRFs
collected in SPARF can be found in Figure 14. A total of 200K GPU hours are
used in the optimization process to collect SPARF. The whole SRFs are easily
convertible to high-quality meshes using Marching Cubes [37] as shown in
Figure 15. While the SPARF dataset is indeed larrge in total ( 3.4 TB), its
posed-images part is only 360 GB, which makes it manageble for training on
other applications that require dense posed images.

SRF | Voxewl | Nb. of | Nb. of
---|---|---|---
Type | Resolution | Variants | SRFs
Partial | 32 |  $4\times$1-view | 158,816
$4\times$3-view | 158,816
128 |  $4\times$1-view | 158,816
$4\times$3-view | 158,816
512 |  $4\times$1-view | 158,816
$4\times$1-view | 158,816
Whole | 32 |  $1\times$400-view | 39,704
128 |  $1\times$400-view | 39,704
512 |  $1\times$400-view | 39,704
Total | - | - | 1,072,008

Table 5: SPARF Anatomy. We show the distribution of the one million SRFs
collected in SPARF between multiple resolutions and between whole and partial
SRFS.

Whole SRF | Partial (3 Views) | Partial (1 View)
---|---|---
|  |
|  |
|  |

Figure 14: Whole _vs_. Partial SRFs. The partial SRFs are used instead of the
few images that generated them as input to the learning pipeline to generate
the whole SRFs

SRF Renderings | Extracted Mesh
---|---
|  |  |
|  |  |
|  |  |
|  |  |

Figure 15: Extracting 3D Meshes from SRFs. Since SPARF and SuRFNet live on the
3D voxel’s space, extracting the mesh is straightforward with one pass of
MarchingCubes [37].

###  B.2 SuRFNet Training

We use a voxel resolution of $128^{3}$ of the SPARF dataset in most of the
learning experiments and visualizations in this work, unless otherwise clearly
stated. This choice is to reduce the computational cost of training the heavy
pipeline and to facilitate the development of proper learning methods on SRFs.
The input SRF is normalized with a fixed value of 10,0000 for the density and
10 for the colors, to ensure the distribution lies within -1 to 1. The
Q-Gaussian std $\sigma$ is set to 0.444 (studied more in Section C). The
strides for the SuRFNEt are all set 2, while the network depth $l=3$ modules.
The batch size for training is 14 when A100 GPUs are used and 6 when V100 GPUs
are used. The training saturates at 100 epochs. The optimizer used is AdamW
[38] with a learning rate of 0.01, a momentum of 0.9, a weight decay of
$1e-5$, and a learning rate exponential decay rate of 0.99. The
hyperparameters $\lambda_{R},\lambda_{\alpha},\lambda_{\rho}$ are all set
independently to each class, where a different network is trained on each
class separately. Most classes have
$\lambda_{\alpha}=30.0,\lambda_{\rho}=1.0,\lambda_{R}$. We did not prune the
output sparse voxel as this leads to harming performance most of the time and
increase the problem of vanishing gradients. The background color of the
rendered images $\mathcal{R}_{\phi}\left(\mathbf{F}(\mathcal{X})\right)$ is
masked out from the perceptual loss and the density component $\alpha$ of the
output SRF is also not affected by the perceptual loss, as this can cause
excessive densities around the object, leading to deteriorating the SRF output
perceptuality. During training with the perceptual loss, three randomly
selected images from three different $\phi$ as used as labels for the three
rendered images from the output
$\mathcal{R}_{\phi}\left(\mathbf{F}(\mathcal{X})\right)$. The SuRFNet is
jointly predicting the density and radiance of spherical harmonics colors, but
with different heads. More setup details can be found in the attached code and
analyzed further in Section C. We train a separate model for each class, to
maintain high-quality generation of 3D SRFs.

Retraining Baselines. To compare to the preprinted baselines PixelNeRF and
VisionNerf ( which use $64\times 64$ resolution), we upsample their resolution
at inference at test poses while using their pretrained weights of the NMR
dataset. The upsampling is using the bicubic sampling of the Pytorch
Transforms library. These baselines are used in this works by default unless
otherwise specified.

Retraining the methods from scratch on the high resolution $400\times 400$ is
computationally expensive. To illustrate the retraining cost, VisionNeRF [34]
was originally trained on 16 A100 GPUs for a week just to converge on
$64\times 64$ resolution. SPARF’s $400\times 400$ resolution images would need
$\sim$ 39 times as much time/compute due to per-pixel sampling. In contrast,
our SuRFNet was trained on a single V100 GPU for 3 days, which allows for
fine-tuning of the learning pipeline for quick convergence. However, for a
fair comparison, We retrain PixelNeRF [75] on the 13 categories and report the
3-view and 1-view PSNR results in Table 2. We see that our SuRFNet indeed
surpasses this baseline trained on the same SPARF data. The results of
retraining are not very different from the pretrained weights (slight
improvement) because training on SPARF high-resolution images is unstable
using these 2D-based NeRF networks that need a per-pixel sampling, which
diverges the training in some cases. The full results are shown in Table 6

##  Appendix C Additional Analysis

| SPARF Classes |
---|---|---
Baselines | chair | watercraft | rifle | display | lamp | speaker | cabinet | bench | car | airplane | sofa | table | phone | mean
Plenoxels [14] (1V) | 10.1 | 12.1 | 12.6 | 8.7 | 14.7 | 8.7 | 10.9 | 11.4 | 7.7 | 14.0 | 9.7 | 10.5 | 9.5 | 10.8
[HTML]EFEFEF Plenoxels [14] (3V) | 10.8 | 13.3 | 15.6 | 9.7 | 16.2 | 10.1 | 12.1 | 12.1 | 9.0 | 15.4 | 11.4 | 10.8 | 10.2 | 12.1
PixelNeRF [75] (1V) | 10.8 | 14.1 | 14.2 | 9.0 | 15.6 | 9.2 | 10.5 | 12.4 | 10.1 | 15.7 | 11.1 | 10.6 | 11.1 | 11.9
[HTML]EFEFEF PixelNeRF [75] (3V) | 11.0 | 14.1 | 14.2 | 9.3 | 15.7 | 9.4 | 10.6 | 12.7 | 10.1 | 15.7 | 11.3 | 10.9 | 11.4 | 12.0
PixelNerf ∗ (1V) | 13.8 | 12.2 | 15.0 | 17.5 | 19.0 | 11.2 | 17.5 | 13.3 | 12.2 | 17.9 | 11.3 | 11.7 | 14.7 | 14.5
[HTML]EFEFEF PixelNerf ∗ (3V) | 17.5 | 13.6 | 16.2 | 11.7 | 20 | 15.6 | 13.1 | 17.6 | 12.1 | 18.2 | 16.2 | 12.1 | 10.7 | 15.0
VisionNeRF [34] (1V) | 16.5 | 18.4 | 18.5 | 15.1 | 19.3 | 13.2 | 16.1 | 16.3 | 13.8 | 21.8 | 15.1 | 14.8 | 14.0 | 16.4
[HTML]EFEFEF SuRFNet (ours) (1V) | 15.7 | 15.5 | 19.1 | 14.1 | 18.5 | 14.5 | 18.7 | 15.6 | 18.1 | 20.3 | 16.3 | 14.1 | 17.4 | 16.8
SuRFNet (ours) (3V) | 18.6 | 20.7 | 20.9 | 17.1 | 21.2 | 18.5 | 21.7 | 17.6 | 18.9 | 21.9 | 20.4 | 16.7 | 20.0 | 19.5

Table 6: SPARF Benchmark on Novel View Synthesis (Normal Test). We compare the
validation PSNR of some of the widely used novel view synthesis techniques on
the SPARF dataset for the generalization of novel view synthesis beyond a
single example and on the normal testing-views tracks similar to the ones seen
in training views. One view (1V) and three views (3V) inputs are reported, and
∗ indicates retraining the baseline backbone on the high-resolution images of
the SPARF dataset.

###  C.1 Shiny Objects

Some of the rendered objects have reflective materials, resulting in distorted
optimized radiance fields for these shapes despite using all of the views. We
separate these distorted SRFs (only 76 shapes in total) from the SPARF dataset
(see Figure 16).

Figure 16: Shiny Objects Corrupts SRFs:. Optimizing SRFs on shiny objects with
a reflective material (left) results in distorted radiance fields (right).
These distorted SRFs (of 76 shapes in total) were separated from the main
classes in SPARF.

###  C.2 Effect of Dataset Size

We study the effect of increasing the dataset size (Whole SRFs and Partial
SRFs) on the generalization performance of SuRFNet in Figure 18,17. It shows
that as the dataset size increase ( normalized the number of shapes in each
class ), the generalization performance increase. This scalability effect
underlines the importance of SPARF. However, as can be seen from these two
figures, partial SRFs scalability is more important than increasing whole
SRFs, which justifies collecting 4 variants per resolution ( as detailed in
Table 5). This observation aligns with previous generative models in the
literature [19, 65, 20]

Figure 17: Scaling-Up Training on SRFs: Partial SRFs. As the training data
(partial SRFs) of radiance fields increase, the generalization improves, as
can be seen in the car class here. The 3-view and 1-view metrics are reported
with test and OOD metrics.  Figure 18: Scaling-Up Training on SRFs: Whole
SRFs. As the training data of radiance fields increase, the generalization
improves across different classes in SPARF.

###  C.3 Loss Ablation Study

Loss Sampling. We study the effect of the sampling strategy with a different
number of input images at test time on the performance of SuRFNet in Table 7.
It shows that using a uniform sampling strategy depletes the learning capacity
of the network and can degrade performance. The effect is more evident when
the number of views is one, where the partial SRFs are more sparse and the
training is delicate.

| 1-view |  | 3-view
---|---|---|---
Strategy | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
uniform | 20.03 | 0.94 | 0.10 |  | 21.00 | 0.94 | 0.09
Q-Gaus. | 20.55 | 0.93 | 0.09 |  | 21.83 | 0.94 | 0.08

Table 7: Effect of Loss Sampling Strategy.We study the effect of Loss sampling
strategy (uniform _vs_. Q-Gaussian) on airplane class.

Loss Hyperparameters. For the density threshold $\alpha_{\text{dense}}$
defined in Eq 2 and 3, the validation accuracies of SuRFNet on car class are
13, 14.7, 67.8, 67, 67, 67.2, 62.5 for the values of -0.01, -0.001, 0, 0.001,
0.003, 0.01, 0.03 of $\alpha_{\text{dense}}$ respectively. The hyperparameter
$\sigma$ which governs the spread of the Q-Gaussian loss is studied as
follows. the validation accuracies of SuRFNet on airplane class are 52, 70.4,
71.7, 72.1, 72.2, 72.4, 72.3 for the values of 0.1, 0.2, 0.3, 0.4, 0.6, 0.8,
and 1.0 of $\sigma$ respectively. The number of coordinates $\mathbf{c}$
sampled in the Q-Gaussian loss is proportional to the number of coordinates in
the input SRFs with multiplier $K=40$. For different values of this multiplier
1, 5, 10, 20, 40, 80, 200, the validation accuracies of SuRFNet trained on
airplane class are 72.2, 72.7, 72.9, 72.5, 71.8, 71.7, and 71.6 respectively.

###  C.4 Faulty Textures

In some rare instance of the shapes in ShapeNet [5], some objects have doubled
textures in some areas. This occurs in less than 1% of the data and leads the
renderer to render the background instead in these areas (highlighted with
green). See Figure 19 for examples of these cases.

Figure 19: Rare Cases of Faulty Textures. Some objects in ShapeNet [5] have
doubled textrures in some parts, leading to faulty renderings.

output w/o $L_{R}$ | output w/ $L_{R}$ | whole SRF
---|---|---
|  |

Figure 20: Effect of the Perceptual Loss $L_{R}$. Adding a perceptual loss on
volume-rendered images during training SuRFNet insures the rendered images
remain closer to how they should be rendered, as the 3D radiance colors
supervision won’t guarantee the rendering quality. (left): without perceptual
loss , (middle): with the loss.

###  C.5 The Irregularity of SRFs

The optimized whole SRFs used in our training are irregular 3D data
structures. They hold many non-empty voxels that contain low-density radiance
information that does not affect rendering. As can be seen in Figure 21, the
non-empty and low-density components do not affect rendering but contain
radiance information ( _e.g_. black albedo) that affects the 3D learning
pipeline. This motivates the use of the specialized losses proposed in this
work, in order to tackle these challenges associated with SRFs.

|  |
---|---|---
|  |

Figure 21: The Irregularity of SRFs. The optimized SRFs used in our training
are irregular 3D data structures. they include non-empty voxels that contain
low-density radiance information that does not affect rendering. (top):
renderings of a whole SRF, (bottom): renderings of the same SRF when
densifying non-empty voxels.

##  Appendix D Additional Results

Additional results of normal test tracks benchmark of SPARF are presented in
Table 6. Please see figures 25 and 26 for differences between the normal
train/test track and the OOD hard track .More comparisons and generations are
provided in Figures 30,31,29. Regarding real images, we ahow more Co3D images
and their SRFs and PixelNeRF reconstruction in Figure 32. Note that the goal
of this figure is to demonstrate the quality of the renderings form proper
SRFs (similar ot the ones used in training SuRFNet), compared to a 2D-based
network ( like PixelNeRF [75]). It is not used to evaluate generalization
ability from few input views on real images, but to show the potential of
training on real images.

one important aspect to consider is the 3D consistency of our SuRFNet
renderings compared to the other 2D methods, especially when moving out-of-
distribution of the training views. This is one of the most important aspects
we investigate in our work that previous works in the literature have
overlooked. Figure 22 demonstrates that as the testing rendered views move
outside of the training distribution (going right), SuRFNet generally produces
more consistent renderings than all previous 2D-based methods [[75, 34]].

$\xrightarrow{\quad\quad\text{Going out-of-distribution of training views
}\quad\quad}$
---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 22: Going Out-of-distribution of Training Views. Renderings are shown
of ground truth using whole SRFs (first row), SurFNet [ours] (second row),
VisionNeRF [34] (third row), and PixelNeRF [3] (bottom row). As rendered views
move outside of the training distribution (going right), SuRFNet generally
produces more 3D-consistent renderings than previous 2D-based methods.

SPARF (ours) | SRN [59] | NMR [49]
---|---|---
|  |
|  |
|  |
|  |

Figure 23: SPARF _vs_. other Datasets . SPARF offers a large-scale high-
resolution dataset compared to other posed multi-view datasets. Note that SRN
[59] has only cars and chairs, while NMR [49] and SPARF has 13 classes.

Figure 24: SPARF: a Large Dataset for 3D Shapes Radiance Fields and Novel
Views Synthesis.

Train | Test | Hard Test (OOD)
---|---|---
|  |

Figure 25: Cameras Setups for Different SPARF Splits. Here, we show different
visualizations of the camera setups of the three splits of SPARF. (Train): 400
determinsitic spherical views, (Test): 20 random spherical views, (hard OOD
Test): 10 random views.

Train/Test Track
---
|  |  |  |  |  |
OOD Hard Test Track
|  |  |  |  |  |
Train/Test Track
|  |  |  |  |  |
OOD Hard Test Track
|  |  |  |  |  |

Figure 26: SPARF Splits. SPARF has three main splits for every 3D shape:
training views (400 views), test views (20 views), and OOD “hard" views (10
views) as can be shown in the examples above.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 27: SRFs: The optimized Sparse Radiance Fields in SPARF 1. A total of
one million SRFs have been collected in SPARF, including on multiple voxel
resolutions: 32 (top), 128 (middle), and 512 (bottom) for every 3D shape.

|  |  |  |  |
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 28: SRFs: The optimized Sparse Radiance Fields in SPARF 2. A total of
one million SRFs have been collected in SPARF, including on multiple voxel
resolutions: 32 (top), 128 (middle), and 512 (bottom) for every 3D shape.

Ground Truth Whole SRFs | Generated SRFs ($512^{3}$ resolution)
---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 29: SuRFNet: Generating High-Resolution Radiance Fields. We show some
volume-rendered sequences based on our SuRFNet voxel radiance field outputs
(512 resolution), given only 3 images of each shape. This demonstrates the
capability of SuRFNet to generate high-resolution sparse voxel SRFs. Note
that, here, SURFNet is overfitting on a small dataset in these examples and is
not meant for shape generalization.

Input View | PixelNeRF [75] | VisionNeRF [34] | SuRFNet (ours) | Ground Truth
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |

Figure 30: Qualititve Comparisons 1. We show different render from our SuRFNet
outputs generated from a single image compared to other methods (pixel-Nerf
[75], and VisionNerf [34] ) and whole SRF ”GT" renderings. Note that the
predicted views are outside the training views distribution (zoomed in
randomly). This test highlights the weakness of the 2D-based baselines [75,
34] outside the training track, while our 3D approach maintains multi-view
consistency everywhere.

Input Views | PixelNeRF [75] | VisionNeRF [34] | SuRFNet (ours) | Ground Truth
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |

Figure 31: Qualititve Comparisons 2. We show different render from our SuRFNet
outputs generated from 3 input images compared to other methods (pixel-Nerf
[75], and VisionNerf [34] ) and whole SRF ”GT" renderings. Note that the
predicted views are outside the training views distribution (zoomed in
randomly). This test highlights the weakness of the 2D-based baselines [75,
34] outside the training track, while our 3D approach maintains multi-view
consistency everywhere.

Real | Generated
---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

Figure 32: Real images We show real images of Co3D [55] (left) and the
corresponding generated views from our SRFs (rows’ top part) and pretrained
PixelNeRF [75] (rows’ bottom part).