# Adaptive Shells for Efficient Neural Radiance Field Rendering

Zian Wang  0000-0003-4166-3807 NVIDIA, University of Toronto, Vector
InstituteTorontoCanada<EMAIL_ADDRESS>,  Tianchang Shen  0000-0002-7133-2761
NVIDIA, University of Toronto, Vector InstituteTorontoCanada<EMAIL_ADDRESS>,  Merlin Nimier-David  0000-0002-6234-3143 NVIDIAZürichSwitzerland
<EMAIL_ADDRESS>,  Nicholas Sharp  0000-0002-2130-3735
NVIDIASeattleUSA<EMAIL_ADDRESS>,  Jun Gao  0000-0002-3521-0417 NVIDIA,
University of Toronto, Vector InstituteTorontoCanada<EMAIL_ADDRESS>,
Alexander Keller  0000-0002-9144-5982 NVIDIABerlinGermany<EMAIL_ADDRESS>,
Sanja Fidler  0000-0003-1040-3260 NVIDIA, University of Toronto, Vector
InstituteTorontoCanada<EMAIL_ADDRESS>,  Thomas Müller
0000-0001-7577-755X NVIDIAZürichSwitzerland<EMAIL_ADDRESS>and  Zan
Gojcic  0000-0001-6392-2158 NVIDIAZürichSwitzerland<EMAIL_ADDRESS>
###### Abstract.

Neural radiance fields achieve unprecedented quality for novel view synthesis,
but their volumetric formulation remains expensive, requiring a huge number of
samples to render high-resolution images. Volumetric encodings are essential
to represent fuzzy geometry such as foliage and hair, and they are well-suited
for stochastic optimization. Yet, many scenes ultimately consist largely of
solid surfaces which can be accurately rendered by a single sample per pixel.
Based on this insight, we propose a neural radiance formulation that smoothly
transitions between volumetric- and surface-based rendering, greatly
accelerating rendering speed and even improving visual fidelity. Our method
constructs an explicit mesh envelope which spatially bounds a neural
volumetric representation. In solid regions, the envelope nearly converges to
a surface and can often be rendered with a single sample. To this end, we
generalize the NeuS (Wang et al., 2021) formulation with a learned spatially-
varying kernel size which encodes the spread of the density, fitting a wide
kernel to volume-like regions and a tight kernel to surface-like regions. We
then extract an explicit mesh of a narrow band around the surface, with width
determined by the kernel size, and fine-tune the radiance field within this
band. At inference time, we cast rays against the mesh and evaluate the
radiance field only within the enclosed region, greatly reducing the number of
samples required. Experiments show that our approach enables efficient
rendering at very high fidelity. We also demonstrate that the extracted
envelope enables downstream applications such as animation and simulation.

Neural Radiance Fields, Fast Rendering, Level Set Methods, Novel View
Synthesis

††submissionid: 774††submissionid: 774††ccs: Computing methodologies
Rendering††ccs: Computing methodologies Shape representations††ccs: Computing
methodologies Reconstruction Figure 1.  This work presents an approach for
efficiently rendering neural radiance fields by restricting volumetric
rendering to a narrow band around the object. _Left:_ We first fit a dense
neural volume using a new spatially-varying kernel that automatically adapts
to be large in volumetric regions such as hair or grass, and small in sharp-
surface regions such as skin or furniture. We then extract an explicit
bounding mesh of the region to be rendered whose width is determined by the
kernel, and render at real-time rates. _Right:_ the proposed method is general
and effective across a wide range of data and well-suited for downstream
applications such as simulation and animation. The face model of the Khady
synthetic human shown _left_ is courtesy of texturing.xyz.

##  1\. Introduction

Neural radiance fields, which we will refer to as _NeRFs_ , have recently
emerged as a powerful 3D representation enabling photorealistic novel-view
synthesis and reconstruction. Unlike traditional explicit methods for novel-
view synthesis, NeRFs forego high-quality mesh reconstruction and explicit
surface geometry in favor of neural networks, which encode the volumetric
density and appearance of a scene as a function of 3D spatial coordinates and
viewing direction. However, the high visual fidelity of NeRFs comes at a great
computational cost, as the volume rendering formulation requires a large
number of samples along each ray and ultimately prevents real-time synthesis
of high-resolution novel views. In tandem, explicit reconstruction and novel-
view synthesis have continued to make great progress by leveraging advances in
inverse rendering and data-driven priors, but a fidelity gap remains. The goal
of this work is to close this gap by developing a neural volumetric
formulation that leverages explicit geometry to accelerate performance,
without sacrificing quality.

Much recent and concurrent work has likewise sought to improve the efficiency
of NeRF representations and volume rendering. An important step towards this
goal was the evolution from a global large multi-layer perceptron (MLP)
representation (Mildenhall et al., 2020) to local sparse feature fields
combined with shallow MLP decoders (Sara Fridovich-Keil and Alex Yu et al.,
2022; Müller et al., 2022). This resulted in several orders-of-magnitude
speed-ups. Complementary research to improve the efficiency of NeRFs proposed
replacing the neural networks by simpler functions such as spherical
harmonics, or baking the volumetric representation onto proxy geometry that
accelerates rendering (Yariv et al., 2023; Chen et al., 2023). The latter
formulation enables especially large speedups and facilitates real-time
rendering even on commodity devices (Chen et al., 2023). Yet, doing so
compromises the quality as the scene content is projected onto proxy geometry.

In this work, we instead aim to make NeRF rendering more efficient while
maintaining or even improving the perceptual quality. To this end, we propose
a narrow-band rendering formulation that enables efficient novel-view
synthesis, while enjoying the desirable properties of the volumetric
representation (Figure 1 _left_). Our method is inspired by the insight that
different regions of the scene benefit from different styles of rendering.
Indeed, fuzzy surfaces with intricate geometry and complex transparency
patterns benefit greatly from exhaustive volume rendering, while conversely,
opaque smooth surfaces can be well—or potentially even better—represented by a
single sample where the ray intersects the surface. This observation allows us
to better distribute the computational cost across the rays by assigning as
many samples as needed to faithfully represent the ground-truth appearance.

Figure 2.  One state-of-the-art approach to accelerate volumetric rendering is
to skip empty voxels, however this still requires multiple samples within
occupied voxels (_left_). Our approach extracts a narrow band mesh, for which
a single sample at the midpoint is a very good approximation of the surface
(_right_).

With the introduction of auxiliary acceleration data structures that promote
empty space skipping (Müller et al., 2022), NeRFs can already render images
with a varying number of samples per ray. Still, there remain many challenges
that prevent the current formulations from efficiently adapting to the local
complexity of the scene (Figure 2). First, the memory footprint of grid-based
acceleration structures scales poorly with resolution. Second, the smooth
inductive bias of MLPs hinders learning a sharp impulse or step function for
volume density, and even if such an impulse was learned it would be difficult
to sample it efficiently. Finally, due to the lack of constraints, the
implicit volume density field fails to accurately represent the underlying
surfaces (Wang et al., 2021), which often limits their application in
downstream tasks that rely on mesh extraction.

To remedy the last point, (Wang et al., 2021; Yariv et al., 2021; Wang et al.,
2022a) propose to optimize a signed distance function (SDF) along with a
kernel size encoding the spread of the density, rather than optimizing density
directly. While this is effective for improving surface representations, the
use of a global kernel size contradicts the observation that different regions
of the scene demand adaptive treatment.

To address the above challenges, we propose a new volumetric neural radiance
field representation. In particular: i) We generalize the NeuS (Wang et al.,
2021) formulation with a _spatially-varying_ kernel width that converges to a
wide kernel for fuzzy surfaces, while collapsing to an impulse function for
solid opaque surfaces without additional supervision. This improvement alone
results in an increased rendering quality across all scenes in our
experiments. ii) We use the learned spatially-varying kernel width to extract
a mesh envelope of a narrow band around the surface. The width of the
extracted envelope adapts itself to the complexity of the scene and serves as
an efficient auxiliary acceleration data structure. iii) At inference time, we
cast rays against the envelope in order to skip empty space and sample the
radiance field only in regions which contribute significantly to the
rendering. In surface-like regions, the narrow band enables rendering from a
single sample, while progressing to a wider kernel and local volumetric
rendering for fuzzy surfaces.

The experiments of Section 4 validate the effectiveness of our formulation
across several data sets. In addition, the applications of Section 5
demonstrate the benefits of our representation.

Figure 3.  Overview of the proposed approach. We demonstrate high-fidelity,
efficient neural implicit scene reconstruction by efficiently-sampling
volumetric rendering inside of an explicit thin shell, which is automatically
fit from visual objectives.

##  2\. Related Work

Synthesizing novel views from a set of images is a longstanding problem in the
fields of computer vision and graphics. The classical approaches to novel-view
synthesis can be roughly categorized based on the coverage density of the
input images. In particular, light field interpolation methods (Levoy and
Hanrahan, 1996; Gortler et al., 1996; Davis et al., 2012) assume that the
input views are sampled densely and close to the target view. When the input
views are sparse, classical methods usually follow a two-stage approach: In
the first stage, they construct a proxy geometry from the images using a
combination of a multi-view stereo pipeline (Schönberger and Frahm, 2016;
Schönberger et al., 2016) and point cloud reconstruction methods (Kazhdan et
al., 2006; Kazhdan and Hoppe, 2013). In the second stage, the input images are
then unprojected onto the geometry either directly in terms of RGB colors
(Debevec et al., 1996; Wood et al., 2000; Buehler et al., 2001; Waechter et
al., 2014) or, more recently, latent features (Riegler and Koltun, 2020,
2021). Other lines of research have developed specialized methods for certain
classes of objects, such as faces (_e.g._ Bi et al. (2021))—although we show
results on synthetic human and animal data (Figure 1), the approach presented
here is entirely general.

#### Neural Radiance Fields (NeRFs)

NeRF (Mildenhall et al., 2020) have revolutionized the prevailing paradigm of
novel-view synthesis, by using a neural network to represent the scenes as a
volumetric (radiance) field that may be queried at any location to return the
view-dependent radiance and volume density. Mildenhall et al. (2020)
synthesize novel views by querying the radiance field along the image rays and
accumulating the appearance using volume rendering. The photorealistic quality
of NeRF has inspired a large body of follow-up work. NeRF++ (Zhang et al.,
2020) analyzed the difficulties of NeRF to represent unbounded scenes and
proposed a background formulation based on the inverted sphere representation.
MipNeRF (Barron et al., 2021) addressed the aliasing effects with an
integrated positional encoding. This work was later extended to unbounded
scenes (Barron et al., 2022) by contracting the volume and using an additional
proposal network. (Deng et al., 2022) and (Niemeyer et al., 2022) tackled the
challenging setting with sparse input views and proposed to regularize the
volumetric representation using depth supervision or smoothness constraints
and data priors based on normalizing flows, respectively. NeRF-W (Martin-
Brualla et al., 2021) has shown how NeRF can be extended to unstructured
collections of images captured in-the-wild, by using per-frame learnable
latent codes to compensate for appearance differences and a transient
embedding to remove dynamic objects. Alternative representations to neural
fields include point clouds (Rückert et al., 2021; Kopanas et al., 2021),
spheres (Lassner and Zollhöfer, 2021), and 3D Gaussians (Kerbl et al., 2023).

#### Implicit surface representation

The NeRF formulation has two main shortcomings when it comes to modeling
surfaces: i) Besides a lack of regularization of the density field, ii)
surface extraction has to be performed at an arbitrary level-set of the
density field. In combination, these lead to noisy and low-fidelity surface
reconstruction. However, with small changes in the formulation, implicit
representations combined with volume rendering (Yariv et al., 2020; Zhang et
al., 2021; Wang et al., 2021; Yariv et al., 2021; Oechsle et al., 2021; Wang
et al., 2022b) still appear as a promising alternative to classical surface
reconstruction approaches from image data (Schönberger et al., 2016). For
example, instead of directly optimizing the density field, (Wang et al., 2021;
Yariv et al., 2021) proposed to decompose it into an SDF and a global kernel
size that defines the spread of the density. This allows for extracting
accurate surfaces from the zero-level set of the SDF, which can also be
regularized using the Eikonal constraint. Similar to NeRFs, implicit surface
representations were also combined with local feature fields and auxiliary
acceleration data structures (Wang et al., 2022a; Zhao et al., 2022; Li et
al., 2023; Rosu and Behnke, 2023; Tang et al., 2023) with the goal of improved
efficiency and representation capacity. While our method is built on the NeuS
(Wang et al., 2021) formulation, our main goal is not to improve the accuracy
of the extracted surface. Instead, we utilize the SDF to extract a narrow
shell that allows us to adapt the representation to the local complexity of
the scene and in turn to accelerate rendering.

#### Accelerating neural volume rendering

One of the main limitations of NeRFs is the computational complexity of neural
volume rendering which slows down both training and inference. Recently,
various different directions to accelerate NeRFs have been explored. For
example, replacing a global MLP with a (sparse) local feature field combined
with a shallow MLP (Liu et al., 2020; Yu et al., 2021; Müller et al., 2022;
Sun et al., 2022; Chen et al., 2022) or the spherical harmonics embedding
(Sara Fridovich-Keil and Alex Yu et al., 2022; Karnewar et al., 2022; Chen et
al., 2022), partitioning the volume into a large number of local (shallow)
MLPs (Reiser et al., 2021; Rebain et al., 2020), or using efficient sampling
strategies (Neff et al., 2021; Hu et al., 2022; Lin et al., 2022; Kurz et al.,
2022), or image-space convolutions (Cao et al., 2022; Wan et al., 2023).
However, even the most optimized volumetric representations (Müller et al.,
2022) are still much slower than pure surface-based approaches such as
NvDiffRec (Munkberg et al., 2022).

To further increase the efficiency of the inference phase, volumetric
representations can be baked onto a proxy surface geometry (Chen et al., 2023;
Yariv et al., 2023; Wan et al., 2023) that can be efficiently rendered using
high-performance rasterization pipelines. An alternative _”baking”_ strategy
is to precompute the outputs of the neural network and store them on a
(sparse) discrete grid that acts as a lookup during inference (Hedman et al.,
2021; Reiser et al., 2023). In this work, we investigate an alternate approach
to speeding up the (volumetric) rendering, by adapting the number of samples
required to render each pixel to the underlying local complexity of the scene.
Note that our formulation is complementary to the _”baking”_ approaches and we
consider the combination of both an interesting avenue for future research.

##  3\. Method

Our method (see Figure 3) builds on NeRF (Mildenhall et al., 2020) and NeuS
(Wang et al., 2021). Specifically, we generalize NeuS (Wang et al., 2021) with
a new spatially-varying kernel (Section 3.2), which improves the quality and
guides the extraction of a narrow-band shell (Section 3.3). Then, the neural
representation is fine-tuned (Section 3.5) within the shell that significantly
accelerates rendering (Section 3.4).

###  3.1. Preliminaries

NeRF (Mildenhall et al., 2020) represents a scene as a volumetric radiance
field that maps a 3D point $\mathbf{x}\in\mathbb{R}^{3}$ and a viewing
direction $\mathbf{d}\in\mathbb{R}^{3}$ to the volume density $\sigma$ and the
emitted view-dependent color $\mathbf{c}\in\mathbb{R}^{3}$. This volumetric
field is represented by a neural network $\textrm{NN}_{\theta}(\cdot)$ with
parameters $\theta$, such that
$(\mathbf{c},\sigma)=\textrm{NN}_{\theta}(\mathbf{x},\mathbf{d})$. The scene
can then be rendered along a ray $\mathbf{r}=\mathbf{o}+\tau\mathbf{d}$ with
origin $\mathbf{o}\in\mathbb{R}^{3}$ and direction
$\mathbf{d}\in\mathbb{R}^{3}$ from $\tau_{n}$ to $\tau_{f}$ via standard
volumetric rendering

(1)
$\mathbf{c}(\mathbf{r})=\int_{\tau_{n}}^{\tau_{f}}\exp\Big{[}\int_{\tau_{n}}^{\tau}-\sigma(\mathbf{r}(z))dz\Big{]}\sigma(\mathbf{r}(\tau))\mathbf{c}(\mathbf{r}(\tau),\mathbf{d})d\tau\,,$

which is approximated by numerical integration

(2)
$\mathbf{c}(\mathbf{r})=\sum_{i=1}^{N_{r}}\exp\Big{[}-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\Big{]}(1-\exp(-\sigma_{i}\delta_{i}))\mathbf{c}(\mathbf{r},\mathbf{d})_{i}\,,$

where $N_{r}$ denotes the number of samples along the ray $\mathbf{r}$ and
$\delta_{i}$ is the distance between two adjacent samples.

To improve geometric surface quality in NeRF-like scene reconstructions, NeuS
(Wang et al., 2021) and VolSDF (Yariv et al., 2021) propose to replace the
learned density $\sigma$ by a learned signed distance field $f$, and then
transform $f$ to $\sigma$ for rendering via a sigmoid-shaped map. The
formulation of NeuS optimizes an SDF
$(\mathbf{c},f)=\textrm{NN}_{\theta}(\mathbf{x},\mathbf{d})$ along with a
global kernel size $s$ that controls the sharpness of the implied density. To
evaluate volume rendering (Equation 2) the SDF value $f$ at $\mathbf{x}$ is
transformed to a density $\sigma$ by

(3)
$\sigma=\max\left(-\frac{\frac{d\Phi_{s}}{d\tau}(f)}{\Phi_{s}(f)},0\right),\qquad\Phi_{s}(f)=(1+\exp(-f/s))^{-1},$

where $f$ is implicitly $f(\mathbf{r}(\tau))$ along a ray. Intuitively, a
small $s$ results in a wide kernel with a fuzzy density, while in the limit
$\lim_{s\to 0}{d\Phi_{s}}/{d\tau}$ approximates a sharp impulse function (see
inset). This SDF-based formulation allows for the use of an Eikonal
regularizer during training, which encourages the learned $f$ to be an actual
distance function, resulting in a more accurate surface reconstruction. The
relevant losses are discussed in Section 3.5.

###  3.2. Spatially-Varying Kernel Size

The NeuS SDF formulation is highly effective, yet, it relies on one _global_
kernel size. In combination with the Eikonal regularization this implies a
constant spread of the volume density across the whole scene. However, a one-
size-fits-all approach does not adapt well to scenes that contain a mixture of
“sharp” surfaces (_e.g._ furniture or cars) and “fuzzy” volumetric regions
(_e.g._ hair or grass).

Our first contribution is to augment the NeuS formulation with a spatially-
varying, _locally_ learned kernel size $s$ as an additional neural output that
is dependent on the input 3D position $\mathbf{x}$. The extended network
becomes $(\mathbf{c},f,s)=\textrm{NN}_{\theta}(\mathbf{x},\mathbf{d})$ (see
the implementation details in Section 4.1). During training, we additionally
include a regularizer that promotes the smoothness of the kernel size field
(Section 3.5). This neural field can still be fit from only color image
supervision, and the resulting spatially-varying kernel size automatically
adapts to the sharpness of the scene content (Figure 7). This enhanced
representation is independently valuable, improving reconstruction quality in
difficult scenes, but importantly it will serve to guide our explicit shell
extraction in Section 3.3, which greatly accelerates rendering.

Figure 4.  After fitting an initial SDF and spatially varying kernel, we apply
level set flows to extract an adaptive shell via dilation and erosion. For
illustrative purposes, the adaptive shell is enlarged; in practice it very
tightly encloses sharp surfaces.

###  3.3. Extracting an Explicit Shell

The adaptive shell delimits the region of space which contributes
significantly to the rendered appearance, and is represented by two explicit
triangle meshes. When $s$ is large the shell is thick, corresponding to
volumetric scene content, and when $s$ is small the shell is thin,
corresponding to surfaces. After the implicit fields $s$ and $f$ have been fit
as described in Section 3.2, we extract this adaptive shell once as a post-
process.

In Equation 3 the magnitude of the quantity $f/s$ in the sigmoid exponent
determines the rendering contribution along a ray (see the inset figure in
Section 3.1). It is tempting to simply extract a band where $|f/s|<\eta$ for
some $\eta$ as the region that makes a significant contribution to the
rendering. However, the learned functions quickly become noisy away from the
$f=0$ level set, and cannot be sufficiently regularized without destroying
fine details. Our solution is to separately extract an inner boundary as an
erosion of the $f=0$ level set, and an outer boundary as its dilation (Figure
4), both implemented via a regularized, constrained level set evolution
tailored to the task.

In detail, we first sample the fields $f$ and $s$ at the vertices of a regular
grid. We then apply a level set evolution to $f$, producing a new eroded field
$\textrm{SDF}_{-}$, and extract the $\textrm{SDF}_{-}=0$ level set as the
inner shell boundary via marching cubes. A separate, similar evolution yields
the dilated field $\textrm{SDF}_{+}$, and the $\textrm{SDF}_{+}=0$ level set
forms the outer shell boundary. We define both level sets separately: the
dilated outer surface should be smooth to avoid visible boundary artifacts,
while the eroded inner surface needs not be smooth, but must only exclude
regions which certainly do not contribute to the rendered appearance.

Recall that a basic level set evolution of a field $a$ is given by ${\partial
a}/{\partial t}=-\left|\nabla a\right|v$, where $v$ is the desired scalar
outward-normal velocity of the level set. Our constrained, regularized flow on
$f$ is then

(4) $\frac{\partial f}{\partial t}=-\left|\nabla
f\right|\bigg{(}v(f_{0},s)+\lambda_{\textrm{curv}}\nabla\cdot\dfrac{\nabla
f}{|\nabla f|}\bigg{)}\omega(f),$

where $f_{0}$ here denotes the initial learned SDF, the divergence term is a
curvature smoothness regularizer with weight $\lambda_{\textrm{curv}}$. The
soft falloff $\omega$ (see inset) limits the flow to a window around the level
set:

(5) $\omega(f)=\tfrac{1}{2}\big{(}1+\cos(\pi\
\textrm{clamp}(f/\zeta,-1.,1.)\big{)},$

with window width $\zeta$. To dilate the level set, the velocity is chosen to
fill all regions with density $\sigma>\sigma_{\textrm{min}}$ for a ray
incoming in the normal direction

(6)
$v_{\textrm{dilate}}(f_{0},s)=\begin{cases}\beta_{d}\sigma(f_{0},s)&\sigma(f_{0},s)>\sigma_{\textrm{min}}\\\
0&\sigma(f_{0},s)\leq\sigma_{\textrm{min}}\end{cases},$

with $\beta_{d}$ as a scaling coefficient. We use $\zeta=0.1$, and
$\lambda_{\textrm{curv}}=0.01$. To erode the level set, the velocity is
inversely-proportional to density, so that the shell expands inward quickly
for low density regions and slowly for high density regions

(7)
$v_{\textrm{erode}}(f_{0},s)=\min\big{(}v_{\textrm{max}},\beta_{e}\dfrac{1}{\sigma(f_{0},s)}\big{)},$

where here we use $\zeta=0.05$, and $\lambda_{\textrm{curv}}=0$. These
velocities lead to a short-distance flow, and thus a narrow shell where $s$ is
small and the content is surface-like. They lead to a long-distance flow and
hence a wide shell where $s$ is large and the content is volume-like.

Figure 5.  Given kernel size $s$ and SDF $f$ learned from Section 3.2 (_top_),
we apply a erosion and dilation flows to $f$ (_bottom middle and left_) to
extract a narrow shell in which we efficiently render (_bottom right_). Here,
we visualize each quantity on a 2D slice through a scene. For clarity, we show
the fields only nearby the adaptive shell that is ultimately rendered.

We compute the dilated field $\textrm{SDF}_{+}$ and eroded field
$\textrm{SDF}_{-}$ respectively by forward-Euler integrating this flow on the
grid for $50$ steps of integration, computing derivatives via spatial finite
differences. We do not find numerical redistancing to be necessary. Finally,
we clamp the results $\textrm{SDF}_{-}\leftarrow$
$\max(f_{0},\textrm{SDF}_{-})$ and $\textrm{SDF}_{+}\leftarrow$
$\min(f_{0},\textrm{SDF}_{+})$, to ensure that the eroded field only shrinks
the level set, and the dilated flow only grows the level set. The
$\textrm{SDF}_{+}=0$ and $\textrm{SDF}_{-}=0$ level sets are extracted via
marching cubes as the outer and inner shell boundary meshes $\textsf{M}_{+}$
and $\textsf{M}_{-}$, respectively. Figure 5 visualizes the resulting fields.
Further details are provided in Procedure Appendix and Appendix of the
Appendix.

###  3.4. Narrow-Band Rendering

The extracted adaptive shell serves as an auxiliary acceleration data
structure to guide the sampling of points along a ray (Equation 2), enabling
us to efficiently skip empty space and sample points only where necessary for
high perceptual quality. For each ray, we use hardware-accelerated ray tracing
to efficiently enumerate the ordered intervals defined by the intersection of
the ray and the

adaptive shell. Within each interval we query equally-spaced samples. Our
renderer does not require any dynamic adaptive sampling or sample-dependent
termination criteria, which facilitates high-performance parallel evaluation.

In detail, we first build ray tracing acceleration data structures for both
the outer mesh $\textsf{M}_{+}$ and inner mesh $\textsf{M}_{-}$ We then cast
each ray against the meshes, yielding a series of intersections where the ray
enters or exits the mesh, partitioning the ray into zero or more intervals
contained in the shell (see inset). For each interval with width $w$, a target
inter-sample spacing $\delta_{s}$, and a single-sample threshold
$w_{\textrm{s}}$, we compute the number of samples as
$\mathrm{ceil}(\mathrm{max}(w-w_{\textrm{s}},0)/\delta_{s})+1$. We cap the
maximum number of samples to $N_{\textrm{max}}$, and equidistantly sample the
interval. Note that if the interval has $w<w_{\textrm{s}}$, a single sample is
taken at the center of the interval. When an interval ends because the ray
hits the inner mesh $\textsf{M}_{-}$, we do not process any subsequent
samples, as this represents the interior of a solid object. Otherwise, we
process intervals until the ray exits the scene or we hit a maximum cap,
accumulating the contributions as in Equation 2.

Note that this procedure can be implemented by first generating all samples
within all intervals, and then performing a single batched MLP inference pass,
which improves throughput. For surfaces, our narrow-band sampling often
amounts to just a hardware-accelerated ray tracing, followed by a single
network evaluation, while for fuzzy regions it densely samples only where
necessary—in either case, performance is greatly accelerated (Table 1). More
algorithmic details are included in Procedure Appendix of the Appendix.

Figure 6. Pure surface-based representations struggle to represent fuzzy
surfaces such as the tail. On the other hand, our method adapts the narrow
shell to the local complexity of the scene, using a single sample  for the
sharp skin surface and up to 16 samples  for the tail.

###  3.5. Losses and Training

We optimize the parameters of our representation in two stages. In the first
stage, we use the fully volumetric formulation described in Sections 3.1 and
3.2 and minimize the following objective

(8)
$\mathcal{L}=\mathcal{L}_{\mathbf{c}}+\lambda_{e}\mathcal{L}_{e}+\lambda_{s}\mathcal{L}_{s}+\lambda_{\mathbf{n}}\mathcal{L}_{\mathbf{n}}$

with the weights $\lambda_{\mathbf{c}}=1$, $\lambda_{e}=0.1$,
$\lambda_{\mathbf{n}}=0.1$, and $\lambda_{s}=0.01$ for all experiments. Here
$\mathcal{L}_{\mathbf{c}}$ is the standard pixel-wise color loss against
calibrated ground-truth images

(9)
$\mathcal{L}_{\mathbf{c}}=\frac{1}{|\mathcal{R}|}\sum_{\mathbf{r}\in\mathcal{R}}|\mathbf{c}(\mathbf{r})-\mathbf{c}_{\textrm{gt}}(\mathbf{r})|$

and $\mathcal{L}_{e}$ is the Eikonal regularizer as in (Wang et al., 2021)

(10)
$\mathcal{L}_{e}=\frac{1}{|\mathcal{X}|}\sum_{\mathbf{x}\in\mathcal{X}}\left(||\nabla
f(\mathbf{x})||_{2}-1\right)^{2},$

where $\mathcal{R}$ and $\mathcal{X}$ denote the set of rays and samples along
the rays, respectively. $\nabla f(\mathbf{x})$ can be obtained either
analytically (Wang et al., 2021; Yariv et al., 2021; Wang et al., 2022a) or
through finite differences (Li et al., 2023; Wang et al., 2023). We use the
latter approach. The loss $\mathcal{L}_{s}$ regularizes the spatially varying
kernel size introduced in our formulation for smoothness

(11)
$\mathcal{L}_{s}=\frac{1}{\mathcal{X}}\sum_{\mathbf{x}\in\mathcal{X}}||\log\big{[}s(\mathbf{x})\big{]}-\log\big{[}s(\mathbf{x}+\mathcal{N}(0,\varepsilon^{2}))\big{]}||_{2},$

where $\mathcal{N}(0,\varepsilon^{2})$ denotes samples from the normal
distribution with standard deviation $\varepsilon$. Lastly, we incorporate the
loss

(12)
$\mathcal{L}_{\mathbf{n}}=\frac{1}{|\mathcal{X}|}\big{|}\big{|}\mathbf{n}(\mathbf{x})-\frac{\nabla
f(\mathbf{x})}{||\nabla f(\mathbf{x})||_{2}}\big{|}\big{|}_{2},$

internal to our network architecture (Section 4.1). Like NeuS, we will
leverage geometric normals as an input to a shading subnetwork, but we find
that predicting these normals internally improves inference performance _vs._
gradient evaluation. $\mathcal{L}_{\mathbf{n}}$ serves to train these
predicted normals to remain approximately faithful to ones obtained through
the finite differences of the underlying SDF field $\nabla f(\mathbf{x})$
(Wang et al., 2023; Li et al., 2023).

Figure 7.  The original NeuS (Wang et al., 2021) formulation uses a single
global kernel size $s$. On complex scenes with varying content, the global $s$
value converges to an average which is too small for volumetric parts  , and
too large for sharp surfaces  . Instead, our locally varying kernel size
adapts to the scene, in-turn allowing us to reduce the number of samples to a
single sample  for sharp surfaces and up to 32 samples  for the fern (_top
right_). NeuS uses a constant 384 samples per pixel (_bottom right_).

After the implicit field has been fit, we extract the adaptive shell as in
Section 3.3. While initial training requires dense sampling along rays, our
explicit shell now allows narrow-band rendering to concentrate samples only in
significant regions. We therefore fine-tune the representation within the
narrow band, now with only $\mathcal{L}_{\mathbf{c}}$—it is no longer
necessary to encourage a geometrically-nice representation, as we have already
extracted the shell and restricted the sampling to a small band around the
scene content. Disabling regularization enables the network to devote its
whole capacity to fit the visual appearance, which leads to improved visual
fidelity (Table 4). In Procedure Appendix of the Appendix, we also present the
training pipeline with algorithm details.

##  4\. Experiments

In this section, we provide low-level details of our implementation and
evaluate our method in terms of rendering quality and efficiency on four data
sets that range from synthetic object-level _Shelly_ , _NeRFSynthetic_
(Mildenhall et al., 2020) and tabletop _DTU_ (Jensen et al., 2014) data, to
real-world large outbound scenes _MipNeRF360_ (Barron et al., 2022). For
comparisons, we treat Instant NGP (Müller et al., 2022) as our volumetric
baseline, due to its balance between high fidelity and efficiency. In
addition, we compare to prior methods that were optimized either for fidelity
(Yuan et al., 2022; Wang et al., 2021; Barron et al., 2021, 2022) or rendering
efficiency (Chen et al., 2023; Yariv et al., 2023; Guo et al., 2023).

When running NeRF (Mildenhall et al., 2020) and Mip-NeRF (Barron et al., 2021)
on _DTU_ and _Shelly_ , we use the implementation from Nerfstudio (Tancik et
al., 2023). For other methods and experiment settings, we use their official
implementations.

###  4.1. Architecture Details

Following the state of the art in neural volumetric rendering, we represent
our neural field as a combination of a feature field and a small (decoder)
neural network. Specifically, we use a multi-resolution hash encoding (Müller
et al., 2022) $\Psi(\cdot)$ with 14 levels, where each level is represented by
a hash-table with $2^{22}$ two-dimensional features. The voxel grid resolution
of our feature field grows from $16^{3}\to 4096^{3}$ for _Shelly_ and
_NeRFSynthetic_ , and from $16^{3}\to 8192^{3}$ for the other data sets. The
features at each level are obtained through tri-linear interpolation before
being concatenated to form the feature embedding
$\Psi(\cdot)\in\mathbb{R}^{28}$. This is further concatenated with the sample
coordinates and input to the geometry network
$(f,1/s,\mathbf{f}_{\textrm{geo}},\mathbf{n})=\textrm{NN}^{\textrm{geo}}_{\theta}([\Psi(\mathbf{x}),\mathbf{x}])$
which is an MLP with a single hidden layer and $31\to 64\to 31$ dimensions.
Apart from the SDF value $f$ and kernel size $s$,
$\textrm{NN}^{\textrm{geo}}_{\theta}$ also outputs a geometry latent feature
$\mathbf{f}_{\textrm{geo}}\in\mathbb{R}^{26}$ and the normal vector
$\mathbf{n}\in\mathbb{R}^{3}$ which are combined with $\mathbf{x}$ and the
encoded view direction $\mathbf{d}$ as input to the radiance network
$\mathbf{c}=\textrm{NN}^{\textrm{rad}}_{\theta}([\gamma(\mathbf{d}),\mathbf{f}_{\textrm{geo}},\mathbf{n},\mathbf{x}])$
that predicts the emitted color. Here, $\textrm{NN}^{\textrm{rad}}_{\theta}$
is an MLP with two hidden layers and dimensions $48\to 64\to 64\to 3$. To
reduce the computational cost, we directly predict the normal vector
$\mathbf{n}$ with an MLP rather than computing it as the gradient of the
underlying SDF field as done in NeuS (Wang et al., 2021). Finally, to encode
the input direction $\mathbf{d}$, we use the spherical harmonic basis up to
degree 4, such that $\gamma(\mathbf{d})\in\mathbb{R}^{16}$. The dimensions of
all layers in both networks and the feature field were selected for high
throughput on modern GPU devices.

###  4.2. Implementation

The training stage of our method is implemented in PyTorch (Paszke et al.,
2017), while the inference stage is implemented in Dr.Jit (Jakob et al.,
2022). To achieve real-time inference rates, we rely on the automatic kernel
fusion performed by Dr.Jit as well as GPU-accelerated ray-mesh intersection
provided by OptiX (Parker et al., 2010). While the inference pass is
implemented with high-level Python code, the asynchronous execution of large
fused kernels hides virtually all of the interpreter’s overhead. Combined with
the algorithmic improvements described above, we achieve frame rates from 40
fps (25 $\nicefrac{{\text{ms}}}{{\text{frame}}}$) on complex outdoor scenes to
300 fps (3.33 $\nicefrac{{\text{ms}}}{{\text{frame}}}$) on object-level
scenes, at 1080p resolution on a single RTX 4090 GPU. A performance comparison
to Instant NGP (Müller et al., 2022) on four data sets is given in Table 1.
Note that in this work, we focused on inference performance only, and have not
yet applied these performance optimizations to the training procedure.
Detailed pseudo-code is given in Procedures Appendix, Appendix, Appendix and
Appendix of the Appendix.

Table 1.  Performance comparisons on all four data sets, measured at 1080p without GUI overhead using an RTX 4090 GPU. Our adaptive sample placement and mesh-based empty-space skipping technique allows us to outperform Instant NGP without compromising visual fidelity. _Note that Instant NGP’s performance on the DTU data set was hindered by a large number of background samples, and is therefore not necessarily indicative of a real use case: the user may specify a tighter scene bounding box to focus the samples on the main scene contents._ | Ours |  Instant NGP (Müller et al., 2022)
---|---|---
|  Sample count $\downarrow$ |  $\nicefrac{{\text{ms}}}{{\text{frame}}}$ $\downarrow$ |  FPS $\uparrow$ |  Sample count $\downarrow$ |  $\nicefrac{{\text{ms}}}{{\text{frame}}}$ $\downarrow$ |  FPS $\uparrow$
_Shelly_ | 2.07 | 3.81 | 262.69 | 2.89 | 11.74 | 85.16
_DTU_ | 5.11 | 6.37 | 157.00 | 56.10 | 123.31 | 8.10
_NeRFSynthetic_ | 1.98 | 3.56 | 280.68 | 3.19 | 14.20 | 70.40
_MipNeRF360_ | 17.05 | 27.64 | 36.18 | 45.62 | 93.57 | 10.69

###  4.3. Evaluation Metrics

In order to evaluate the rendering quality, we report the commonly used peak
signal-to-noise ratio (_PSNR_), learned perceptual image patch similarity
(_LPIPS_), and structural similarity (_SSIM_) metrics. Unfortunately,
evaluating the efficiency of the methods is less straightforward as the
complexity of the method is often intertwined with the selected hardware and
low-level implementation details. Indeed, reporting only the number of frames-
per-second (FPS) or the time needed to render a single frame may paint an
incomplete picture. We therefore additionally report the number of samples
along the ray that are required to render each pixel. While the number of
samples along the ray also does not tell the whole story as the per-sample
evaluation can have different computational complexity, combining all metrics
provides a good assessment of the computational complexity of the individual
methods.

Table 2. Quantitative results on _Shelly_ data set, _DTU_ data set and _NeRFSynthetic_ data set. We report PSNR, LPIPS and SSIM. Our method achieves better results across all metrics on _Shelly_ and _DTU_ and comparable results on _NeRFSynthetic_. _Real-time_ denotes methods that achieve ¿30FPS at 1080p. On _Shelly_ and _DTU_ , we run NeRF and Mip-NeRF with Nerfstudio (Tancik et al., 2023), and use official implementation for other methods. Baselines of _NeRFSynthetic_ are from the original papers. Detailed results for each object/scene are provided in the Supplement.  |  | _Shelly_ | _DTU_ | _NeRFSynthetic_
---|---|---|---|---
|  |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
_offline_ |  NeRF (Mildenhall et al., 2020) | 31.27 | 0.893 | 0.157 | 28.51 | 0.894 | 0.183 | 31.01 | 0.947 | 0.081
NeuS (Wang et al., 2021) | 29.98 | 0.893 | 0.158 | 28.92 |  0.913  | 0.168 | / | / | /
Mip-NeRF (Barron et al., 2021) | 32.59 | 0.899 | 0.148 | 28.90 | 0.900 | 0.179 |  33.09  |  0.961  |  0.043
Ours (full ray) |  34.26  |  0.932  |  0.104  |  33.51  | 0.901 |  0.081  |  32.51  |  0.962  |  0.048
_real-time_ |  I-NGP (Müller et al., 2022) |  33.22  |  0.922  |  0.125  |  31.37  |  0.932  |  0.139  |  33.18  | / | /
MobileNeRF (Chen et al., 2023) | 31.62 | 0.911 | 0.129 | / | / | / | 30.90 | 0.947 | 0.062
VMesh (Guo et al., 2023) | / | / | / | / | / | / | 30.70 | 0.947 | 0.060
Ours |  36.02  |  0.954  |  0.079  |  33.37  |  0.964  |  0.077  | 31.84 |  0.957  |  0.056

###  4.4. _Shelly_ Data Set

The _NeRFSynthetic_ data set that was introduced in (Mildenhall et al., 2020)
is still one of the most widely used data sets to evaluate novel-view
synthesis methods. Yet, it mainly consists of objects with sharp surfaces that
can be well-represented by surface rendering methods (Munkberg et al., 2022),
and thus does not represent the challenge of general scene reconstruction.
This motivated us to introduce a new synthetic data set, which we name
_Shelly_. It covers a wider variety of appearance including fuzzy surfaces
such as hair, fur, and foliage. _Shelly_ contains six object-level scenes:
Khady, Pug, Kitty, Horse, Fernvase and Woolly. For each scene, we have
rendered 128 training and 32 test views from random camera positions
distributed on a sphere with a fixed radius. We are grateful to the original
artists of these models: jhon maycon, Pierre-Louis Baril, abdoubouam, ckat609,
the BlenderKit team, and textures.xyz.

Figure 8.  A gallery of results on the test-views of our _Shelly_ data set.

Table 2 shows quantitative results, while the novel views are qualitatively
compared in Figure 8. Our method significantly outperforms prior methods
across all quality metrics achieving more than 2dB higher PSNR than Instant
NGP. Figure 8 demonstrates that surface-based rendering methods (MobileNerf)
struggle to represent fuzzy surfaces. On the other hand, our method aligns its
representation to the complexity of the scene. For example, Figure 6 shows
that our method represents the skin of the horse as a sharp surface, while
using a wider kernel for its tail, which benefits from volumetric rendering.

###  4.5. _DTU_ Data Set

We consider 15 tabletop scenes from the _DTU_ data set (Jensen et al., 2014).
These scenes were captured by a robot-held monocular RGB camera, and are
commonly used to evaluate implicit surface representations. We follow prior
work (Wang et al., 2021; Yariv et al., 2021) and task the methods to represent
the full scene, but evaluate the performance only within the provided object
masks.

Table 2 depicts that our method outperforms all baselines across all
evaluation metrics. Qualitative results are provided in Figure 9. Different
from the _Shelly_ data set, the performance of _Ours_ on the _DTU_ data set is
slightly lower than that of _Ours (full ray)_ in terms of PSNR. We hypothesize
that this is due to the distribution of the camera poses that observe the
scene only from a single direction. This hinders constraining the neural field
and hence also the adaptive shell extraction. The same reason also contributes
to a significant increase in the sample count for Instant NGP (see Table 1).

Figure 9.  A gallery of results on the _DTU_ data set.

###  4.6. _NeRFSynthetic_ Data Set

The _NeRFSynthetic_ data set introduced in (Mildenhall et al., 2020) contains
8 synthetic objects rendered in Blender and is widely adopted to evaluate the
quality of novel view synthesis methods.

As shown in Table 2, our method can achieve comparable quality to the state of
the art methods Mip-NeRF (Barron et al., 2021) and I-NGP (Müller et al.,
2022), but with a much faster runtime performance (Table 1). Our method also
achieves better image quality compared to recent works optimized for rendering
efficiency (Chen et al., 2023; Guo et al., 2023)111Note that VMesh (Guo et
al., 2023) also optimizes for disk storage, which is an orthogonal direction
to this paper..

###  4.7. _MipNeRF360_ Data Set

The MipNeRF-360 data set (Barron et al., 2022) is a challenging real-world
data set that contains large indoor and outdoor scenes captured from
$360^{\circ}$ camera views222In our evaluation, we exclude the two scenes with
license issues: Flowers and Treehill.. The scenes feature a complex central
object accompanied by a highly detailed background. To better represent the
background, we follow (Yariv et al., 2023) and extend our method with the
scene contraction proposed in (Barron et al., 2022) (more details are provided
in the supplemental document).

Table 3 provides the quantitative results and the qualitative comparison is
depicted in Figure 10. Our method achieves comparable performance to other
interactive methods. Directly compared to I-NGP, our proposed narrow-band
formulation can reduce the number of samples by a factor of three, resulting
in fivetimes higher average frame rates at comparable rendering quality. We
note that on this data set, performance and quality depend significantly on
the background, which our approach is not specialized to handle.

Figure 10.  A gallery of results on the test-views of the MipNerf360 data set.

###  4.8. Performance Evaluation

We compare the performance of our method to the most efficient volumetric
baseline, Instant NGP (Müller et al., 2022), in Table 1. To ensure a fair
comparison, we render the same test views for both methods at 1080p resolution
and remove the GUI overhead. The comparison was run on a single RTX 4090 GPU.
Our narrow-band rendering formulation can efficiently reduce the number of
samples along the ray (up to 10 times) which results in significantly reduced
inference time per frame. On the challenging outbound 360 scenes, our method
already runs at real-time rates. Yet, additional speed-ups could be achieved
by further studying the interaction of our adaptive sample placement with the
spatial remapping employed in these scenes.

Table 3. Quantitative results on the _MipNeRF360_ data set. We report the
PSNR, LPIPS and SSIM results for each object and compare them to baselines.
Our method achieves a performance comparable to the baselines while being
significantly faster during inference (see Table 1). In our comparison, we
exclude the two scenes with license issues: Flowers, Treehill.

|  | Outdoor scenes | Indoor scenes
---|---|---|---
|  |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
_offline_ |  NeRF (Mildenhall et al., 2020) | 22.20 | 0.485 | 0.501 | 26.84 | 0.790 | 0.370
Mip-NeRF (Barron et al., 2021) | 22.02 | 0.505 | 0.484 | 26.98 | 0.798 | 0.361
Mip-NeRF 360 (Barron et al., 2022) |  25.92  |  0.747  |  0.244  |  31.72  |  0.917  |  0.179
Ours (full ray) |  24.30  |  0.703  |  0.316  | 29.04 |  0.900  |  0.239
_interactive_ |  I-NGP (Müller et al., 2022) |  23.90  |  0.648  | 0.369 |  29.47  |  0.877  |  0.273
MobileNeRF (Chen et al., 2023) | 22.90 | 0.524 | 0.463 | 25.74 | 0.757 | 0.453
BakedSDF (Yariv et al., 2023) | 23.40 | 0.577 |  0.351  | 27.20 | 0.845 | 0.300
Ours | 23.17 | 0.606 | 0.389 |  29.19  | 0.872 | 0.285

Table 4. Ablating our method on the _Shelly_ data set. SV Kernel denotes the
spatially varying kernel as introduced in Section 3.2. Band, fixed denotes the
shell is not adaptive but extracted for a given SDF threshold.

Model PSNR $\uparrow$ LPIPS $\downarrow$ SSIM $\uparrow$ Sample $\downarrow$
Ours (full ray, w/o SV kernel) 32.99 0.115 0.921 384 Ours (full ray) 34.26
0.104 0.932 384 Ours (band, fixed $\pm$0.05) 33.83 0.110 0.928 4.51 Ours
(band, fixed $\pm$0.02) 31.14 0.136 0.913 2.29 Ours (keep regularization)
34.22 0.085 0.948 1.74 Ours 36.02 0.079 0.954 1.74

Figure 11.  Ablating the effect of sample count on image quality and runtime
performance. We vary the sample count, and plot the PSNR change (left) and
relative runtime performance (right) compared to the default hyperparameters
denoted as “$1\times$ sample count”. We experiment with six scenes from the
_Shelly_ (fernvase, khady, kitten) and _MipNeRF360_ (bicycle, garden, room)
data sets.

###  4.9. Ablation Study

We ablate our design choices on the _Shelly_ data set in Table 4. In line with
our motivation in Section 3.2, the spatially-varying kernel size provides the
required flexibility to adapt to the local complexity of the scene which
results in improvement across all metrics. Using a fixed SDF threshold to
extract the band requires seeking a compromise between an adaptive shell that
is too narrow to represent fuzzy surfaces (threshold 0.02) or an increased
sample count (threshold 0.05). Instead, our formulation can automatically
adapt to the local complexity of the scene leading to higher quality metrics
and lower sample count. As described in Section 3.5, we disable the
regularization terms after shell extraction to devote more capacity to fit the
visual appearance. Comparing _Ours (keep regularization)_ with _Ours_ , this
leads to improved visual fidelity.

In Figure 11, we ablate our method and study how image quality and runtime
change with different sample counts. We vary the sample step size $\delta_{s}$
in narrow-band rendering (Section 3.4) to produce varying sample counts, and
keep other hyperparameters unchanged. The PSNR is sensitive to sample counts
when the samples are insufficient ($0.25\times$-$1\times$), and the image
quality starts to saturate as the sample counts go higher
($1\times$-$4\times$). In most scenes, the runtime performance is
approximately linear w.r.t. the sample count. For simpler scenes such as
Kitten and Fernvase, smaller sample counts ($0.25\times$-$1\times$) do not
further reduce the runtime due to a mixture of fixed overheads (e.g. Python
interpreter and Dr.Jit tracing) and under-utilization of the GPU.

##  5\. Applications

Our method directly constructs an explicit outer shell mesh $\textsf{M}_{+}$
which by definition contains all regions of space that contribute to the
rendered appearance. This property has great utility for use in downstream
applications.

So far our scenes have represented entirely static content, yet, the world is
full of motion. Cage-based deformation methods have shown promise for enabling
dynamic, non-rigid motion in NeRF and other volumetric representations (Joshi
et al., 2007; Lee et al., 2018; Yuan et al., 2022; Xu and Harada, 2022; Garbin
et al., 2022). The basic idea is to construct a coarse tetrahedral cage around
a neural volume, deform the cage, and use it to render the deformed appearance
of the underlying volume. Our approach perfectly supports such techniques, as
the outer shell mesh $\textsf{M}_{+}$ guides the construction of a cage which
will surely contain the content.

We first dilate and tetrahedralize the outer mesh $\textsf{M}_{+}$ with
FastTetWild (Hu et al., 2020) to produce a tetrahedral mesh that encapsulates
the scene. This mesh acts as a proxy for performing physics simulations,
animations, editing, and other operations. To render our representation after
deforming the tetrahedral cage, any deformation is transferred to
$\textsf{M}_{+}$ and $\textsf{M}_{-}$ via barycentric interpolation, using
precomputed barycentric coordinates generated as a preprocess. Ray directions
are likewise transformed via finite differences. After the transformation, we
proceed with rendering as usual in the reference space of our representation,
as described in Section 3.4. Note that even in the presence of deformations,
the rendering process still benefits from our efficient adaptive shell
representation, and is able to efficiently sample the underlying neural
volume.

We show two examples of applying physical simulation and animation to the
reconstructed objects in Figure 12; please see the supplemental video for
dynamic motion. In the animation example, we manually drive the motion of
plants in a vase according to an analytical wind-like spatial function. Other
animation schemes, such as blend shapes or character rigs could potentially be
substituted to drive the motion. In the physical simulation example, we
simulate the reconstructed asset via finite-element elastic simulation on the
cage tetrahedra including collision penalties (Jatavallabhula et al., 2021).

Figure 12. Our representation is well-suited for animation (_top_) and
physical simulation (bottom). The visual quality is preserved under
deformation: the original shape is shown in leftmost column, with deformations
in the middle and rightmost column. For details, please zoom into the fuzzy
regions (e.g. fur, leaves), and refer to the supplemental video.

##  6\. Discussion

Recent work has developed schemes to accelerate and improve the quality of
NeRF-like scene representations. Section 4 provides comparisons to selected,
particularly relevant methods. Note that due to the high research activity in
the field, it is impossible to compare to all techniques and for many
approaches implementations are not available. Hence, we offer additional
comments on some related work:

  * •

MobileNeRF (Chen et al., 2023), BakedSDF (Yariv et al., 2023), NeRFMeshing
(Rakotosaona et al., 2023), and nerf2mesh (Tang et al., 2023) post-process
NeRF-like models and extract meshes to accelerate inference, similar to this
work. However, these approaches constrain appearance to surfaces, sacrificing
quality. Our method instead retains a full volumetric representation and
nearly full-NeRF quality, at the cost of moderately more expensive inference
(though still real-time on modern hardware).

  * •

DuplexRF (Wan et al., 2023) also extracts an explicit shell from the
underlying neural field and uses it to accelerate rendering, although it does
so with a very different neural representation, prioritizing performance.
Their shell is directly extracted from two thresholds of the radiance field,
which requires the careful selection of the thresholds and results in a noisy
shell that is not adapted to the local complexity of the scene in contrast to
our approach.

  * •

VMesh (Guo et al., 2023) builds upon the similar insight that different parts
of the scene require different treatment. However, their formulation assumes
an additional voxel-grid data structure to mark the volumetric areas that
contribute to the final rendering. This approach suffers from poor complexity
scaling as with the auxiliary acceleration data structure of (Müller et al.,
2022). Instead, our method uses an explicit, adaptive shell to delimit the
areas that contribute to the rendering. Apart from lower complexity, our
formulation seamlessly enables further applications as discussed in Section 5.

##  7\. Conclusion and Future Work

In this work we focus on efficiently rendering NeRFs. Our first stage of
training (Section 3.2) is largely similar to that of (Li et al., 2023), and
likely can be accelerated by algorithmic advancements and low-level tuning
similar to our inference pipeline (Wang et al., 2022a).

Although our method offers large speedups for high-fidelity neural rendering
and runs at real-time rates on modern hardware (Table 1, it is still
significantly more expensive than approaches such as MeRF (Reiser et al.,
2023) that precompute the neural field outputs and bake them onto a discrete
grid representation. Our formulation is complimentary to that of MeRF (Reiser
et al., 2023) and we hypothesize that combining both approaches will lead to
further speedups, potentially reaching the performance—at high quality—of the
methods that bake the volumetric representation to explicit meshes and can run
in real-time even on commodity hardware (_e.g._ (Chen et al., 2023)).

Our method does not guarantee to capture thin structures—if the extracted
adaptive shell omits a geometric region, it can never be recovered during
fine-tuning and will always be absent from the reconstruction. Artifacts of
this form are visible in some _MipNeRF360_ scenes. Future work will explore an
iterative procedure, in which we alternately tune our reconstruction and adapt
the shell to ensure that no significant geometry is missed. Other artifacts
occasionally present in our reconstructions include spurious floating geometry
and poorly-resolved backgrounds; both are common challenges in neural
reconstructions and our approach may borrow solutions from other work across
the field (_e.g._ (Niemeyer et al., 2022)).

More broadly, there is great potential in combining recent neural
representations with high-performance techniques honed for real-time
performance in computer graphics. Here, we have shown how ray tracing and
adaptive shells can be used to greatly improve performance.

###### Acknowledgements.

The authors are grateful to Alex Evans for the insightful brainstorming
sessions and his explorations in the early stages of this project. We also
appreciate the feedback received from Jacob Munkberg, Jon Hasselgren, Chen-
Hsuan Lin, and Wenzheng Chen during the project. Finally, we would like to
thank Lior Yariv for providing the results of BakedSDF.

## References

  * (1)
  * Barron et al. (2021) Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. 2021.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.  _ICCV_ (2021).
  * Barron et al. (2022) Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022\.  Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields.  _CVPR_ (2022).
  * Bi et al. (2021) Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei, Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih. 2021.  Deep relightable appearance models for animatable faces.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–15.
  * Buehler et al. (2001) Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001.  Unstructured Lumigraph Rendering. In _Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’01)_. Association for Computing Machinery, New York, NY, USA, 425–432.
  * Cao et al. (2022) Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, and Jian Ren. 2022.  Real-Time Neural Light Field on Mobile Devices.  _arXiv preprint arXiv:2212.08057_ (2022).
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  TensoRF: Tensorial Radiance Fields. In _European Conference on Computer Vision (ECCV)_.
  * Chen et al. (2023) Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2023.  MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. In _The Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Davis et al. (2012) Abe Davis, Marc Levoy, and Fredo Durand. 2012.  Unstructured Light Fields.  _Comput. Graph. Forum_ 31, 2pt1 (2012), 305–314.
  * Debevec et al. (1996) Paul E. Debevec, Camillo J. Taylor, and Jitendra Malik. 1996\.  Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach. In _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’96)_. Association for Computing Machinery, 11–20.
  * Deng et al. (2022) Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. 2022\.  Depth-supervised NeRF: Fewer Views and Faster Training for Free. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Garbin et al. (2022) Stephan J Garbin, Marek Kowalski, Virginia Estellers, Stanislaw Szymanowicz, Shideh Rezaeifar, Jingjing Shen, Matthew Johnson, and Julien Valentin. 2022.  VolTeMorph: Realtime, Controllable and Generalisable Animation of Volumetric Representations.  _arXiv preprint arXiv:2208.00949_ (2022).
  * Gortler et al. (1996) Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996.  The Lumigraph. In _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’96)_. Association for Computing Machinery, 43–54.
  * Guo et al. (2023) Yuan-Chen Guo, Yan-Pei Cao, Chen Wang, Yu He, Ying Shan, Xiaohu Qie, and Song-Hai Zhang. 2023.  VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesis.  _arXiv preprint arXiv:2303.16184_ (2023).
  * Hedman et al. (2021) Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec. 2021\.  Baking Neural Radiance Fields for Real-Time View Synthesis.  _ICCV_ (2021).
  * Hu et al. (2022) Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. 2022.  EfficientNeRF Efficient Neural Radiance Fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 12902–12911.
  * Hu et al. (2020) Yixin Hu, Teseo Schneider, Bolun Wang, Denis Zorin, and Daniele Panozzo. 2020.  Fast tetrahedral meshing in the wild.  _ACM Transactions on Graphics (TOG)_ 39, 4 (2020), 117–1.
  * Jakob et al. (2022) Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022.  DR. JIT: a just-in-time compiler for differentiable rendering.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–19.
  * Jatavallabhula et al. (2021) Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque, Kevin Xie, Kenny Erleben, et al. 2021\.  gradsim: Differentiable simulation for system identification and visuomotor control.  _arXiv preprint arXiv:2104.02646_ (2021).
  * Jensen et al. (2014) Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. 2014.  Large scale multi-view stereopsis evaluation. In _2014 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 406–413.
  * Joshi et al. (2007) Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. 2007.  Harmonic Coordinates for Character Articulation.  _ACM Trans. Graph._ 26, 3 (jul 2007), 71–es.  https://doi.org/10.1145/1276377.1276466
  * Karnewar et al. (2022) Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. 2022.  ReLU Fields: The Little Non-Linearity That Could. In _ACM SIGGRAPH 2022 Conference Proceedings_ _(SIGGRAPH ’22)_. Association for Computing Machinery, New York, NY, USA, Article 27, 9 pages.  https://doi.org/10.1145/3528233.3530707
  * Kazhdan et al. (2006) Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006\.  Poisson Surface Reconstruction. In _Proceedings of the Fourth Eurographics Symposium on Geometry Processing_ _(SGP ’06, Vol. 256)_. Eurographics Association, 61–70.
  * Kazhdan and Hoppe (2013) Michael M. Kazhdan and Hugues Hoppe. 2013.  Screened poisson surface reconstruction.  _ACM Trans. Graph._ 32, 3 (2013), 29:1–29:13.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (July 2023).  https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * Kopanas et al. (2021) Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. 2021.  Point-Based Neural Rendering with Per-View Optimization.  _Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering)_ 40, 4 (June 2021).  http://www-sop.inria.fr/reves/Basilic/2021/KPLD21
  * Kurz et al. (2022) Andreas Kurz, Thomas Neff, Zhaoyang Lv, Michael Zollhöfer, and Markus Steinberger. 2022\.  AdaNeRF: Adaptive Sampling for Real-time Rendering of Neural Radiance Fields. In _European Conference on Computer Vision (ECCV)_.
  * Lassner and Zollhöfer (2021) Christoph Lassner and Michael Zollhöfer. 2021.  Pulsar: Efficient Sphere-based Neural Rendering.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2021), 1440–1449.
  * Lee et al. (2018) Minjae Lee, David Hyde, Michael Bao, and Ronald Fedkiw. 2018\.  A skinned tetrahedral mesh for hair animation and hair-water interaction.  _IEEE transactions on visualization and computer graphics_ 25, 3 (2018), 1449–1459.
  * Levoy and Hanrahan (1996) Marc Levoy and Pat Hanrahan. 1996.  Light Field Rendering. In _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’96)_. Association for Computing Machinery, 31–42.
  * Li et al. (2023) Max Zhaoshuo Li, Thomas Müller, Alex Evans, Russell H. Taylor, Mathias Unberath, Ming-Yu Liu, and Chen-Hsuan Lin. 2023.  Neuralangelo: High-Fidelity Neural Surface Reconstruction. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Lin et al. (2022) Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2022.  Efficient Neural Radiance Fields with Learned Depth-Guided Sampling. In _SIGGRAPH Asia Conference Proceedings_.
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural Sparse Voxel Fields.  _NeurIPS_ (2020).
  * Martin-Brualla et al. (2021) Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. 2021.  NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In _CVPR_.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _ECCV_.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant Neural Graphics Primitives with a Multiresolution Hash Encoding.  _ACM Trans. Graph._ 41, 4, Article 102 (July 2022), 15 pages.  https://doi.org/10.1145/3528223.3530127
  * Munkberg et al. (2022) Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. 2022\.  Extracting Triangular 3D Models, Materials, and Lighting From Images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 8280–8290.
  * Neff et al. (2021) Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. 2021.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks.  _Computer Graphics Forum_ 40, 4 (2021).
  * Niemeyer et al. (2022) Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan. 2022.  RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs. In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_.
  * Oechsle et al. (2021) Michael Oechsle, Songyou Peng, and Andreas Geiger. 2021\.  UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. In _International Conference on Computer Vision (ICCV)_.
  * Parker et al. (2010) Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010.  OptiX: A General Purpose Ray Tracing Engine.  _ACM Trans. Graph._ 29, 4, Article 66 (jul 2010), 13 pages.  https://doi.org/10.1145/1778765.1778803
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017\.  Automatic differentiation in PyTorch.  _NeurIPS Workshop on Autodiff_ (2017).
  * Rakotosaona et al. (2023) Marie-Julie Rakotosaona, Fabian Manhardt, Diego Martin Arroyo, Michael Niemeyer, Abhijit Kundu, and Federico Tombari. 2023.  NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes.  _arXiv preprint arXiv:2303.09431_ (2023).
  * Rebain et al. (2020) Daniel Rebain, Wei Jiang, Soroosh Yazdani, Ke Li, Kwang Moo Yi, and Andrea Tagliasacchi. 2020\.  DeRF: Decomposed Radiance Fields.  _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020), 14148–14156.
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. In _International Conference on Computer Vision (ICCV)_.
  * Reiser et al. (2023) Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T Barron, and Peter Hedman. 2023.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _arXiv preprint arXiv:2302.12249_ (2023).
  * Riegler and Koltun (2020) Gernot Riegler and Vladlen Koltun. 2020.  Free View Synthesis. In _European Conference on Computer Vision_.
  * Riegler and Koltun (2021) Gernot Riegler and Vladlen Koltun. 2021.  Stable View Synthesis. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Rosu and Behnke (2023) Radu Alexandru Rosu and Sven Behnke. 2023.  PermutoSDF: Fast Multi-View Reconstruction with Implicit Surfaces using Permutohedral Lattices. In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Vol. 2.
  * Rückert et al. (2021) Darius Rückert, Linus Franke, and Marc Stamminger. 2021\.  Adop: Approximate differentiable one-pixel point rendering.  _arXiv preprint arXiv:2110.06635_ (2021).
  * Sara Fridovich-Keil and Alex Yu et al. (2022) Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022\.  Plenoxels: Radiance Fields without Neural Networks. In _CVPR_.
  * Schönberger and Frahm (2016) Johannes Lutz Schönberger and Jan-Michael Frahm. 2016.  Structure-from-Motion Revisited. In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Schönberger et al. (2016) Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016.  Pixelwise View Selection for Unstructured Multi-View Stereo. In _European Conference on Computer Vision (ECCV)_.
  * Sun et al. (2022) Cheng Sun, Min Sun, and Hwann-Tzong Chen. 2022.  Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In _CVPR_.
  * Tancik et al. (2023) Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. 2023.  Nerfstudio: A Modular Framework for Neural Radiance Field Development. In _ACM SIGGRAPH 2023 Conference Proceedings_ _(SIGGRAPH ’23)_.
  * Tang et al. (2023) Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng. 2023.  Delicate textured mesh recovery from nerf via adaptive surface refinement.  _arXiv preprint arXiv:2303.02091_ (2023).
  * Waechter et al. (2014) Michael Waechter, Nils Moehrle, and Michael Goesele. 2014\.  Let There Be Color! Large-Scale Texturing of 3D Reconstructions. In _Computer Vision – ECCV 2014_ (Heidelberg) _(Lecture Notes in Computer Science, Vol. 8693)_ , David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer, 836–850.  https://doi.org/10.1007/978-3-319-10602-1_54
  * Wan et al. (2023) Ziyu Wan, Christian Richardt, Aljaž Božič, Chao Li, Vijay Rengarajan, Seonghyeon Nam, Xiaoyu Xiang, Tuotuo Li, Bo Zhu, Rakesh Ranjan, et al. 2023\.  Learning Neural Duplex Radiance Fields for Real-Time View Synthesis.  _arXiv preprint arXiv:2304.10537_ (2023).
  * Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021\.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _NeurIPS_ (2021).
  * Wang et al. (2022a) Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. 2022a.  NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction.
  * Wang et al. (2022b) Yiqun Wang, Ivan Skorokhodov, and Peter Wonka. 2022b.  HF-NeuS: Improved Surface Reconstruction Using High-Frequency Details.  _arXiv preprint arXiv:2206.07850_ (2022).
  * Wang et al. (2023) Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler. 2023.  Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wood et al. (2000) Daniel N. Wood, Daniel I. Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H. Salesin, and Werner Stuetzle. 2000.  Surface Light Fields for 3D Photography. In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’00)_. ACM Press/Addison-Wesley Publishing Co., 287–296.
  * Xu and Harada (2022) Tianhan Xu and Tatsuya Harada. 2022.  Deforming radiance fields with cages. In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_. Springer, 159–175.
  * Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021\.  Volume rendering of neural implicit surfaces. In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Yariv et al. (2023) Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. 2023\.  BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis.  _arXiv preprint arXiv:2302.14859_ (2023).
  * Yariv et al. (2020) Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. 2020.  Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021\.  PlenOctrees for Real-time Rendering of Neural Radiance Fields. In _ICCV_.
  * Yuan et al. (2022) Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. 2022\.  NeRF-editing: geometry editing of neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18353–18364.
  * Zhang et al. (2021) Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. 2021.  PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2020) Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. 2020.  NeRF++: Analyzing and Improving Neural Radiance Fields.  _arXiv:2010.07492_ (2020).
  * Zhao et al. (2022) Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang, Minye Wu, Lan Xu, and Jingyi Yu. 2022.  Human Performance Modeling and Rendering via Neural Animated Mesh.  _ACM Trans. Graph._ 41, 6, Article 235 (2022), 17 pages.

## Appendix

We provide additional algorithmic details and pseudocode. In the first phase
of training, our method adaptively extracts an explicit mesh envelope which
spatially bounds the neural volumetric representation: level set evolution and
shell extraction are shown in Procedures Appendix and Appendix. In the second
phase of training, as well as inference, we leverage the extracted shells to
sample query points only where they are needed. We cast rays against the shell
meshes and compute query locations in the narrow band between the outer and
inner shell. This is detailed in Procedure Appendix. Note that one ray may
intersect with multiple narrow bands, however it always terminates when
encountering an inner shell. Finally, we include the overall training pipeline
in Procedure Appendix.

Procedure 1 LevelSetEvolution$(f,v_{\text{evolve}},T,\Delta
t,\zeta,\lambda_{\textrm{curv}})$

1: level set field $f\in\mathbb{R}^{X\times Y\times Z}$, velocity
$v_{\text{evolve}}\in\mathbb{R}^{X\times Y\times Z}$, evolution steps $T$,
timestep $\Delta t$, soft falloff threshold $\zeta$, curvature regularization
weight $\lambda_{\textrm{curv}}$

2: Evolved level set field $f_{T}$

3:$f_{0}\leftarrow f$ $\triangleright$Initialize level set field

4:for $i\leftarrow 0$ to $T-1$ do

5: $\Big{[}\frac{\partial f_{i}}{\partial
t}\Big{]}_{\textrm{mot}}\leftarrow-\left|\nabla f_{i}\right|v_{\text{evolve}}$
$\triangleright$Motion term

6: $\Big{[}\frac{\partial f_{i}}{\partial
t}\Big{]}_{\textrm{curv}}\leftarrow-\lambda_{\textrm{curv}}\left|\nabla
f_{i}\right|\big{(}\nabla\cdot\dfrac{\nabla f}{|\nabla f|}\big{)}$
$\triangleright$Curvature term

7: $\omega_{i}\leftarrow\tfrac{1}{2}\big{(}1+\cos(\pi\
\textrm{clamp}(f_{i}/\zeta,-1.,1.)\big{)}$ $\triangleright$Soft falloff (Eq.
5)

8: $\frac{\partial f_{i}}{\partial
t}\leftarrow\omega_{i}\,\big{(}\Big{[}\frac{\partial f_{i}}{\partial
t}\Big{]}_{\textrm{mot}}+\Big{[}\frac{\partial f_{i}}{\partial
t}\Big{]}_{\textrm{curv}}\big{)}$

9: ${f}_{i+1}={f}_{i}+\Delta t\,\frac{\partial f_{i}}{\partial t}$
$\triangleright$Update level set field

10:Return $f_{T}$

Procedure 2
ShellExtraction$(f,s,\tau_{d},\beta_{d},\sigma_{\textrm{min}},\beta_{e},v_{\textrm{max}})$

1: signed distance field $f\in\mathbb{R}^{X\times Y\times Z}$, spatially-
varying kernel size $s\in\mathbb{R}^{X\times Y\times Z}$, grid size
$\tau_{d}$, dilation hyperparameters $\beta_{d},\sigma_{\textrm{min}}$,
erosion hyperparameters $\beta_{e},v_{\textrm{max}}$

2: outer mesh $M_{+}$ and inner mesh $M_{-}$

3:$\alpha\leftarrow\frac{\textrm{Sigmoid}\big{(}(f-\tau_{d}/2)/s\big{)}-\textrm{Sigmoid}\big{(}(f+\tau_{d}/2)/s\big{)}}{\textrm{Sigmoid}\big{(}(f-\tau_{d}/2)/s\big{)}}$
$\triangleright$Eq. 3, as in NeuS

4:

5: $\triangleright$Level set dilation for outer mesh

6:$v_{\textrm{dilate}}\leftarrow\text{Where}(\alpha>\sigma_{\textrm{min}},\beta_{d}\alpha,0)$
$\triangleright$Eq. 6

7:$f_{\textrm{dilate}}\leftarrow\text{LevelSetEvolution}(f,v_{\textrm{dilate}},T=50,\Delta
t=0.1,$

8:$\hskip 116.00023pt\zeta=0.1,\lambda_{\textrm{curv}}=0.01$)

9:$f_{\textrm{dilate}}\leftarrow\min(f_{0},f_{\textrm{dilate}})$
$\triangleright$Clip SDF for a strict dilation

10:$M_{+}\leftarrow\text{MarchingCubes}(f_{\textrm{dilate}})$

11:

12: $\triangleright$Level set erosion for inner mesh

13:$v_{\textrm{erode}}\leftarrow\min(v_{\textrm{max}},\beta_{e}/\alpha)$
$\triangleright$Eq. 7

14:$f_{\textrm{erode}}\leftarrow\text{LevelSetEvolution}(f,v_{\textrm{erode}},T=50,\Delta
t=0.1,$

15:$\hskip 116.00023pt\zeta=0.05,\lambda_{\textrm{curv}}=0$)

16:$f_{\textrm{erode}}\leftarrow\max(f_{0},f_{\textrm{erode}})$
$\triangleright$Clip SDF for a strict erosion

17:$M_{-}\leftarrow\text{MarchingCubes}(f_{\textrm{erode}})$

18:

19:Return $M_{+},M_{-}$

Procedure 3 NarrowBandSampling$(M_{+},M_{-},r,o,w_{s},\delta_{s},$
$\color[rgb]{1,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{1,1,1}\pgfsys@color@gray@stroke{1}\pgfsys@color<EMAIL_ADDRESS>143.6865pt\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{N_{max},dp_{max})}$

1: outer mesh $M_{+}$ and inner mesh $M_{-}$, ray origin
$\mathbf{o}\in\mathbb{R}^{3}$ and direction $\mathbf{r}\in\mathbb{R}^{3}$,
target inter-sample spacing $\delta_{s}$, single-sample threshold $w_{s}$,
maximum number of samples per interval $N_{max}$, and a maximum cap for depth
peeling $dp_{max}$

2: a list of distances $\tau_{s}$ to sampled points along the ray

3:$\tau_{s}\leftarrow\text{empty list}$, $n_{hits}\leftarrow 0$
$\triangleright$Initialize Steps

4:$\triangleright$Find hits of the inner mesh

5:$rayHits_{M_{-}}\leftarrow$ CastRay($M_{-},\mathbf{o},\mathbf{r}$)

6:if $\text{HasNext}(rayHits_{M_{-}})$ then

7: $(d_{M_{-}},flag)\leftarrow\text{GetNextHit}(rayHits_{M_{-}})$

8:else

9: $d_{M_{-}}\leftarrow\infty$

10:$\triangleright$Loop through the hits of the outer mesh

11:$rayHits_{M_{+}}\leftarrow$ CastRay($M_{+},\mathbf{o},\mathbf{r}$)

12:while $\text{HasNext}(rayHits_{M_{+}})$ AND $n_{hits}<dp_{max}$ do

13: $(hitDistance,flag)\leftarrow\text{GetNextHit}(rayHits_{M_{+}})$

14: $n_{hits}\leftarrow n_{hits}+1$

15: if $flag=\text{ENTERING}$ then $\triangleright$Ray enters the mesh band

16: $d_{enter}\leftarrow$ $hitDistance$

17: else if $flag=\text{EXITING}$ then $\triangleright$Ray exits the mesh band

18: $\triangleright$Compute samples between enter and exit

19: $d_{exit}\leftarrow$ Min($hitDistance$, $d_{M_{-}}$)

20: $w\leftarrow d_{exit}-d_{enter}$

21: $N\leftarrow$ Min(Ceil(Max($w-w_{s},0$)/$\delta_{s}$)+1,$N_{max}$)

22: $\tau_{s}\leftarrow\tau_{s}+\text{Linspace}(d_{enter},d_{exit},N+2)[1:-1]$

23: if $hitDistance$ $>d_{M_{-}}$ then

24: break $\triangleright$Terminate if beyond the inner mesh

25:end while

26:Return $\tau_{s}$

Procedure 4 Training Pipeline

1: rays $\mathcal{R}$, ground-truth pixel colors $\mathcal{C}$, training
iterations for the first stage $N_{1}$ and second stage $N_{2}$, network
$\textrm{NN}_{\theta}$

2: optimized network parameters $\theta$, shell $M_{+},M_{-}$
$\triangleright$First stage training with full ray volume rendering

3:for $i\leftarrow 0$ to $N_{1}-1$ do

4: Sample data $\textbf{r}_{i}\in\mathcal{R},\textbf{c}_{i}\in\mathcal{C}$

5:
$output\leftarrow\text{VolumeRendering}(\textrm{NN}_{\theta},\textbf{r}_{i})$

6: $\mathcal{L}\leftarrow\text{Loss}(output,\textbf{c}_{i})$
$\triangleright$Compute loss with Eq. 8

7: Update network:
$\theta\leftarrow\theta-\eta\frac{\partial\mathcal{L}}{\partial\theta}$

8:$\triangleright$Extract adaptive shells

9:$M_{+},M_{-}\leftarrow\text{ShellExtraction}(\textrm{NN}_{\theta})$
$\triangleright$Section 3.3

10:$\triangleright$Second stage training with narrow-band rendering

11:for $i\leftarrow 0$ to $N_{2}-1$ do

12: Sample data $\textbf{r}_{i}\in\mathcal{R},\textbf{c}_{i}\in\mathcal{C}$

13:
$output\leftarrow\text{NarrowBandRendering}(\textrm{NN}_{\theta},\textbf{r}_{i},M_{+},M_{-})$

14:
$\mathcal{L}_{\mathbf{c}}\leftarrow\text{ColorLoss}(output,\textbf{c}_{i})$
$\triangleright$Compute loss with Eq. 9

15: Update network:
$\theta\leftarrow\theta-\eta\frac{\partial\mathcal{L}_{\mathbf{c}}}{\partial\theta}$

16:Return $\theta$, $M_{+},M_{-}$