# HyperPlanes: Hypernetwork Approach to Rapid NeRF Adaptation

Paweł Batorski  Dawid Malarz  Marcin Przewięźlikowski  Marcin Mazur  Slawomir
Tadeja  Przemysław Spurek

###### Abstract

Neural radiance fields (NeRFs) are a widely accepted standard for synthesizing
new 3D object views from a small number of base images. However, NeRFs have
limited generalization properties, which means that we need to use significant
computational resources to train individual architectures for each item we
want to represent. To address this issue, we propose a few-shot learning
approach based on the hypernetwork paradigm that does not require gradient
optimization during inference. The hypernetwork gathers information from the
training data and generates an update for universal weights. As a result, we
have developed an efficient method for generating a high-quality 3D object
representation from a small number of images in a single step. This has been
confirmed by direct comparison with the state-of-the-art solutions and a
comprehensive ablation study.

Machine Learning, ICML

##  1 Introduction

Neural Radiance Field (NeRF) (Mildenhall et al., 2020) is a fully connected,
non-convoluted neural network architecture that can generate new, vivid
renderings of complex 3D objects from a handful of images. The latter has to
be taken from known viewpoints as NeRF represents a given object using a 5D
vector comprised of the coordinates of the camera’s position and viewing
direction. As we render the coloring and density along rays passing through
the scene, the object’s shape and color scheme are directly encoded in the
neural network’s weights with the loss function inspired by classical volume
rendering (Kajiya & Von Herzen, 1984). Consequently, NeRF can synthesize high-
quality views of 3D objects utilizing the relationship between the small set
of input images and well-known computer graphics methods (e.g. ray tracing or
radiosity) (Mildenhall et al., 2020).

These unique characteristics make NeRF an ideal candidate to help tackle
shortcomings experienced by other emerging technologies that classical
computer graphics cannot fully resolve (Kostiuk et al., 2023). For example,
the seamless generation of plausible 3D objects is a crucial, underexplored
challenge, especially in the case of content generation for computer games
(Kostiuk et al., 2023), virtual training (Bozzi et al., 2023), and other
application areas relying on spatial graphical interfaces, e.g., virtual
reality (VR) (Tadeja et al., 2023) or augmented reality (AR) (Bozzi et al.,
2023). Here, NeRF offers close-to-revolutionary advantages over the classical
methods, such as their speed, quality of renders, or the ability to generate
complex 3D objects from a handful of images (Mildenhall et al., 2020) that
populate such virtual environments (Deng et al., 2022b).

Ground truth HyperPlanes Figure 1: Our HyperPlanes model can (in a single
update) produce 3D objects that achieve a superior PSNR ($\uparrow$) when
compared to those generated by a vanilla NeRF trained for approximately 36000
epochs. This results in _$380\times speedup$_ (10 seconds for adapting and
rendering with HyperPlanes vs. 63.7 minutes for training NeRF) in object
reconstruction speed while achieving better or comparable quality.

However, NeRF also suffers from significant constraints. Notably, it has to be
trained separately on each object as the architecture does not generalize to
new, previously unencountered data (Liu et al., 2020). In addition, training
times are often lengthy since the neural network’s weights must encode the
object’s shape characteristics (Liu et al., 2020).

To overcome these constraints, we can employ classical generative models to
force NeRF to generalize to unseen objects. Here, the existing literature
provides examples of GANs (e.g., GRAF (Schwarz et al., 2020), $\pi$-GAN (Chan
et al., 2021), EG3D (Chan et al., 2022)), autoencoders (e.g., Point2NeRF
(Zimny et al., 2022)), and diffusion models (e.g., NFD (Shue et al., 2023),
NeRFDiff (Gu et al., 2023), SSDNeRF (Chen et al., 2023a)). Such models can
generate new 3D objects, but using them to produce NeRF representations from
existing images is complex and typically requires lengthy, resource-consuming,
and costly training (Zimny et al., 2022). In contrast, to make it practical
and applicable in real-life scenarios, we want to take just a few photographs
with a smartphone or any other mass-market video-capturing device and convert
them to NeRF without unnecessary training (Müller et al., 2022).

To address the mentioned problem, we can use meta-learning techniques such as
MAML (Finn et al., 2017) or one of its extensions (e.g., Reptile (Nichol et
al., 2018)). The main idea is to find such parameters of a model that can be
easily adapted to a new task in a few or even a single gradient step. Thus,
MAML algorithms aim to find model weights sensitive to task changes. Small
parameter shifts will significantly improve the loss function of any task
drawn from the tasks’ distribution when changed in the direction of its
gradient. Unfortunately, using such approaches to fit a high-quality NeRF
model needs a few hundred gradient steps to update the universal weights.
Moreover, a few thousand updates are required in inference time (Tancik et
al., 2021).

In contrast to all the presented methods and their inherent limitations, we
propose to rely on the hypernetwork paradigm (Sendera et al., 2023;
Przewiezlikowski et al., 2022) for a few-shot training. Such an approach does
not require gradient optimization in the inference time. Instead, hypernetwork
aggregates information from training data and produces an update for universal
weights. Thus, by combining hypernetworks and NeRFs with partially non-
trainable MultiPlaneNeRFs (Zimny et al., 2023), we have developed an efficient
method for generating 3D representations from 2D images in a single step. To
that end, we contribute in this paper a new method called HyperPlanes that
uses the hypernetworks in NeRF training. The resulting model can generate a
NeRF representation from a few existing images in a single step. Consequently,
it can instantly generate views of new, unseen objects, as demonstrated
through extensive experimental studies presented in our paper.

##  2 Related works

There are many ways in which we can represent 3D objects, including point
clouds (Yang et al., 2022), octrees (Häne et al., 2017), voxel grids (Choy et
al., 2016), multi-view images (Liu et al., 2022), or deformable meshes (Li et
al., 2017). In contrast to these discrete representations, NeRF (Mildenhall et
al., 2020) relies on a fully connected, non-convoluted neural network to store
information about the scene. Thus allowing the rendering of new, formerly
unseen views of a given 3D object (Mildenhall et al., 2020).

#### Representations and rendering

To generate new views of a given scene, the NeRF model or any of its multiple
generalizations uses differentiable volumetric rendering (Liu et al., 2020;
Barron et al., 2021; Niemeyer et al., 2022; Roessle et al., 2022). These
architectures, however, have their own caveats, such as extensive training
time.

(a) Data processed by the HyperPlanes model. We consider support ImagePlanes
$I_{1},\ldots,I_{n}$ with corresponding viewing directions
$\mathbf{d}_{I_{1}},\ldots,\mathbf{d}_{I_{n}}$. Given a point $\mathbf{x}$, we
project it onto $I_{1},\ldots,I_{n}$, obtaining the points
$\mathbf{x}_{I_{1}},\ldots,\mathbf{x}_{I_{n}}$, and interpolate their colors
$I_{1}[\mathbf{x}_{I_{1}}],\ldots,I_{n}[\mathbf{x}_{I_{n}}]$. The goal is to
render the color of the projection of $\mathbf{x}$ onto each query ImagePlane
$J_{i}$ (related to a viewing direction $\mathbf{d}_{J_{i}}$).

(b) Architecture of the HyperPlanes model. Hypernetwork $\mathcal{H}$ predicts
updates $\Delta\theta$ to parameters $\theta$ of target network $\mathcal{F}$
(PointMultiPlaneNeRF) based on selected support ImagePlanes
$I_{1},\ldots,I_{k}$, their respective viewing directions
$\mathbf{d}_{I_{1}},\ldots,\mathbf{d}_{I_{k}}$, and $\theta$ itself. Next,
$\mathcal{F}$ (now parameterized with $\theta+\Delta\theta$) infers the color
$\mathbf{c}$ and the volume $\sigma$ of the projection $\mathbf{x}_{J_{i}}$ of
the point $\mathbf{x}$ onto a query ImagePlane $J_{i}$ (related to a viewing
direction $\mathbf{d}_{J_{i}}$) using Eq. (4).

Figure 2: Visualization of data utilized by HyperPlanes (left), and
architecture (right).

To address these limitations, numerous new approaches have been proposed. For
instance, a 3D asset can be represented by an orthogonal tensor component
(Chen et al., 2022), or we can extend the NeRF model with point clouds or
depth maps (Wei et al., 2021; Deng et al., 2022a; Roessle et al., 2022).

Another set of methods focused on voxel grids. We can optimize such feature
voxel grids to warrant fast radiance field reconstruction (Sun et al., 2022),
which can also be split into a separate, multilevel grid (Müller et al.,
2022). Moreover, a sparse voxel grid can use its nodes as repositories with
spherical harmonic coefficients and densities (Fridovich-Keil et al., 2022).

Another group of NeRF models are built to use only a handful of input images
(Chen et al., 2021; Yu et al., 2021; Wang et al., 2021). The majority of these
techniques utilize a form of a feature extractor generalizable thanks to
training on a diversified catalogue of objects (Wang et al., 2021).
Furthermore, 3D CNN can be employed to aggregate information from multiple
sweeping planes warping 2D neural features. The assemblage process is then
carried out by a ray transformer network (Chen et al., 2021). Other authors
decided to add a convolution layer to transfer input views as means of
representing the NeRF model (Yu et al., 2021).

An alternative approach to produce generalized NeRF is to use generative
models. For example, HoloGAN (Nguyen-Phuoc et al., 2019) and BlockGAN (Nguyen-
Phuoc et al., 2020) use an implicit 3D representation to model real-world
objects.

In GRAF (Schwarz et al., 2020) and $\pi$-GAN (Chan et al., 2021), the authors
use implicit neural radiance fields to generate 3D-aware images and geometry.
In ShadeGAN (Pan et al., 2021), the shading-guided pipeline is used, and in
GOF (Xu et al., 2021), the sampling area of each camera ray is gradually
reduced. In GIRAFFE (Niemeyer & Geiger, 2021), low-resolution feature maps are
generated first. Then, a 2D CNN is used to produce higher resolution outputs.

In StyleSDF (Or-El et al., 2022), the authors merge an SDF-based 3D
representation with a StyleGAN2 for image generation. In EG3D (Chan et al.,
2022), the StyleGAN2 generator and a TriPlane representation of 3D objects are
utilized. Such models outperform other solutions in the quality of generated
objects but are extremely difficult to train (Chan et al., 2022).

The TriPlane representation from EG3D (Chan et al., 2022) finds many
applications in diffusion-based generative models. In (Shue et al., 2023), the
authors propose to train the diffusion model to generate a TriPlane
representation. The model is trained in two steps. First, we replace the data
by a TriPlane representation, after which, we train the final diffusion model.
A similar strategy is used in (Müller et al., 2023), but with the voxel
representation instead of the TriPlane. In (Gu et al., 2023), the authors
propose to modify the U-Net component in diffusion models. On the other hand,
(Chen et al., 2023b) introduces a single-step training for the NeRF-based
diffusion model.

In (Zimny et al., 2023), the authors propose MultiPlaneNeRF, which is an
alternative model to the TriPlane decoder that uses existing images as a non-
trainable representation. In our project, we use the MultiPlaneNeRF
architecture because we do not need a large feature extractor, which is
essential in hypernetwork-based few-shot learning.

#### Few-shot learning

One of the most crucial limitations of contemporary deep learning models is
the requirement for large amounts of data to train them (Wang et al., 2020).
To address this, a wealth of few-shot learning (FSL) approaches have been
proposed in recent years, including (Munkhdalai & Yu, 2017; Munkhdalai et al.,
2018; Raghu et al., 2019; Ye et al., 2020; Zhmoginov et al., 2022) and
others111For a comprehensive survey of different few-shot learning techniques,
see (Wang et al., 2020).. Typically, such methods involve training in a meta-
learning framework, wherein they learn to solve disjoint diverse tasks, each
defined by a support set – a small amount (i.e. few shots) of labeled data,
sampled from a large dataset (Wang et al., 2020; Hospedales et al., 2020).

The most popular family of FSL approaches are the optimization-based methods,
which train general models that can adapt their parameters to the support set
at hand in several gradient steps (Finn et al., 2017; Nichol et al., 2018;
Raghu et al., 2019; Rajeswaran et al., 2019; Finn et al., 2018). While the
models such as MAML (Finn et al., 2017) and Reptile (Nichol et al., 2018) have
been developed primarily for classification tasks, they have also been
successfully used for adapting NeRF representations for rendering new objects
(Tancik et al., 2021). Nevertheless, to obtain sharp renderings in practice,
the model must be updated for a few thousand gradient updates, which increases
its inference time and computational footprint, limiting its usefulness in
practice.

Also, hypernetworks (Ha et al., 2016) have been previously proposed to
alleviate the mentioned problem of few-shot classification models by producing
an update in one step without gradient optimization in inference time (Sendera
et al., 2023; Przewiezlikowski et al., 2022). In contrast, in this paper we
present a hypernetwork that achieves this goal for NeRF representations.

##  3 HyperPlanes: hypernetwork approach to one-shot NeRF adaptation

This section introduces our novel HyperPlanes model. Specifically, we first
present the background and notation for NeRF architecture, training, and
adaptation. Next, we describe a method for using the hypernetwork paradigm to
construct a NeRF representation from 2D images without needing gradient
optimization during inference.

#### Neural Radiance Field (NeRF)

Vanilla NeRF (Mildenhall et al., 2020) is a model for representing individual
complex 3D scenes using a neural architecture. It takes a 5D vector as input
comprised of spatial coordinates ${\bf x}=(x,y,z)$ and a viewing direction
${\bf d}=(\phi,\psi)$. It uses these data to learn rendering the emitted RGB
color ${\bf c}={\bf c}({\bf x},{\bf d})$ and the volume density
$\sigma=\sigma({\bf x})$. Roughly speaking, NeRF training boils down to
minimizing a total squared error between the expected color $\hat{C}({\bf r})$
of a camera ray ${\bf r}={\bf o}+t{\bf d}$ with near and far scene bounds
$t_{n}$ and $t_{f}$ (conceptually expressed as an integral over
$t\in[t_{n},t_{f}]$ involving the volume density $\sigma({\bf r}(t))$ and the
color ${\bf c}({\bf r}(t),{\bf d})$, but numerically estimated by means of
stratified sampling) and its true color $C({\bf r})$. Formally, the goal is to
optimize the multilayer perceptron (MLP) network

$\begin{array}[]{l}\mathcal{F}_{\text{NeRF}}({\bf x},{\bf d};\theta)=({\bf
c},\sigma)\end{array}$ (1)

by applying the following loss function:

$\begin{array}[]{l}\mathcal{L}_{\text{NeRF}}(\cdot;\theta)=\sum_{{\bf
r}}\|\hat{C}({\bf r})-C({\bf r})\|_{2}^{2},\end{array}$ (2)

where the sum is taken over the set of sampled rays stored in batches and
passing through the scene.

In practice, NeRF encodes the structure of an individual 3D object in neural
network weights. Such an approach has important downsides, including time-
consuming training needed separately for each 3D object and weak
generalization to unseen data. A possible answer to these problems is using a
MultiPlaneNeRF architecture, which we briefly outline in the next paragraph.

#### MultiPlaneNeRF

The difference between NeRF and MultiPlaneNeRF (Zimny et al., 2023)
architectures is that the latter uses pre-existing 2D images, which are a non-
trainable planar representation of a 3D object, to train a lightweight
implicit decoder that aggregates image-based input to produce NeRF output.
More formally, given a set $\mathcal{I}=\\{I_{1},\ldots,I_{n}\\}$ of fixed 2D
images, referred to as ImagePlanes, representing different views of a
considered 3D scene, for any point ${\bf x}=(x,y,z)\in\mathbb{R}^{3}$ we first
project it onto each $I_{i}$, obtaining a collection of 2D positions ${\bf
x}_{I_{1}},\ldots,{\bf x}_{I_{n}}$, and then linearly interpolate the RGB
color $I_{i}[{\bf x}_{I_{i}}]\in\mathbb{R}^{3}$ of the point ${\bf
x}_{I_{i}}\in\mathbb{R}^{2}$ from the colors of its four nearest pixels. As a
result, we have an input ${\bf z}={\bf z}({\bf
x},\mathcal{I})\in\mathbb{R}^{5n}$ (consisting of all ${\bf x}_{I_{i}}$s and
$I_{i}[{\bf x}_{I_{i}}]$s) to an implicit decoder that returns the color ${\bf
c}={\bf c}({\bf z},{\bf d})$ and the volume density $\sigma=\sigma({\bf z})$
using the NeRF rendering procedure. The resulting small MLP network

$\begin{array}[]{l}\mathcal{F}_{\text{MultiPlaneNeRF}}({\bf z},{\bf
d},\mathcal{I};\theta)=({\bf c},\sigma)\end{array}$ (3)

is then trained with the loss function
$\mathcal{L}_{\text{MultiPlaneNeRF}}(\cdot,\mathcal{I};\theta)$ as in Eq. (2).

As the authors of (Zimny et al., 2023) show, MultiPlaneNeRF is a
computationally efficient 3D scene representation model with good
generalization capabilities when trained on a large dataset. However, it still
suffers from lower rendering quality than vanilla NeRF, which has the
advantage of learning each object separately. One of the methods to overcome
this problem is to apply an appropriate few-shot parameter adaptation. Before
discussing the details, it is important to note that our further studies are
based on a modified version of the MultiPlaneNerf architecture. Namely, we
assume that the color-volume outcome depends on both the point ${\bf x}$ and
its 2D projections ${\bf z}$. This leads to the following neural model,
referred to as PointMultiPlaneNeRF:

$\begin{array}[]{l}\mathcal{F}_{\text{PointMultiPlaneNeRF}}({\bf x},{\bf
z},{\bf d},\mathcal{I};\theta)=({\bf c},\sigma),\end{array}$ (4)

where ${\bf c}={\bf c}({\bf x},{\bf z},{\bf d})$ and $\sigma=\sigma({\bf
x},{\bf z})$.

#### Few-shot NeRF adaptation

Brain-inspired few-shot learning methods aim to train a model that can easily
adapt to unseen data using only a few samples or training iterations. In a
simple meta-learning scenario used in our approach (Wang et al., 2020), we
deal with a series of randomly selected tasks $\mathcal{T}$, each consisting
of a corresponding loss $\mathcal{L}$ and data distribution $q$ (from which we
draw all samples). During training, the model learns a new task $\mathcal{T}$,
typically using several gradient descent iterations for $\mathcal{L}$, from a
few support samples. Next, it is tested on different query samples. The model
update depends on the influence of parameter changes on the test error. It is
adapted to a new task during inference by applying the same training procedure
with support samples starting from the learned optimal parameters.

When applying a few-shot learning approach to a PointMultiPlaneNeRF
architecture, the task is related to a representation of a single 3D object
and consists of the corresponding loss expressed in Eq. (2) and ImagePlanes
retrieved from a dataset of its existing 2D views. However, if we use a common
implementation (e.g. MAML222Note that to update the weights MAML also uses
time-consuming “gradient through gradient” second-order computations.), many
gradient descent iterations may be required, especially during inference, to
ensure superior rendering performance. To avoid this problem, our proposed
solution incorporates the hypernetwork paradigm, briefly outlined in the
following paragraph.

#### Hypernetworks

Hypernetworks (Ha et al., 2016) are neural models that generate weights for a
target network to solve a specific task. In our HyperPlanes approach, we take
inspiration from recent successes of hypernetworks in few-shot learning
(Sendera et al., 2023; Przewiezlikowski et al., 2022) within the NeRF
adaptation scenario. Here, such techniques reduced the number of iterations on
support samples to a single step without gradient optimization.

#### HyperPlanes

To formally summarize the above discussion, consider a few-shot learning
scenario with individual tasks of the form

$\begin{array}[]{l}\mathcal{T}=\\{\mathcal{L}_{\text{PointMultiPlaneNeRF}}(\cdot,\mathcal{S};\theta),\mathcal{S},\mathcal{Q}\\},\end{array}$
(5)

where $\mathcal{L}_{\text{PointMultiPlaneNeRF}}(\cdot,\mathcal{S};\theta)$ is
the loss expressed by Eq. (2) (replacing NeRF with the corresponding
PointMultiplaneNeRF), while $S=\\{I_{1},...,I_{n}\\}$ and
$\mathcal{Q}=\\{J_{1},\ldots,J_{n}\\}$ denote disjoint sets of support and
query ImagePlanes retrieved from a dataset of existing 2D views of a given 3D
object. Our goal is to train the PointMultiPlaneNeRF architecture

$\begin{array}[]{l}\mathcal{F}_{\text{PointMultiPlaneNeRF}}({\bf x},{\bf
z},{\bf d},\mathcal{S};\theta)=({\bf c},\sigma),\end{array}$ (6)

which we treat as a target network modeling universal weights.

The main idea behind our HyperPlanes approach is to use information extracted
from selected support data (instead of many gradient descent iterations) to
find an optimal task-related parameter update. This allows the HyperPlanes
target network weights to be switched between completely different 3D objects.

Specifically, we distinguish in a hypernetwork architecture
$\mathcal{H}(\cdot;\delta)$ a trainable encoder, which transforms the
ImagePlanes $I_{1},\ldots,I_{k}$ (where $k\leq n$ is treated as a
hyperparameter), hereafter called HyperPlanes, together with the corresponding
viewing directions ${\bf d}_{I_{1}},\ldots,{\bf d}_{I_{k}}$ and the current
weights $\theta$ of the target network given by Eq. (6), into their low-
dimensional code. The resulting representation is then propagated to produce
the required update $\Delta\theta$. Thus, the new weights
$\theta_{\mathcal{T}}$ are calculated using the following formula:

$\theta_{\mathcal{T}}=\theta_{\mathcal{T}}(\theta,\delta)=\theta+\Delta\theta,$
(7)

where

$\Delta\theta=\mathcal{H}(I_{1},\ldots,I_{k},{\bf d}_{I_{1}},\ldots,{\bf
d}_{I_{k}},\theta;\delta).$ (8)

The data and architecture of the HyperPlanes model are visualized in Figure
2). In the following paragraph, we present the successive steps of the
training procedure.

#### Training procedure

Training the HyperPlanes model is done in the following steps. First, we
sample a batch $\mathcal{B}$ from the given dataset $\mathcal{D}$ of training
tasks such as in Eq. (5). Next, for each task $\mathcal{T}\in\mathcal{B}$ we
compute new task-specific weights $\theta_{\mathcal{T}}$ according to Eqs. (7)
and (8). Finally, we update $\theta$ (global target network weights) and
$\delta$ (hypernetwork weights) in separate gradient steps using the Adam
optimizer (Kingma & Ba, 2014) and the following loss function:

$\begin{array}[]{l}{\mathcal{L}}_{\text{HyperPlanes{}}}(\cdot;\theta,\delta)\\\
\ \ \
=\sum_{\mathcal{T}\in\mathcal{B},J_{i}\in\mathcal{Q}}\mathcal{L}_{\text{PointMultiPlaneNeRF}}(\cdot,{\bf
d}_{J_{i}},\mathcal{S};\theta_{\mathcal{T}}),\end{array}$ (9)

where ${\bf d}_{J_{i}}$ is the viewing direction corresponding to the query
ImagePlane $J_{i}$.

For clarity, the entire training procedure is provided in Algorithm 1 (see the
appendix).

Ground truth MultiPlaneNeRF HyperPlanes

Figure 3: Sample images from the MultiPlaneNeRF and HyperPlanes-100 models
trained on the ShapeNet 200$\times$200 dataset, showing reconstructed Cars,
Chairs, and Planes alongside their ground truths. All images were taken from
the same viewing direction. For more examples, see Figure 14 in the appendix.

#### Fine-tuning

During inference in a few-shot learning scenario, when both sets of support
and query samples are available, we can pre-train the hypernetwork on the
support set while keeping the target network frozen. This approach refines the
hypernetwork update for a specific object, yielding notably improved results
with just a few fine-tuning iterations. Importantly, since the fine-tuning is
exclusively performed on the support set, there is no risk of data leakage.

##  4 Experiments

In this section, we perform an experimental analysis of our approach. In
Section 4.1, we thoroughly evaluate the HyperPlanes model on the ShapeNet
dataset (Chang et al., 2015). Section 4.2 details an ablation study of
different components of the proposed solution. As performance evaluation
measures, we employ peak signal-to-noise ratio (PSNR) and structural
similarity index (SSIM) (see, e.g., (Wang et al., 2004)). For a qualitative
evaluation, see Figure 3. Our source code is available at
https://github.com/gmum/HyperPlanes.

###  4.1 Evaluation on the ShapeNet dataset

#### Hypernetwork vs. gradient-based update

In Table 1, we compare HyperPlanes with a set of models proposed in (Tancik et
al., 2021) that were trained with the REPTILE algorithm (Nichol et al., 2018),
which requires gradient-based fine-tuning to quickly adapt to the
representation of an unseen object at hand. Each of the reference approaches
requires 2000 fine-tuning steps. The comparison is made on the car, chair, and
lamp classes of the ShapeNet 128$\times$128 dataset, and in each case we use
$5$ HyperPlanes as input to the hypernetwork. The results are reported in
terms of PSNR values between the true and rendered images. It is clear that
our HyperPlanes model achieves superior results, even though it does not
require gradient-based fine-tuning to adapt to the object at hand.

Table 1: Comparison of HyperPlanes with the few-shot learning methods proposed in (Tancik et al., 2021) in terms of the PSNR ($\uparrow$) metric. All models were trained on the car, chair, and lamp classes of ShapeNet 128$\times$128\. It is evident that our solution provides superior results for different object categories. | Chairs | Cars | Lamps
---|---|---|---
Standard | 12.49 | 11.45 | 15.47
MV Matched | 16.40 | 22.39 | 20.79
MV Shuffled | 10.76 | 11.30 | 13.88
MV Meta | 18.85 | 22.80 | 22.35
SV Meta | 16.54 | 22.10 | 20.95
HyperPlanes (ours) | 24.34 | 27.80 | 25.95

#### HyperPlanes fine-tuning

In Table 2, we show that HyperPlanes benefits from additional gradient-based
fine-tuning. We compare MultiPlaneNERF with HyperPlanes (using 25 HyperPlanes)
trained on the ShapeNet 200$\times$200 car, chair, and plane classes. During
the inference, we fine-tune the hypernetwork component of the model for 100
steps to the object at hand. The results show that HyperPlanes significantly
outperforms MultiPlaneNeRF.

#### Generalization between object types

In Table 2, we demonstrate the versatility of our solution across different
object types. We compare MultiPlaneNeRF and HyperPlanes on the task of
rendering objects from the classes on which the models were not trained. We
also analyze the change in PSNR with different numbers of fine-tuning
iterations. The results show that HyperPlanes with 100 inference steps
outperforms the standard MultiPlaneNeRF in most cases, HyperPlanes without
fine-tuning, and HyperPlanes with only 10 fine-tuning steps. Notably, even
with only 10 fine-tuning steps, our model shows significant performance gains
over MultiPlaneNeRF in most experimental settings.

Table 2: The PSNR ($\uparrow$) values between ground truth and reconstructed images obtained by MultiPlaneNeRF, HyperPlanes without fine-tuning, HyperPlanes with 10 fine-tuning iterations (HyperPlanes-10), and HyperPlanes with 100 fine-tuning iterations (HyperPlanes-100) trained and tested on different classes (car, chair, and plane) of the ShapeNet 200$\times$200 dataset. All results were averaged over 5 runs.  Trained on | Tested on | MultiPlaneNeRF | HyperPlanes-0 | HyperPlanes-10 | HyperPlanes-100 |
---|---|---|---|---|---|---
Cars | Cars | 24.91 | 28.48 | 29.31 | 29.89 |
Chairs | 22.15 | 18.58 | 20.21 | 23.13 |
Planes | 21.32 | 20.26 | 21.58 | 25.03 |
Chairs | Cars | 24.41 | 24.64 | 26.14 | 27.57 |
Chairs | 24.26 | 22.95 | 25.07 | 27.08 |
Planes | 20.84 | 18.36 | 21.49 | 26.03 |
Planes | Cars | 24.19 | 13.57 | 22.35 | 25.35 |
Chairs | 21.69 | 13.15 | 16.77 | 21.12 |
Planes | 24.27 | 26.94 | 28.08 | 28.58 |

#### HyperPlanes inference vs. NeRF training

We compare the number of iterations required for the Vanila NeRF to converge
to a similar level of PSNR as a single HyperPlanes update (without fine-
tuning) when presented with an unseen object. Specifically, we selected two
objects from the car and chair classes of ShapeNet 128$\times$128, represented
by 50 ImagePlanes taken from different perspectives. We divide the set of
ImagePlanes into 25 training and 25 test images for NeRF training and, in the
case of HyperPlanes, use the training images as a support set to generate the
necessary update. The results (averaged over 3 runs) shown in Figure 1
highlight the advantageous generalization capabilities of HyperPlanes, which
achieves superior results in a single step compared to NeRF trained for
approximately 36000 epochs.

Figure 4: Comparison of PSNR ($\uparrow$) of the HyperPlanes training process
when using different target network architectures (NeRF, MultiPlaneNeRF, and
PointMultiPlaneNeRF) and without weights and viewing directions in the
hypernetwork input. Results (averaged over 5 runs) were obtained on the car
class of the ShapeNet 128$\times$128 dataset after 40 epochs. For extended
results, see Figure 12 in the appendix.

###  4.2 Ablation study

#### Selection of a target network architecture

We need to explain why PointMultiPlaneNeRF is the foundation of our target
network instead of NeRF or MultiPlaneNeRF. As shown in Figure 4, it is clear
that PointMultiPlaneNeRF learns much faster in the HyperPlanes setting
compared to MultiPlaneNeRF and NeRF. The benefits of including
PointMultiPlaneNeRF as the target network extend beyond the training process,
as shown in Table 3, which illustrates the differences observed on the test
set measured by PSNR and SSIM.

Table 3: Comparison of the performance (in terms of PSNR ($\uparrow$)) of
HyperPlanes on the test dataset when using different target network
architectures. Results (averaged over 5 runs) were obtained on the car class
of the ShapeNet 128$\times$128 dataset after 40 epochs and are presented with
their standard deviations.

| NeRF | MultiPlaneNeRF | PointMultiPlaneNeRF
---|---|---|---
PSNR | 19.78 $\pm$ 0.22 | 23.96 $\pm$ 0.19 |  24.35 $\pm$ 0.27
SSIM | 0.881 $\pm$ 0.002 | 0.912 $\pm$ 0.001 |  0.927 $\pm$ 0.002

#### Selection of the number of HyperPlanes

We analyze the influence of the number of HyperPlanes used as input to the
hypernetwork. As shown in Figure 5, all considered setups yield comparable
results on average. However, architectures that use a smaller number of
HyperPlanes show distinct instability, leading to significant variations in
results over multiple runs. In contrast, the HyperPlanes model using 25 planes
demonstrates remarkable stability, with results showing minimal variation
between iterations.

Figure 5: Boxplots for PSNR ($\uparrow$) final values obtained by the
HyperPlanes model with different numbers of HyperPlanes. Results were obtained
on the planes class of the ShapeNet 200$\times$200 dataset after 40 epochs.
For extended results, see Figure 10 in the appendix.

#### Selection of the number of ImagePlanes

The authors of (Zimny et al., 2023) suggest using 45 ImagePlanes as input to
MultiPlaneNeRF. However, this poses a challenge in our scenario because the
ShapeNet dataset contains only 50 plane images of each 3D object, which need
to be split between sets of support and query samples. Our default
configuration assigns 25 support ImagePlanes and 25 query ImagePlanes. In
Figure 6, we present the results of an experiment in which we explore how the
performance of our approach changes as we add more and more (from 25 to 50)
support ImagePlanes to the target network (we treat the sets of support and
query samples as unified). It is evident that using 25 support ImagePlanes is
an optimal choice, implying that including additional planes introduces
unnecessary noise to our solution.

Figure 6: Boxplots for PSNR ($\uparrow$) final values obtained by the HyperPlanes model with different numbers of ImagePlanes. Results were obtained on the planes class of the ShapeNet 128$\times$128 dataset after 40 epochs. For extended results, see Figure 11 in the appendix.  Table 4: Time (in seconds) required to infer an image from a single viewing direction, obtained on the car class of ShapeNet 200$\times$200\. Results (averaged over all samples) are presented with their standard deviations. HyperPlanes-0 | HyperPlanes-10 | HyperPlanes-100
---|---|---
25.41 $\pm$ 0.14 | 27.13 $\pm$ 0.12 | 41.3 $\pm$ 0.14

(a) Sample single-view images inferred after training on the chair class of
ShapeNet 200$\times$200.

(b) Sample single-view images inferred after training on the car class of
ShapeNet 200$\times$200.

Figure 7: Comparison of the inference capabilities of the HyperPlanes model
with respect to the number of fine-tuning (FT) iterations. The PSNR
($\uparrow$) values shown were calculated for entire 3D objects.

#### Fine-tuning

We perform an experiment that emphasizes the importance of using fine-tuning
techniques during inference. Despite the high performance of the trained
HyperPlanes model, some examples in the test set show suboptimal
reconstruction, as shown in Figure 7. In contrast, only 10 iterations of pre-
training (using support samples) of the hypernetwork (with the target network
frozen) yield significantly improved results. In addition, the results are
even better after 100 iterations of fine-tuning. In the Table 4 we present the
time differences between various inference setups. Intuitively, performing
additional fine-tuning iterations requires more processing time. This
introduces a trade-off between inference time and the quality of the desired
results. Achieving optimal outcomes involves navigating this balance between
training time and result quality.

#### Impact of the content of the HyperPlanes model architecture

We explore the importance of including different input information in the
architecture of HyperPlanes, i.e., the weights and view directions in the
hypernetwork input, as well as the coordinates of the rendered point in the
target network (which distinguishes our PointMultiPlaneNeRF from
MultiPlaneNeRF). The results of a conducted experiment, shown in Figure 8,
demonstrate a remarkable discrepancy in PSNR between MultiPlaneNeRF and
PointMultiPlaneNeRF backbone. While excluding weights or view directions from
the hypernetwork input results in a smaller difference, it contributes
significantly to the stabilization of the training process, ensuring
consistent results over multiple runs.

Figure 8: Impact of using weights (W) and view directions (VD) as hypernetwork
input, and MultiPlaneNeRF instead of PointMultiPlaneNeRF as target network,
expressed in terms of PSNR ($\uparrow$). Results (averaged over 5 runs) were
obtained on the planes of the ShapeNet 200$\times$200 dataset after 40 epochs.
For extended results, see Figure 13 in the appendix.

##  5 Conclusions

The paper presents a few-shot training approach for NeRF representation of 3D
objects using the hypernetwork paradigm. Our solution does not require
gradient optimization during inference, resulting in the novel HyperPlanes
model, which is a computationally efficient single-step inference method that
requires only a small number of input images. This leads to an exceptionally
accelerated object reconstruction (even 380 times faster than the vanilla NeRF
trained for 36000 epochs), which was confirmed by direct experimental
comparison with existing state-of-the-art solutions and an extensive ablation
study. As a result, we have developed a technique that enables the instant
generation of views of new, unseen 3D objects, making it applicable to spatial
computing and other computer graphics applications.

#### Limitations

A potential drawback of the proposed solution is that it may not be able to
generate reconstructions of comparable quality to those generated by a vanilla
NeRF with extremely extensive training time. This issue will be addressed as a
starting point for our future work.

#### Societal impact

Our research aims to advance the field of machine learning. It has many
potential societal impacts, none of which we feel need to be highlighted here.

## References

  * Barron et al. (2021) Barron, J. T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., and Srinivasan, P. P.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5855–5864, 2021.
  * Bozzi et al. (2023) Bozzi, L. O. S., Samson, K. D. G., Tadeja, S., Pattinson, S., and Bohne, T.  Towards Augmented Reality Guiding Systems: An Engineering Design of an Immersive System for Complex 3D Printing Repair Process.  In _2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_ , pp. 384–389, 2023.  doi: 10.1109/VRW58643.2023.00084.
  * Chan et al. (2021) Chan, E. R., Monteiro, M., Kellnhofer, P., Wu, J., and Wetzstein, G.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5799–5809, 2021.
  * Chan et al. (2022) Chan, E. R., Lin, C. Z., Chan, M. A., Nagano, K., Pan, B., De Mello, S., Gallo, O., Guibas, L. J., Tremblay, J., Khamis, S., et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16123–16133, 2022.
  * Chang et al. (2015) Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., and Yu, F.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * Chen et al. (2021) Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., and Su, H.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 14104–14113. IEEE Computer Society, 2021.
  * Chen et al. (2022) Chen, A., Xu, Z., Geiger, A., Yu, J., and Su, H.  Tensorf: Tensorial radiance fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , pp. 333–350. Springer, 2022.
  * Chen et al. (2023a) Chen, H., Gu, J., Chen, A., Tian, W., Tu, Z., Liu, L., and Su, H.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  _arXiv preprint arXiv:2304.06714_ , 2023a.
  * Chen et al. (2023b) Chen, H., Gu, J., Chen, A., Tian, W., Tu, Z., Liu, L., and Su, H.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  In _ICCV_ , 2023b.
  * Choy et al. (2016) Choy, C. B., Xu, D., Gwak, J., Chen, K., and Savarese, S.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In _ECCV_ , pp. 628–644. Springer, 2016.
  * Deng et al. (2022a) Deng, K., Liu, A., Zhu, J.-Y., and Ramanan, D.  Depth-supervised nerf: Fewer views and faster training for free.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12882–12891, 2022a.
  * Deng et al. (2022b) Deng, N., He, Z., Ye, J., Duinkharjav, B., Chakravarthula, P., Yang, X., and Sun, Q.  Fov-nerf: Foveated neural radiance fields for virtual reality.  _IEEE Transactions on Visualization and Computer Graphics_ , 28(11):3854–3864, 2022b.  doi: 10.1109/TVCG.2022.3203102.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pp. 1126–1135. PMLR, 2017.
  * Finn et al. (2018) Finn, C., Xu, K., and Levine, S.  Probabilistic model-agnostic meta-learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pp. 9537–9548, 2018.
  * Fridovich-Keil et al. (2022) Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., and Kanazawa, A.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5501–5510, 2022.
  * Gu et al. (2023) Gu, J., Trevithick, A., Lin, K.-E., Susskind, J. M., Theobalt, C., Liu, L., and Ramamoorthi, R.  Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion.  In _International Conference on Machine Learning_ , pp. 11808–11826. PMLR, 2023.
  * Ha et al. (2016) Ha, D., Dai, A., and Le, Q. V.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Häne et al. (2017) Häne, C., Tulsiani, S., and Malik, J.  Hierarchical surface prediction for 3d object reconstruction.  In _2017 International Conference on 3D Vision (3DV)_ , pp. 412–420. IEEE, 2017.
  * Hospedales et al. (2020) Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A.  Meta-learning in neural networks: A survey, 2020.
  * Kajiya & Von Herzen (1984) Kajiya, J. T. and Von Herzen, B. P.  Ray tracing volume densities.  _ACM SIGGRAPH computer graphics_ , 18(3):165–174, 1984.
  * Kingma & Ba (2014) Kingma, D. and Ba, J.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_ , 12 2014.
  * Kostiuk et al. (2023) Kostiuk, I., Stachura, P., Tadeja, S. K., Trzciński, T., and Spurek, P.  Hypercolor: A hypernetwork approach for synthesizing autocolored 3-d models for game scenes population.  _IEEE Transactions on Games_ , 15(4):613–622, 2023.  doi: 10.1109/TG.2022.3190792.
  * Li et al. (2017) Li, T., Bolkart, T., Black, M. J., Li, H., and Romero, J.  Learning a model of facial shape and expression from 4d scans.  _ACM Trans. Graph._ , 36(6):194–1, 2017.
  * Liu et al. (2020) Liu, L., Gu, J., Zaw Lin, K., Chua, T.-S., and Theobalt, C.  Neural sparse voxel fields.  _Advances in Neural Information Processing Systems_ , 33:15651–15663, 2020.
  * Liu et al. (2022) Liu, Z., Zhang, Y., Gao, J., and Wang, S.  Vfmvac: View-filtering-based multi-view aggregating convolution for 3d shape recognition and retrieval.  _Pattern Recognition_ , 129:108774, 2022.  ISSN 0031-3203.
  * Mildenhall et al. (2020) Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Müller et al. (2023) Müller, N., Siddiqui, Y., Porzi, L., Bulo, S. R., Kontschieder, P., and Nießner, M.  Diffrf: Rendering-guided 3d radiance field diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4328–4338, 2023.
  * Müller et al. (2022) Müller, T., Evans, A., Schied, C., and Keller, A.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, July 2022.  doi: 10.1145/3528223.3530127.
  * Müller et al. (2022) Müller, T., Evans, A., Schied, C., and Keller, A.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Munkhdalai & Yu (2017) Munkhdalai, T. and Yu, H.  Meta networks.  In _International Conference on Machine Learning_ , pp. 2554–2563. PMLR, 2017.
  * Munkhdalai et al. (2018) Munkhdalai, T., Yuan, X., Mehri, S., and Trischler, A.  Rapid adaptation with conditionally shifted neurons.  In _International Conference on Machine Learning_ , pp. 3664–3673. PMLR, 2018.
  * Nguyen-Phuoc et al. (2019) Nguyen-Phuoc, T., Li, C., Theis, L., Richardt, C., and Yang, Y.-L.  Hologan: Unsupervised learning of 3d representations from natural images.  In _2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , pp. 2037–2040. IEEE, 2019.
  * Nguyen-Phuoc et al. (2020) Nguyen-Phuoc, T. H., Richardt, C., Mai, L., Yang, Y., and Mitra, N.  Blockgan: Learning 3d object-aware scene representations from unlabelled images.  _Advances in Neural Information Processing Systems_ , 33:6767–6778, 2020.
  * Nichol et al. (2018) Nichol, A., Achiam, J., and Schulman, J.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Niemeyer & Geiger (2021) Niemeyer, M. and Geiger, A.  Giraffe: Representing scenes as compositional generative neural feature fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11453–11464, 2021.
  * Niemeyer et al. (2022) Niemeyer, M., Barron, J. T., Mildenhall, B., Sajjadi, M. S., Geiger, A., and Radwan, N.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5480–5490, 2022.
  * Or-El et al. (2022) Or-El, R., Luo, X., Shan, M., Shechtman, E., Park, J. J., and Kemelmacher-Shlizerman, I.  Stylesdf: High-resolution 3d-consistent image and geometry generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 13503–13513, 2022.
  * Pan et al. (2021) Pan, X., Xu, X., Loy, C. C., Theobalt, C., and Dai, B.  A shading-guided generative implicit model for shape-accurate 3d-aware image synthesis.  _Advances in Neural Information Processing Systems_ , 34:20002–20013, 2021.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S.  Pytorch: An imperative style, high-performance deep learning library.  In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.
  * Przewiezlikowski et al. (2022) Przewiezlikowski, M., Przybysz, P., Tabor, J., Zięba, M., and Spurek, P.  Hypermaml: Few-shot adaptation of deep models with hypernetworks.  2022\.
  * Raghu et al. (2019) Raghu, A., Raghu, M., Bengio, S., and Vinyals, O.  Rapid learning or feature reuse? towards understanding the effectiveness of maml.  _arXiv preprint arXiv:1909.09157_ , 2019.
  * Rajeswaran et al. (2019) Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S.  Meta-learning with implicit gradients.  _Advances in Neural Information Processing Systems_ , 32:113–124, 2019.
  * Roessle et al. (2022) Roessle, B., Barron, J. T., Mildenhall, B., Srinivasan, P. P., and Nießner, M.  Dense depth priors for neural radiance fields from sparse input views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12892–12901, 2022.
  * Schwarz et al. (2020) Schwarz, K., Liao, Y., Niemeyer, M., and Geiger, A.  Graf: Generative radiance fields for 3d-aware image synthesis.  _Advances in Neural Information Processing Systems_ , 33:20154–20166, 2020.
  * Sendera et al. (2023) Sendera, M., Przewięźlikowski, M., Karanowski, K., Zięba, M., Tabor, J., and Spurek, P.  Hypershot: Few-shot learning by kernel hypernetworks.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 2469–2478, 2023.
  * Shue et al. (2023) Shue, J. R., Chan, E. R., Po, R., Ankner, Z., Wu, J., and Wetzstein, G.  3d neural field generation using triplane diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20875–20886, 2023.
  * Sun et al. (2022) Sun, C., Sun, M., and Chen, H.-T.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5459–5469, 2022.
  * Tadeja et al. (2023) Tadeja, S., Godula, K., Cybulski, A., Bohne, T., and Wozniak, M.  Towards More Effective VR-Based Presentations of Real-World Assets: Showcasing Mobile MRI to Medical Practitioners and Technicians.  In _2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)_ , pp. 183–186, 2023.  doi: 10.1109/VRW58643.2023.00045.
  * Tancik et al. (2021) Tancik, M., Mildenhall, B., Wang, T., Schmidt, D., Srinivasan, P. P., Barron, J. T., and Ng, R.  Learned initializations for optimizing coordinate-based neural representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2846–2855, 2021.
  * Wang et al. (2021) Wang, Q., Wang, Z., Genova, K., Srinivasan, P., Zhou, H., Barron, J. T., Martin-Brualla, R., Snavely, N., and Funkhouser, T.  Ibrnet: Learning multi-view image-based rendering.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition)_ , pp. 4688–4697. IEEE Computer Society, 2021.
  * Wang et al. (2020) Wang, Y., Yao, Q., Kwok, J., and Ni, L. M.  Generalizing from a few examples: A survey on few-shot learning, 2020\.
  * Wang et al. (2004) Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Wei et al. (2021) Wei, Y., Liu, S., Rao, Y., Zhao, W., Lu, J., and Zhou, J.  Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5610–5619, 2021.
  * Xu et al. (2021) Xu, X., Pan, X., Lin, D., and Dai, B.  Generative occupancy fields for 3d surface-aware image synthesis.  _Advances in Neural Information Processing Systems_ , 34:20683–20695, 2021.
  * Yang et al. (2022) Yang, F., Davoine, F., Wang, H., and Jin, Z.  Continuous conditional random field convolution for point cloud segmentation.  _Pattern Recognition_ , 122:108357, 2022.
  * Ye et al. (2020) Ye, H.-J., Hu, H., Zhan, D.-C., and Sha, F.  Few-shot learning via embedding adaptation with set-to-set functions.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8808–8817, 2020.
  * Yu et al. (2021) Yu, A., Ye, V., Tancik, M., and Kanazawa, A.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4578–4587, 2021.
  * Zhmoginov et al. (2022) Zhmoginov, A., Sandler, M., and Vladymyrov, M.  Hypertransformer: Model generation for supervised and semi-supervised few-shot learning, 2022.
  * Zimny et al. (2022) Zimny, D., Trzciński, T., and Spurek, P.  Points2nerf: Generating neural radiance fields from 3d point cloud.  _arXiv preprint arXiv:2206.01290_ , 2022.
  * Zimny et al. (2023) Zimny, D., Tabor, J., Zięba, M., and Spurek, P.  Multiplanenerf: Neural radiance field with non-trainable representation.  _arXiv preprint arXiv:2305.10579_ , 2023.

##  Appendix A Additional experimental study

#### Selection of updated target network layers

This experiment aims to investigate the reasoning behind selectively
generating updates for specific layers in the target network. Figure 9
examines two scenarios: one in which updates are generated for all target
network parameters, and the other in which updates are generated only for the
last layer, i.e., the RGB color and opacity layer. The experiment demonstrates
a significant performance advantage for the scenario where updates are
generated only for the last selected layers. A possible explanation for this
phenomenon is that generating updates for only a few layers is essential,
while generating updates for more layers introduces unnecessary noise that the
hypernetwork cannot handle effectively.

Figure 9: Comparison of the HyperPlanes training process when generating
weights for all layers versus selected layers, using PSNR ($\uparrow$), loss
($\downarrow$), and SSIM ($\uparrow$). Results (averaged over 5 runs) were
obtained on the plane class of the ShapeNet 200$\times$200 dataset after 40
epochs. The loss was multiplied by 100.

#### Selection of the number of HyperPlanes

Figure 10 displays the results of our experiment on the impact of selecting
the number of HyperPlanes. It is clear that using 25 HyperPlanes ensures
stable outcomes.

Figure 10: Comparison of the HyperPlanes training process when different
numbers of HyperPlanes are provided to the hypernetwork, using PSNR
($\uparrow$), loss ($\downarrow$), and SSIM ($\uparrow$). Results (averaged
over 5 runs) were obtained on the plane class of the ShapeNet 200$\times$200
dataset after 40 epochs. The loss was multiplied by 100.

#### Selection of the number of ImagePlanes

Figure 11 shows the results of our experiment on the impact of selecting the
number of ImagePlanes. The results indicate that utilizing 25 ImagePlanes has
clear benefits.

Figure 11: Comparison of the HyperPlanes training process when different
numbers of ImagePlanes are provided to the target network, using PSNR
($\uparrow$), loss ($\downarrow$), and SSIM ($\uparrow$). Results (averaged
over 5 runs) were obtained on the plane class of the ShapeNet 200$\times$200
dataset after 40 epochs. The loss was multiplied by 100.

#### Selection of a target network architecture

Figure 12 shows the results of our experiment where we tested different target
networks as the backbone for the HyperPlanes model. The findings clearly
demonstrate the benefits of using the PointMultiPlaneNeRF architecture.

Figure 12: Comparison of the HyperPlanes training process when different
target network architectures are employed, using PSNR ($\uparrow$), loss
($\downarrow$), and SSIM ($\uparrow$). Results (averaged over 5 runs) were
obtained on the car class of the ShapeNet 128$\times$128 dataset after 40
epochs. The loss was multiplied by 100.

#### Impact of the content of HyperPlanes architecture

The outcomes of our experiment on the impact of the content of the HyperPlanes
model architecture are presented in Figure 13. The results indicate the
advantages of utilizing PointMultiPlaneNeRF as the target network, with
viewing directions and weight values as the input for the hypernetwork.

Figure 13: Impact of using weights (W) and view directions (VD) as
hypernetwork input, and MultiPlaneNeRF instead of PointMultiPlaneNeRF as
target network, expressed in terms of PSNR ($\uparrow$), loss ($\downarrow$),
and SSIM ($\uparrow$). Results (averaged over 5 runs) were obtained on the car
class of the ShapeNet 128$\times$128 dataset after 40 epochs. The loss was
multiplied by 100.

#### Single-class inference capabilities

Table 5 explores the capabilities of HyperPlanes and MultiPlaneNeRF in single-
class inference, where the models are trained and tested on the same class.
The models were trained and tested on the same class. Our observations show
that 10 fine-tuning iterations for the HyperPlanes model produce significantly
improved results compared to MultiPlaneNeRF.

##  Appendix B Training details

#### Data

We used the ShapeNet dataset in two different configurations. In the first, we
used the original resolution, following the conventional training and testing
splits (ShapeNet 128$\times$128). In the second, we introduced a new
configuration with higher resolution, accompanied by a revised training and
testing split, analogous to that used in AtlasNet (ShapeNet 200$\times$200).

#### Architecture

Our architecture consists of three main components: hypernetwork, encoder (a
part of the hypernetwork), and target network. Depending on the type of
dataset, we use different parameter configurations for the hypernetwork and
encoder to achieve better results. For example, for a smaller dataset, we used
a ConvNet architecture as the encoder, while for larger datasets, we used
ResNet101.

#### Hyperparameters

In all experiments, we used a batch size that contained only one task. For
each task in the datasets, we chose the number of epochs. For different
classes of Shapenet 128$\times$128 these were: Cars – 1000, Lamps – 1100,
Chairs – 700, while for different classes of Shapenet 200$\times$200 these
were: Cars – 2420, Planes – 2460, Chairs – 1920. Additionally, a grid search
was performed in which we tested different learning rates (from 0.01 to
0.00000001), different numbers of HyperPlanes and ImagePlanes, and different
configurations of the target network, such as number of layers and network
width. For the ablation study, we run each experiment 5 times to obtain the
mean and standard deviation estimates.

#### Implementation

We implemented HyperPlanes in the PyTorch framework (Paszke et al., 2019).
Training was performed on a single NVidia A100 GPU.

Figure 14: Sample images from the MultiPlaneNeRF and HyperPlanes-100 models trained on the ShapeNet 200$\times$200 dataset, showing reconstructed cars, chairs, and planes alongside their ground truths. All images were taken from the same viewing direction. Table 5: The PSNR ($\uparrow$) values on the train and test sets for MultiPlaneNeRF, Autoencoder, HyperPlanes without fine-tuning, HyperPlanes with 10 fine-tuning iterations (HyperPlanes-10), and HyperPlanes with 100 fine-tuning iterations (HyperPlanes-100). Results (averaged over 5 runs) were obtained on the car, chair, and plane classes of the ShapeNet 200$\times$200 dataset. |  | MultiPlaneNeRF | Autoencoder | HyperPlanes | HyperPlanes-10 | HyperPlanes-100
---|---|---|---|---|---|---
Train | Cars | 26.21 | 28.14 | 30.12 | 30.12 | 30.12
Chairs | 25.28 | 23.90 | 28.30 | 28.30 | 28.30
Planes | 25.28 | 24.83 | 30.12 | 30.12 | 30.12
Test | Cars | 24.79 | 20.86 | 28.48 | 29.31 | 29.89
Chairs | 24.26 | 17.17 | 22.95 | 25.07 | 27.08
Planes | 24.26 | 14.18 | 26.94 | 28.08 | 28.58
Algorithm 1 Training procedure for the HyperPlanes model

1: Require: dataset of training tasks $\mathcal{D}$; hyperparameters $\beta$
(stepsize for the Adam optimizer), $k$ (number of HyperPlanes)

2: while not done do

3: sample batch of tasks $\mathcal{B}$ from $\mathcal{D}$

4: for each task
$\mathcal{T}=\\{\mathcal{L}_{\text{PointMultiPlaneNeRF}}(\cdot,\mathcal{S};\theta),\mathcal{S},\mathcal{Q}\\}$
from $\mathcal{B}$ do

5: compute new task-specific weights $\theta_{\mathcal{T}}$ using Eqs. (7) and
(8)

6: end for

7: update $\theta$ and $\delta$ using the Adam optimizer with the stepsize
$\beta$ and the loss function
$\mathcal{L}_{\text{HyperPlanes{}}}(\cdot;\theta,\delta)$ given in Eq. (9)

8: end while

#### ShapeNet 128$\times$128

  * Hypernetwork

    * fully connected layers with ReLU activation functions

    * layer depth: 3

    * layer width: 256

    * learning rate: 0.0001

    * ImagePlanes: 25

    * HyperPlanes: 5

    * Encoder (ConvNet)

      * convolutional layers followed by batch normalization

      * kernel size 3$\times$3 for each layer

      * learning rate: 0.0001

  * Target Network

    * fully connected layers with ReLU activation functions

    * layer depth: 8

    * layer width: 256

    * importance: 128

    * rand: 512

    * samples: 64

    * learning rate: 0.0001

#### ShapeNet 200$\times$200

  * Hypernetwork

    * fully connected layers with ReLU activation functions

    * layer depth: 10

    * layer width: 256

    * learning rate: 0.0001

    * ImagePlanes: 25

    * HyperPlanes: 25

    * Encoder (ResNet101)

      * classic ResNet101 architecture with specific output size

      * learning rate: 0.0001

  * Target Network

    * fully connected layers with ReLU activation functions

    * layer depth: 8

    * layer width: 256

    * importance: 128

    * rand: 512

    * samples: 64

    * learning rate: 0.0001