# NViST: In the Wild New View Synthesis from a Single Image with Transformers

Wonbong Jang Lourdes Agapito
Department of Computer Science, University College London
<EMAIL_ADDRESS>
###### Abstract

We propose NViST, a transformer-based model for novel-view synthesis from a
single image, trained on a large-scale dataset of in-the-wild images with
complex backgrounds. NViST transforms image inputs directly into a radiance
field, adopting a scalable transformer-based architecture. In practice, NViST
exploits the self-supervised features learnt by a masked autoencoder (MAE),
and learns a novel decoder that translates features to 3D tokens via cross-
attention and adaptive layer normalization. Our model is efficient at
inference since only a single forward-pass is needed to predict a 3D
representation, unlike methods that require test-time optimization or sampling
multiple times such as 3D-aware diffusion models. We tackle further
limitations of current new-view synthesis models. First, unlike most
generative models that are trained in a category-specific manner, often on
synthetic datasets or on masked inputs, our model is trained on MVImgNet, a
large-scale dataset of real-world, casually-captured videos containing
hundreds of object categories with diverse backgrounds. Secondly, our model
does not require canonicalization of the training data – i.e. aligning all
objects with a frontal view – only needing relative pose at training time
which removes a substantial barrier to it being used on casually captured
datasets. We show results on unseen objects and categories on MVImgNet and
even casual phone captures. We conduct qualitative and quantitative
evaluations on MVImgNet and ShapeNet to show that our model represents a step
forward towards enabling true in-the-wild novel-view synthesis from a single
image. The project webpage is https://wbjang.github.io/nvist_webpage.

Figure 1: We introduce NViST, a transformer-based architecture that enables
synthesis from novel viewpoints given a single in the wild input image. We
test our model not only on held-out scenes of MVImgNet, a large-scale dataset
of casually captured videos of hundreds of object categories (Right) but also
on out-of-distribution challenging phone-captured scenes (Left).

##  1 Introduction

Learning 3D scene representations from RGB images for novel view synthesis or
3D modelling remains a pivotal challenge for the computer vision and graphics
communities. Traditional approaches, such as structure from motion (SfM) and
multiview stereo pipelines, endeavor to optimize the 3D scene from RGB images
directly by leveraging geometric and photometric consistency. The advent of
Neural Radiance Fields (NeRF) [43] and its subsequent developments has marked
a significant step forward by encoding the entire 3D scene within the weights
of a neural network (or feature grid), from RGB images only. Although NeRF
requires more than dozens of images for training, efforts have been undertaken
to generalize NeRF across multiple scenes by taking a single image as an input
[81, 75, 26, 30, 11, 38, 45]. Nonetheless, learning multiple real-world scenes
with a NeRF-based model still remains a challenging task as there are scale
ambiguities, scene misalignments and diverse backgrounds. The huge success of
2D latent diffusion models [56] has recently sparked interest in 3D diffusion
models. One trend is to make diffusion models 3D-aware by fine-tuning a pre-
trained 2D diffusion model. However, so far these approaches are trained on
masked inputs and do not deal with backgrounds [41, 42, 40, 52]. Other
approaches [23, 9, 69, 13] build the diffusion model on top of volume
rendering in 3D space, but these models are computationally expensive and are
slow to sample from.

Many recent breakthroughs in computer vision can be attributed to the
emergence of very large datasets paired with the transformer architecture. For
instance, MiDAS [53] exploits multiple diverse datasets, leading to robust
performance in zero-shot depth estimation from a single RGB image while SAM
[34] demonstrates that an extensive dataset coupled with a pretrained MAE and
prompt engineering can significantly improve performance in segmentation.
However, in 3D computer vision, scaling up real-world datasets has not been as
straightforward. Synthetic datasets like ShapeNet [10] or Objaverse [17] have
helped to promote progress, but there is a large domain gap.

The recent release of large-scale multiview datasets such as Co3D [55] or
MVImgNet [82] has meant a huge step forward, which coupled with the
availability of robust SfM tools such as COLMAP [62, 61] or ORB-SLAM [47] have
enabled the release of their camera parameters. In this paper we exploit a
subset of MVImgNet [82] which has one order of magnitude more categories and
$\times$2 more objects than Co3D [55].

However, there remain significant challenges to train a new view synthesis
model on such large scale datasets due to the huge diversity of objects, scene
scales, backgrounds, and scene alignment issues. Motivated by this we propose
NViST, a scalable transformer-based architecture trained on a large-scale
real-world dataset to enable in-the-wild new view synthesis from a single
image. Our contributions can be summed up as follows:

  * •

Our novel decoder maps MAE features to 3D tokens via cross-attention and
adaptive layer normalization conditioned on camera parameters to tackle scale
ambiguities.

  * •

Efficient inference as it only needs one feed-forward pass.

  * •

NViST only requires relative pose during training and does not require a
canonicalized dataset.

  * •

Our qualitative and quantitative evaluations on test (unseen) scenes and
categories show competitive performance on challenging real-world scenes.

  * •

We demonstrate good performance on a zero-shot new-view synthesis task on out-
of-distribution scenes captured on a mobile-phone device.

Figure 2: NViST Architecture. NViST is a feed-forward transformer-based model
that takes a single in-the-wild image as input, and renders a novel view. The
encoder, a finetuned Masked Autoencoder (MAE), generates self-supervised
feature tokens and a class token (cls). Our novel decoder conditioned on
normalised focal length and camera distance via adaptive layer normalisation,
translates feature tokens to output tokens via cross-attention, and reasons
about the occluded region through self-attention. We employ MLPs to reshape
the output tokens into a vector-matrix representation that encodes the
radiance field and can be queried to obtain the color and sigma of points to
perform volume rendering. NViST is trained end-to-end via a combination of
losses: photometric $L_{2}$, perceptual $L_{\text{LPIPS}}$, and a distortion-
based regulariser $L_{\text{reg}}$. NViST is efficient at inference since it
requires only a single forward-pass to predict a 3D from an input image.

##  2 Related Work

Transformer and ViT:  The Transformer [71], a feed-forward, attention-based
architecture, was initially introduced as an alternative to Recurrent Neural
Networks. By predicting the next masked tokens with its scalable architecture,
it has brought a revolution in the field of natural language processing.
Transformer has been used in other domains, including the field of computer
vision. Vision Transformer (ViT) [19] utilizes the patches for attention
blocks, and it achieves performance levels comparable to those of
Convolutional Neural Networks (CNN) [24, 65] based architectures in the image
recognition task.

Self-supervised learning and MAE: ViT is trained in a supervised way like
numerous other vision tasks depending on labels. One of the reasons behind the
Transformer’s success story is that it can be trained in a self-supervised
way. Masked Autoencoder (MAE) [25] adopts a ViT architecture, and is trained
by inpainting the randomly masked patches. MAE undergoes fine-tuning for
specific vision tasks, and it demonstrated superior performance to ViT without
the need for data augmentation. DINO [6] also adopts the self-supervised
learning strategy, and its features are trained on an image level, while MAE
is supervised on a patch level.

Neural Implicit Representations:  Neural implicit representations aim to learn
a 3D representation without direct 3D supervision. It has been explored in
previous approaches, including depth prediction [48] and scene representation
through ray marching and coordinate-based MLPs [66]. Mildenhall et al. [43]
proposed Neural Radiance Fields (NeRF), which integrates coordinate-based
MLPs, positional encoding, and volume rendering to perform multi-view stereo
of a given scene. Upon optimisation, NeRF can render novel views with
impressively high quality. Beyond novel view synthesis, NeRF has found utility
in a diverse range of other computer vision tasks such as segmentation [84,
72, 21], surface reconstruction [49, 74, 73, 44], and camera registration [37,
31, 16, 78, 39, 14].

Grid-based representations: One limitation of the original NeRF method is its
lengthy training time. As an alternative to using coordinate-based MLPs, grid-
based approaches [80, 67, 46, 12, 20, 79] have been proposed to expedite
training. In the context of 3D-aware Generative Adversarial Networks (GANs),
EG3D [8] introduced the use of triplanes which project 3D features into three
different planes. TensoRF [12] compared different tensor decomposition methods
and proposed the vector-matrix representation.

Learning multiple scenes based on NeRF:  Generalizing NeRFs with multiple
scenes has been a challenging problem. Several methods utilise the 2D features
from the pre-trained network with the target views [81, 75, 11, 55, 26, 70,
15]. Another approach is to condition the network on latent vectors. [22, 30,
45, 54, 2, 27]. Adding an adversarial loss to NeRF leads to 3D-aware GAN,
allowing consistent rendering from different viewpoints. [7, 8, 63, 36, 5].
Many of the approaches implicitly assume that the objects are aligned and have
the issue of training multiple real scenes.

Diffusion models in 3D:  Latent diffusion [56] and the open-source of Stable
Diffusion pretrained on the large dataset transform the field of image
generation. Directly applying the diffusion models on 3D has issues as there
are no large-scale dataset with the 3D ground-truth, and the field of 3D
computer vision has evolved into learning 3D only from RGB images. Fine-tuning
the pre-trained Stable Diffusion with the large scale synthetic dataset [17]
allows the diffusion models to be 3D-aware [41, 52, 40], or the model directly
learns to condition on images and camera poses [76]. Single Stage Diffusion
NeRF [13] trains the 3D diffusion model and NeRF optimisation together,
assuming NeRF intermediate results as the ground-truth. To perform denoising
in 3D space, there are approaches [1, 33, 68] that back-project image features
to 3D. Instead of denoising in 3D, several methods employ the image-
conditioned NeRF to render features in 3D and learn to denoise in 2D spaces
[23, 9, 69]. Diffusion models usually take a long time to sample, and they
have the issue of directly modeling the geometry in 3D.

3D Representation with Transformers:  Geometry-free methods employing
Transformer architectures have been explored as seen in [35, 57, 58, 59].
VisionNeRF [38] utilizes the ViT encoder to extract features, which are
further mapped into multi-level feature maps via CNN decoder. VQ3D [60]
extends the vector quantization models for 3D, and successfully trains
ImageNet dataset with the pseudo disparity and adversarial losses. MCC [77]
takes a single RGB-D as an input, and predicts the color, occupancy. GINA-3D
[64] leverages cross-attention between 2D patches and triplanes, and and
focuses more on quantization and category-specific generation. Shap-E [32] is
based on diffusion models, and it generates 3D implicit functions from point
clouds and images. The concurrent work LRM [28] extracts image features
through DINO [6], and refines learnable positional embeddings via attention
mechanisms. However, LRM trains the dataset without backgrounds, deploys a
larger model (more than $500$M parameters) and requires much more computing
($128$ A100 GPUs), while NViST learns scenes with backgrounds, and is trained
with a single A100 GPU.

##  3 Methodology

Figure 3: Encoder Output Visualisation: (Top) Input images. (Middle) Features
from a fine-tuned MAE, which serves as initialisation to our encoder. (Bottom)
Feature tokens after end-to-end training, visualized after dimensionality
reduction with PCA to $3$ dimensions. Features appear smoother and more
segment-focused, supporting the fact that updating encoder weights
significantly improves the performance of the model, and is further evidenced
by the quantitative analysis in Table 1

NViST is a feed-forward conditional encoder-decoder model built upon
transformers to predict a radiance field from a single image to enable
synthesis from novel viewpoints. As Figure 2 shows, our architecture is
structured into three key components: two transformer-based modules (encoder
and decoder) and a NeRF renderer, which is composed of a shallow multi-layer
perceptron (MLP) and a differentiable volume rendering module. NViST has
approximately $216$M parameters: $85$M for the encoder, $131$M for the
decoder, and $7$K for the renderer.

###  3.1 Encoder

Input images are first split into a set of fixed size non-overlapping patches
before being fed to the encoder, which predicts feature tokens using a ViT-B
transformer architecture. We formulate the encoder $\mathcal{E}$ as
$F,C=\mathcal{E}(I)$ where $I$ are input images and $F$ and $C$ are the
feature and class tokens respectively. In practice, we use the encoder of a
pre-trained MAE [25], a self-supervised vision learner trained to predict
masked image patches, which we further finetune on the training dataset to
adapt to the different image resolution from ImageNet, on which MAE [25] is
trained. As illustrated in Figure 3, we find that the self-supervised features
learnt by the MAE [25] encapsulate a powerful intermediate representation of
the scene’s geometric and appearance properties that can be leveraged by the
decoder to predict the radiance field. Note that the weights of the encoder
are continuously updated during end-to-end training, since we found that this
enhances the encoder’s ability to generate smoother and more segment-focused
features, as in Figure 3. This leads to the better performance of the model as
shown in Table 1.

Figure 4: Qualitative Results on Test (Unseen) Scenes: We show the
capabilities of NViST to synthesize novel views of unknown scenes. The model
correctly synthesizes images from different viewpoints of various categories
with diverse backgrounds and scales.

###  3.2 Decoder

The goal of the decoder $\mathcal{D}$ is to take the class and feature tokens,
$C$ and $F$, predicted by the encoder along with the camera parameters used as
conditioning (normalized focal length $f$ and camera distance $z$) and predict
the radiance field, which we encode using a vector-matrix (VM) representation,
parameterized with three matrices $M$ and three vectors $V$. A key feature of
our decoder is the use of cross-attention and self-attention mechanisms.
Unlike previous models [81] which project spatially aligned ResNet [24]
features onto target rays, our decoder learns this 2D to 3D correspondence by
learning to associate feature tokens with the relevant output tokens through
cross-attention.

The decoder $\mathcal{D}$ has output tokens $O$ as learnable parameters. For
simplicity, from here on we will use the term output tokens $O$ to refer to
the concatenation of output tokens and class token $C$, to which we further
add positional embeddings similarly to [71].

$M,V=\mathcal{D}(F,C,O,f,z).$ (1)

The decoder applies cross-attention between feature tokens $F$ and output
tokens $O$, and self-attention amongst output tokens $O$. The attention
mechanism allows the network to reason about occlusions, where the class token
$C$ acts as a global latent vector. For each attention block, we apply
adaptive layer normalisation [29, 50], instead of standard layer
normalisation, to condition on camera parameters.

####  3.2.1 Relative Camera Pose

A key strength of our approach is that it does not require all objects/scenes
in the dataset to be brought into alignment by expressing their pose with
respect to a canonical reference frame. Instead, we assume the rotation of the
camera associated with the input image $I_{i}$ to be the identity and we
express the rotation of any other image $I_{j}$ as a relative rotation
$R_{i\rightarrow j}$. We assume that the camera location of the input image is
$T_{i\rightarrow i}=(0,0,z)$, where $z$ is the normalized distance of the
input camera from the origin of the world coordinate frame, located at the
centroid of the 3D bounding box containing the point cloud, and
$T_{i\rightarrow j}={R_{i}}^{T}(T_{j}-T_{i})$.

####  3.2.2 Conditioning on Camera Parameters

There are several ways to apply conditioning such as concatenating camera
parameters to output tokens $O$ or feature tokens $F$. However, in NViST we
apply adaptive layer normalization, as the camera parameters influence the
overall scale of the scene. Conditioning camera parameters improves the model
performance as in Table 1.

Positional encoding:  We apply the positional encoding from NeRF [43] for $f$
and $z$, concatenating up to 4-th sine and cosine embeddings with the original
values $z$ and $f$ as $M=\oplus_{k=1}^{4}(\sin 2^{k}(f),\cos 2^{k}(f),\sin
2^{k}(z),\cos 2^{k}(z))$.

Adaptive Layer Normalisation:  We employ an additional MLP $\mathcal{A}$ to
regress the shift $\delta$, scale $\alpha$ and gate scale $\gamma$ from the
conditioning inputs $(z,f,M)$ for each attention block.

$\alpha,\delta,\gamma=\mathcal{A}(z,f,M)$ (2)

####  3.2.3 Attention Blocks

NViST uses self-attention blocks between output tokens $O$ and cross-attention
blocks between output tokens $O$ and feature tokens $F$. The embedding
dimension for both $O$ and $F$ is $e$. While standard layer normalization is
applied to feature tokens $F$, we apply adaptive layer normalization to output
tokens $O$ using the shift $\delta$ and scale $\alpha$ values regressed by
$\mathcal{A}$, the MLP described in Equation 2.

$\begin{split}O_{n}&=\delta+\alpha\times\text{Norm}(O)\\\ F_{n}&=\text{Layer
Norm}(F).\end{split}$ (3)

Cross-attention can then be expressed as

$\text{Attn}=\text{Softmax}(\frac{O_{n}F_{n}^{T}}{\sqrt{e}}).$ (4)

Finally output tokens are updated via residual connection $O\leftarrow
O+\gamma\times\text{Attn}\cdot F_{n}$, where $\gamma$ is the gate scale also
regressed by $\mathcal{A}$. Note that self-attention is obtained in an
equivalent way, just between output tokens.

Reshaping: We use MLPs to reshape the output tokens into the vector-matrix
representation that encodes the radiance field, adapting their respective
dimensionalities. This is followed by unpatchifying into the $3$ matrices and
$3$ vectors that form the VM representation. Please refer to the supplementary
material for more details.

###  3.3 Rendering

The VM representation of the radiance field predicted by the decoder is used
to query 3D point features which are then decoded by a multi-layer perception
into color $\mathbf{c}$ and density $\mathbf{\sigma}$ and finally rendered via
volumetric rendering.

Vector-Matrix Representation: For a compact yet expressive representation of
the radiance field, we adopt the vector-matrix decomposition proposed by
TensoRF [12] which expresses each voxel as the sum of three vectors and
matrices, one pair per axis. Specifically, a 3D feature grid ($\mathcal{T}$),
is decomposed into three vectors ($V_{r_{1}}^{X},V_{r_{2}}^{Y},V_{r_{3}}^{Z}$)
and three matrices ($M_{r_{1}}^{Y,Z},M_{r_{2}}^{Z,X},M_{r_{3}}^{X,Y}$), each
pair sharing the same channel dimensions ($k$) such that

$\mathcal{T}=\sum_{r_{1}=1}^{k}V^{X}_{r_{1}}\circ
M_{r_{1}}^{Y,Z}+\sum_{r_{2}=1}^{k}V^{Y}_{r_{2}}\circ
M_{r_{2}}^{Z,X}+\sum_{r_{3}=1}^{k}V_{r_{3}}^{Z}\circ M_{r_{3}}^{X,Y}.$ (5)

While the value of density $\mathbf{\sigma}$ is obtained by applying ReLU
activation directly to the feature value $\mathcal{T}_{\mathbf{x}}$ at point
$\mathbf{x}$, the colour $\mathbf{c}$ is predicted with a shallow MLP,
conditioned on the viewing direction. The VM representation outperforms the
method of aggregating features by projecting them onto three 2D planes and
then concatenating them as in Table 1.

Volume Rendering: For each sampled ray $r$, we obtain its final color
$\hat{C}(r)\in\mathbb{R}^{3}$ using volumetric rendering, following the
methodology of NeRF [43]. The transmittance $T_{i}$ is first computed at each
point $\mathbf{x}$ along the ray as
$T_{i}=\exp(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j})$, where $\delta_{i}$ is the
distance between adjacent points $\delta_{i}=t_{i+1}-t_{i}$. The pixel color
is calculated by integrating the predicted color at each $i$-th point
$\mathbf{c}_{i}$, weighted by light absorbance $T_{i}$ \- $T_{i+1}$.

$\hat{C}(r)=\sum_{i=1}^{\text{N}}T_{i}(1-\exp(-\sigma_{i}\delta_{i}))\mathbf{c}_{i}\vspace{-.1cm}$
(6)

###  3.4 Training Losses

We employ a combination of losses to train our architecture in an end-to-end
manner including the L2 photometric rendering loss, a Learned Perceptual Image
Patch Similarity (LPIPS) loss [83], and the distortion-based regulariser
proposed by [3].

Given ground-truth pixel colors $\bf{v}$, estimates $\bf{\hat{v}}$ and
accumulated transmittance values $\bf{w}$ along the points $\bf{x}$ on the
ray, our overall loss function is defined as

$\mathcal{L}=\mathcal{L}_{2}(\bf{\hat{v}},\bf{v})+\lambda\mathcal{L}_{\text{LPIPS}}(\bf{\hat{v}},\bf{v})+\beta\mathcal{L}_{\text{dist}}(\bf{w},\bf{x}).$
(7)

For MVImgNet we set $\lambda=0.1$ and $\beta=0.001$. We found that LPIPS to be
extremely effective for real world datasets as in Table 1.

##  4 Experimental Evaluation

###  4.1 MVImgNet

Train/Test Split: MVImgNet contains casually captured videos of over $6.5$
million real-world scenes across $238$ categories. Our training set, contains
a subset of $1.14$M frames across $38$K scenes of $177$ categories. For the
test set, we hold out every $100^{th}$ scene from each category to a total of
$13,228$ frames, from $447$ scenes and $177$ categories.

Pre-processing: MVImgNet utilises COLMAP to estimate camera matrices and
generate 3D point clouds. Since each scene has its own scale from SfM, we
rescale point clouds to a unit cube, such that the maximum distance along one
axis equals $1$.

Additionally, we center the point clouds in the world coordinate system. We
downsample the original images by $12$ while preserving the aspect ratio.
Camera intrinsics are recalibrated accordingly.

Iterations: We train the model with a single A100 GPU, for over 1 million
iterations, which takes approximately one week to complete. During training,
the encoder processes about 10 million images, which corresponds to
approximately $9$ epochs. For each iteration, we sample $15,000$ rays, and
$14,400$ rays are utilised for synthesising the full image to compute the
LPIPS loss, while the remaining $600$ rays are employed to sample from
different viewpoints.

Implementation Details: The encoder takes images of size $160\times 90$, uses
a patch size of $5$, so the total number of feature tokens is $576$. The
resolution of the vector-matrix representation is $48$. The patch size for the
decoder is $3$, the total number of output tokens $816$, and $16$ heads. The
embedding dimension is $768$ for both encoder and decoder, and we sample $64$
points along the ray. The initial learning rates are 2e-5 for encoder, and
1e-4 for decoder and renderer, and we decay them following the half-cycle
cosine schedule.

Figure 5: Results on Unseen Category: This figure shows how the model
generalises to a novel category unseen at training. We validate our model with
a held-out category (toy-cars). Figure 6: Casual Phone Captures: We
demonstrate the performance of NViST in out-of-distribution scenarios (images
taken with a commodity phone camera). The first row shows that the model can
capture scenes with different categories, and the second row demonstrates how
the model works in an outdoor setting. Third and fourth rows show that the
model performs well with many objects in the scene.

####  4.1.1 Qualitative Results

Results on Test (Unseen) Scenes: As depicted in Figure 4, our model
demonstrates its capability to synthesising new views of unknown scenes.
Figure 4 highlights the model’s ability to handle objects from diverse
categories, such as flowers, bags, helmets and others. We show that NViST can
deal with a variety of backgrounds, such as tables, wooden floors, textureless
walls or more cluttered environments.

Results on Unseen Category: We take a held-out category (toy-cars), and test
the ability of our model to generalize to categories unseen at training.
Figure 5 shows that NViST can synthesize new views of objects from a category
not seen at training time given a single input image.

Casual Phone Captures: Our motivation for this paper was to train a feed-
forward model that we could easily use on casually captured images, for
instance acquired with our own mobile devices. We test the ability of NViST to
deal with out-of-distribution scenes/images by performing zero-shot new-view
synthesis on scenes captured by us on a mobile-phone device. We pre-process
the images in the same manner as MVImgNet, using COLMAP to estimate the focal
length and camera distance parameters to be used as conditioning. Figure 6
shows results on out-of-distribution scenes. The top row highlights the
model’s ability to process a scene with multiple objects from diverse
categories. The second row reveals its competence on outdoor scenes, despite
their limited presence in the training set. The third and fourth rows
illustrate NViST’s ability to learn scenes with a large number of objects.
Figure 1 shows two further examples of results on phone captures.

Depth Estimation: Figure 7 shows qualitative results of the depth predicted by
NViST. Since there is no ground-truth depth for these images, we qualitatively
compare NViST with the recent MiDASv3.1 with Swin2-L384 [4]. We provide MiDAS
with the GT new view images as input, and scale the predicted depth to have
the same range as NViST. This shows that depth estimation with NViST performs
well even though it is not trained with depth supervision, while MiDAS uses
direct depth supervision from multiple datasets.

Figure 7: Depth Estimation: We show examples of the depth estimated on test images. Although NViST focuses on novel view synthesis and is trained with RGB losses only, the depth estimation is consistent and finds good object boundaries. We show a comparison with the state-of-the-art disparity estimator MiDAS v3.1 with Swin2-L384 [4]. We provide MiDAS with the GT new view images as input, and scale the predicted depth to the same range as NViST. This shows that depth estimation with NViST performs well even though it is not trained with depth supervision, while MiDAS uses direct depth supervision from multiple datasets. Figure 8: Qualitative Comparison with PixelNeRF: NViST displays better performance than PixelNeRF [81]. Note that NViST is based on a transformer architecture, uses a perceptual LPIPS loss, and is conditioned on camera parameters to deal with scale ambiguities. | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
Ours | 20.53 | 0.54 | 0.35
w/o LPIPS | 20.12 | 0.53 | 0.52
w/o Camera Conditioning | 19.24 | 0.49 | 0.41
w/o VM Representation | 18.10 | 0.46 | 0.50
w/o Updating Encoder | 18.24 | 0.47 | 0.52
Table 1: Ablation Study: LPIPS loss, conditioning on camera parameters,
updating encoder parameters, and VM representation are very effective. For w/o
VM Representation, we project features onto three 2D planes and concatenate
them.

Comparison with PixelNeRF: We compare with PixelNeRF [81] trained on MVImgNet
using the official code released by the authors. Note that we applied the same
pre-processing as NViST. Figure 8 shows that NViST performs better than
PixelNeRF, especially on occluded regions. While PixelNeRF only conditions on
spatially aligned features, NViST employs self-attention to deal with
occlusions.

####  4.1.2 Quantitative results

| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
---|---|---|---
PixelNeRF | 17.02 | 0.41 | 0.54
Ours | 20.53 | 0.54 | 0.35
Table 2: Quantitative Evaluations on MVImgNet: We compare NViST with PixelNeRF
in the context of novel view synthesis from single images in the wild. NViST
performs better than PixelNeRF on all metrics.

Ablation Study: We conducted an ablation study to analyse the effect of design
choices on the performance of NViST as summarised in Table 1. While the
perceptual LPIPS [83] loss is not commonly employed in NeRF-based methods, it
appears to play a crucial role in improving quality. Conditioning on
normalized focal length and camera distance helps the model deal with scale
ambiguities. While projecting features onto triplanes has been extensively
used before [8, 60, 36], our experiments show that the use of a vector-matrix
(VM) representation [12] improves the performance by a huge margin. Updating
encoder weights also improves the performance as the output from the encoder
is used for cross-attention.

Comparison with PixelNeRF: Table 2 shows a quantitative comparison of NViST
and PixelNeRF [81]. NViST outperforms PixelNeRF on every metric, as PixelNeRF
works well for nearby views but struggles to deal with occlusions. We could
not conduct quantitative comparisons with generative models such as GenVS [9]
or with the large-scale model LRM [28] as their models are not publicly
available. Moreover, GenVS is category-specific and LRM does not model
backgrounds.

Figure 9: Failure Cases This figure illustrates when the model fails to do new view synthesis properly. The toilet scene shows that the model learns geometry in a distorted way. In the motorcycle scene, the model fails to estimate the occluded area and the proper scale. | Cars | Chairs
---|---|---
| PSNR$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | LPIPS$\downarrow$
PixelNeRF [81] | 23.17 | 0.146 | 23.72 | 0.128
VisionNeRF [38] | 22.88 | 0.084 | 24.48 | 0.077
VD [68] | 23.29 | 0.094 | - | -
SSDNeRF [13] | 23.52 | 0.078 | 24.35 | 0.067
Ours | 23.91 | 0.122 | 24.50 | 0.090
Table 3: Quantitative Evaluation on ShapeNet-SRN  NViST performs similar to
other baselines while we adopt the relative pose. VD stands for Viewset
Diffusion. For qualitative comparisons, see supplementary materials.

###  4.2 ShapeNet-SRN

ShapeNet-SRN [66] has two categories (cars and chairs) and is a widely used
benchmark to compare models that perform novel view synthesis from a single
input image. Since all objects are aligned and there is no scale ambiguity,
pre-processing is not needed and we do not use the LPIPS loss as it is a
synthetic dataset. Table 3 shows that NViST performs similar or better to
baseline models on ShapeNet-SRN dataset. Although other models apply absolute
pose, we only employ relative pose, which means we do not fully exploit the
alignment of objects in ShapeNet-SRN.

##  5 Conclusion

In this paper, we introduced NViST, a transformer-based scalable model for new
view synthesis from single in-the-wild image inputs. Our evaluations
demonstrate robust performance on the MVImgNet test set, novel category
synthesis and phone captures containing out-of-distribution scenes. We also
validated our design choices and conducted a quantitative comparison on
MVImgNet and ShapeNet-SRN. Interesting future directions include applying our
model to other vision/graphics tasks such as segmentation or camera/object
pose estimation, extending NViST to handle 4D dynamic scenes, scaling up to
larger datasets, or integrating it with a latent diffusion model to allow
generation or editing of 3D scenes from text.

Limitations:  We believe that some of the loss of sharpness in the final
renders is due to our computational constraints (only 1 A100 GPU), which led
us to downsample images by $\times 12$, train on a fraction of the MVImgNet
dataset, and reduce the total number of iterations. We pushed for a model with
a transformer architecture without adversarial losses or score distillation
sampling [51]. This eased and sped up the training, but may have also
contributed to some loss of detail in the model outputs.

Acknowledgements:  The research presented here has been partly supported by a
sponsored research award from Cisco Research. This project made use of time on
HPC Tier 2 facilities Baskerville (funded by EPSRC EP/T022221/1 and operated
by ARC at the University of Birmingham) and JADE2 (funded by EPSRC
EP/T022205/1). We are grateful to Niloy Mitra and Danail Stoyanov for fruitful
discussions.

## References

  * Anciukevicius et al. [2022] Titas Anciukevicius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J. Mitra, and Paul Guerrero.  RenderDiffusion: Image diffusion for 3D reconstruction, inpainting and generation.  _arXiv_ , 2022.
  * Athar et al. [2022] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu.  Rignerf: Fully controllable neural 3d portraits.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Barron et al. [2022] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5470–5479, 2022.
  * Birkl et al. [2023] Reiner Birkl, Diana Wofk, and Matthias Müller.  Midas v3.1 – a model zoo for robust monocular relative depth estimation.  _arXiv preprint arXiv:2307.14460_ , 2023.
  * Cai et al. [2022] Shengqu Cai, Anton Obukhov, Dengxin Dai, and Luc Van Gool.  Pix2nerf: Unsupervised conditional p-gan for single image to neural radiance fields translation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3981–3990, 2022.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021.
  * Chan et al. [2020] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _arXiv_ , 2020.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16123–16133, 2022.
  * Chan et al. [2023] Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein.  Generative novel view synthesis with 3d-aware diffusion models.  _arXiv preprint arXiv:2304.02602_ , 2023.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2021] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14124–14133, 2021.
  * Chen et al. [2022a] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , pages 333–350. Springer, 2022a.
  * Chen et al. [2023a] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  _arXiv preprint arXiv:2304.06714_ , 2023a.
  * Chen et al. [2022b] Yue Chen, Xingyu Chen, Xuan Wang, Qi Zhang, Yu Guo, Ying Shan, and Fei Wang.  Local-to-global registration for bundle-adjusting neural radiance fields.  _arXiv preprint arXiv:2211.11505_ , 2022b.
  * Chen et al. [2023b] Yuedong Chen, Haofei Xu, Qianyi Wu, Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Explicit correspondence matching for generalizable neural radiance fields.  _arXiv preprint arXiv:2304.12294_ , 2023b.
  * Chng et al. [2022] Shin-Fang Chng, Sameera Ramasinghe, Jamie Sherrah, and Simon Lucey.  Gaussian activated neural radiance fields for high fidelity reconstruction and pose estimation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII_ , pages 264–280. Springer, 2022.
  * Deitke et al. [2023] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13142–13153, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Fridovich-Keil et al. [2023] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12479–12488, 2023.
  * Fu et al. [2022] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao.  Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation.  _arXiv preprint arXiv:2203.15224_ , 2022.
  * Gafni et al. [2021] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8649–8658, 2021.
  * Gu et al. [2023] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi.  Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion.  In _International Conference on Machine Learning_ , pages 11808–11826. PMLR, 2023.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009, 2022.
  * Henzler et al. [2021] Philipp Henzler, Jeremy Reizenstein, Patrick Labatut, Roman Shapovalov, Tobias Ritschel, Andrea Vedaldi, and David Novotny.  Unsupervised learning of 3d object categories from videos in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4700–4709, 2021.
  * Hong et al. [2022] Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang.  Headnerf: A real-time nerf-based parametric head model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20374–20384, 2022.
  * Hong et al. [2023] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan.  Lrm: Large reconstruction model for single image to 3d.  _arXiv preprint arXiv:2311.04400_ , 2023.
  * Huang and Belongie [2017] Xun Huang and Serge Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1501–1510, 2017.
  * Jang and Agapito [2021] Wonbong Jang and Lourdes Agapito.  Codenerf: Disentangled neural radiance fields for object categories.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12949–12958, 2021.
  * Jeong et al. [2021] Yoonwoo Jeong, Seokjun Ahn, Christopher Choy, Anima Anandkumar, Minsu Cho, and Jaesik Park.  Self-calibrating neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5846–5854, 2021.
  * Jun and Nichol [2023] Heewoo Jun and Alex Nichol.  Shap-e: Generating conditional 3d implicit functions.  _arXiv preprint arXiv:2305.02463_ , 2023.
  * Karnewar et al. [2023] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy J Mitra.  Holodiffusion: Training a 3d diffusion model using 2d images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18423–18433, 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kulhánek et al. [2022] Jonáš Kulhánek, Erik Derner, Torsten Sattler, and Robert Babuška.  Viewformer: Nerf-free neural rendering from few images using transformers.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV_ , pages 198–216. Springer, 2022.
  * Le et al. [2023] Eric-Tuan Le, Edward Bartrum, and Iasonas Kokkinos.  Stylemorph: Disentangled 3d-aware image synthesis with a 3d morphable styleGAN.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Lin et al. [2021] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey.  Barf: Bundle-adjusting neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5741–5751, 2021.
  * Lin et al. [2023] Kai-En Lin, Lin Yen-Chen, Wei-Sheng Lai, Tsung-Yi Lin, Yi-Chang Shih, and Ravi Ramamoorthi.  Vision transformer for nerf-based view synthesis from a single input image.  In _WACV_ , 2023.
  * Lin et al. [2022] Yunzhi Lin, Thomas Müller, Jonathan Tremblay, Bowen Wen, Stephen Tyree, Alex Evans, Patricio A Vela, and Stan Birchfield.  Parallel inversion of neural radiance fields for robust pose estimation.  _arXiv preprint arXiv:2210.10108_ , 2022.
  * Liu et al. [2023a] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization, 2023a.
  * Liu et al. [2023b] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9298–9309, 2023b.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Realfusion: 360deg reconstruction of any object from a single image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8446–8455, 2023.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16_ , pages 405–421. Springer, 2020.
  * Mohamed and Agapito [2023] Mirgahney Mohamed and Lourdes Agapito.  Dynamicsurf: Dynamic neural rgb-d surface reconstruction with an optimizable feature grid, 2023.
  * Müller et al. [2022a] Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder.  Autorf: Learning 3d object radiance fields from single view observations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3971–3980, 2022a.
  * Müller et al. [2022b] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022b.
  * Mur-Artal et al. [2015] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos.  Orb-slam: a versatile and accurate monocular slam system.  _IEEE transactions on robotics_ , 31(5):1147–1163, 2015.
  * Niemeyer et al. [2020] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3504–3515, 2020.
  * Oechsle et al. [2021] Michael Oechsle, Songyou Peng, and Andreas Geiger.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5589–5599, 2021.
  * Peebles and Xie [2022] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  _arXiv preprint arXiv:2212.09748_ , 2022.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Qian et al. [2023] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al.  Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors.  _arXiv preprint arXiv:2306.17843_ , 2023.
  * Ranftl et al. [2022] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(3), 2022.
  * Rebain et al. [2022] Daniel Rebain, Mark Matthews, Kwang Moo Yi, Dmitry Lagun, and Andrea Tagliasacchi.  Lolnerf: Learn from one look.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1558–1567, 2022.
  * Reizenstein et al. [2021] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10901–10911, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Sajjadi et al. [2022a] Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al.  Scene representation transformer: Geometry-free novel view synthesis through set-latent scene representations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6229–6238, 2022a.
  * Sajjadi et al. [2022b] Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučić, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf.  Object Scene Representation Transformer.  _NeurIPS_ , 2022b.
  * Sajjadi et al. [2023] Mehdi S. M. Sajjadi, Aravindh Mahendran, Thomas Kipf, Etienne Pot, Daniel Duckworth, Mario Lučić, and Klaus Greff.  RUST: Latent Neural Scene Representations from Unposed Imagery.  _CVPR_ , 2023.
  * Sargent et al. [2023] Kyle Sargent, Jing Yu Koh, Han Zhang, Huiwen Chang, Charles Herrmann, Pratul Srinivasan, Jiajun Wu, and Deqing Sun.  Vq3d: Learning a 3d-aware generative model on imagenet.  _arXiv preprint arXiv:2302.06833_ , 2023.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. [2016] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Schwarz et al. [2020] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3d-aware image synthesis.  _Advances in Neural Information Processing Systems_ , 33:20154–20166, 2020.
  * Shen et al. [2023] Bokui Shen, Xinchen Yan, Charles R Qi, Mahyar Najibi, Boyang Deng, Leonidas Guibas, Yin Zhou, and Dragomir Anguelov.  Gina-3d: Learning to generate implicit neural assets in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4913–4926, 2023.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Sitzmann et al. [2019] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Sun et al. [2022] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5459–5469, 2022.
  * Szymanowicz et al. [2023] Stanislaw Szymanowicz, Christian Rupprecht, and Andrea Vedaldi.  Viewset diffusion: (0-)image-conditioned 3D generative models from 2D data.  In _ICCV_ , 2023.
  * Tewari et al. [2023] Ayush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov, Joshua B. Tenenbaum, Frédo Durand, William T. Freeman, and Vincent Sitzmann.  Diffusion with forward models: Solving stochastic inverse problems without direct supervision.  In _arXiv_ , 2023.
  * Trevithick and Yang [2021] Alex Trevithick and Bo Yang.  Grf: Learning a general radiance field for 3d representation and rendering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15182–15192, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vora et al. [2021] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth.  Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes.  _arXiv preprint arXiv:2111.13260_ , 2021.
  * Wang et al. [2022] Jingwen Wang, Tymoteusz Bleja, and Lourdes Agapito.  Go-surf: Neural feature grid optimization for fast, high-fidelity rgb-d surface reconstruction.  In _2022 International Conference on 3D Vision (3DV)_ , pages 433–442. IEEE, 2022.
  * Wang et al. [2021a] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021a.
  * Wang et al. [2021b] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4690–4699, 2021b.
  * Watson et al. [2022] Daniel Watson, William Chan, Ricardo Martin-Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi.  Novel view synthesis with diffusion models.  _arXiv preprint arXiv:2210.04628_ , 2022.
  * Wu et al. [2023] Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichtenhofer, and Georgia Gkioxari.  Multiview compressive coding for 3D reconstruction.  _arXiv:2301.08247_ , 2023.
  * Yen-Chen et al. [2021] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung-Yi Lin.  inerf: Inverting neural radiance fields for pose estimation.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1323–1330. IEEE, 2021.
  * Yi et al. [2023] Brent Yi, Weijia Zeng, Sam Buchanan, and Yi Ma.  Canonical factors for hybrid neural fields.  In _International Conference on Computer Vision (ICCV)_ , 2023.
  * Yu et al. [2021a] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  _arXiv preprint arXiv:2112.05131_ , 2021a.
  * Yu et al. [2021b] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.  pixelnerf: Neural radiance fields from one or few images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4578–4587, 2021b.
  * Yu et al. [2023] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al.  Mvimgnet: A large-scale dataset of multi-view images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9150–9161, 2023.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhi et al. [2021] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15838–15847, 2021.

Supplementary Material

##  A Implementation Details

Finetuning MAE Encoder: We use the pre-trained MAE [25] with ViT-B [19] from
the original MAE implementation. Those weights are trained for ImageNet [18]
which has a resolution of $224\times 224$ pixels with a patch size $16$. This
means that the model divides the image into $196$ feature tokens. Our image
resolution for MVImgNet [82] is $160\times 90$, and we use an encoder patch
size of $5$, resulting in $576$ patches in the encoder. During fine-tuning, we
initialise the weights of attention blocks with the pre-trained MAE, as the
Transformer architecture allows for arbitrary attention matrix shapes as long
as the embedding dimension remains the same. We fine-tune by randomly masking
out and inpainting patches with L2 reconstruction loss,similar to the approach
used in MAE [25]. The process converges within a single epoch.

Initialisation of Decoder: We initialise the decoder of NViST with the fine-
tuned MAE weights. With the exception of the learnable parameters of
positional embedding of output tokens and the last MLP layers, we initialise
the weights of attention blocks with the fine-tuned MAE weights.

Number of output tokens: For MVImgNet [82], the resolution of vector-
matrix(VM) representation is $48$, and the channel dimension of each matrix
and vector is $32$. The patch size of the decoder is $3$. Each $48\times 48$
matrix $M$ consists of non-overlapping $16\times 16$ patches, and the $48$
dimensional vector $V$ is divided into $16$ patches. Therefore, the total
number of output tokens for VM representation is $818$.

Decoder MLPs and Reshaping: The embedding dimension of the decoder is $768$.
We have $818$ output tokens, and the channel dimension of VM representation
[12] is $32$, with a patch size of $3$ for decoder. For the output tokens
corresponding to the matrices $M$ in the VM representation, we deploy MLP to
reduce the embedding dimension to $288$. For those corresponding to vectors V,
we reduce it to $96$. Subsequently, we reshape them into VM representation.

##  B Qualitative Results on ShapeNet-SRN

We perform a qualitative comparison with VisionNeRF [38] on ShapeNet-SRN [66]
dataset as depicted in Figure 10. VisionNeRF, recognised as one of top-
performing models on ShapeNet-SRN, employs ViT [19] as its encoder. Notably,
VisionNeRF does not utilise any generative approaches, and was trained using
$8$ A100 GPUs. Similarly for MVImgNet, we fine-tune a MAE for the ShapeNet-SRN
dataset and initialise the parameters of both encoder and decoder of NViST
with this fine-tuned MAE for ShapeNet-SRN. The ShapeNet-SRN images are of
resolution $128\times 128$, and we use an encoding patch size of $8$,
resulting in $256$ feature tokens. The resolution of VM representation is
$64$, and the decoder patch size is $4$, so we use $818$ output tokens, each
with an embedding dimension of the Transformer as $768$. We still maintain the
relative pose but do not condition on camera parameters as the dataset is
aligned and does not have scale ambiguities. We train the model with a single
$3090$ GPU with $500,000$ and $700,000$ iterations, respectively for car and
chair.

Figure 10: Qualitative Comparison on ShapeNet-SRN [66]: NViST performs similar
to VisionNeRF which is one of the top-performing models on ShapeNet-SRN
dataset. Note that we do not employ LPIPS and do not condition on camera
parameters for ShapeNet-SRN as it is a synthetic dataset, but we still use the
relative pose even though objects are aligned in 3D.