# Analyzing the factors affecting usefulness of Self-Supervised Pre-trained
Representations for Speech Recognition

###### Abstract

Representations from well-trained Self-supervised learning (SSL) models have
become popular while building Automatic Speech Recognition (ASR) systems when
the amount of labelled data is scarce. Though widely used in practice,
_continued pre-training_ , or pre-training an already pre-trained model on in-
domain data, lacks systematic study. In this work, we first investigate how
factors such as domain, language and size of the upstream SSL pre-training
data affect the final downstream ASR performance. Finally, we investigate the
continued pre-training paradigm and analyze how upstream model selection plays
an essential role in this setup. Through thorough quantitative and qualitative
analysis, we reveal that the performance of ASR systems is sensitive to the
data used for SSL pre-training. Moreover, under extremely low-resource
conditions ($\leq$ 10 hours of fine-tuning data), leveraging features from SSL
models result in significant gains over traditional surface features like
FBank (up to 29% absolute WER improvement).

Index Terms—  low-resource, self-supervised learning, automatic speech
recognition

##  1 Introduction

One of the major problems with building efficient Automatic Speech Recognition
(ASR) systems is that they are data-hungry [1], and with the introduction of
deep learning-based methods, this problem has amplified. Though English has
more than 100k hours of human-transcribed data freely available online, for
languages beyond English, data is scarce. Some languages even lack
professional annotators and exist only in spoken form. With researchers and
businesses finding the true potential of ASR in various Natural Language
Understanding (NLU) systems, there has been an increasing demand for building
such systems in languages beyond English.

Though SSL models learn better task-agnostic features [2], they are not robust
to changes in the domain between the pre-training and ASR fine-tuning data, as
they get biased towards the domain of the unlabeled data [3]. This results in
the need for huge amounts of unlabeled data with a high source similarity to
the final downstream low-resource data in terms of accent, language, and
domain. Also, as rightly pointed out by [4], SSL would be more accessible if
it could be trained with lesser compute.

Main Results. This paper builds on our curiosity to answer the following
question. _“Can we use existing models available online, pre-trained on
thousands of hours of data, to improve performance on a rather low-resource
ASR task with minimal pre-training and fine-tuning steps?”_ and if so, _“How
would a shift in the domain, or language in our labeled fine-tuning data,
compared to the original pre-training unlabeled data, affect the performance
of our model for ASR?”_. Thus, building on the _continued pre-training_
paradigm, we use the 1000 hours of unlabeled data from the Interspeech
GramVaani ASR challenge for pre-training purposes. We use SSL models pre-
trained on varied domains and languages as upstream feature extractors.
Feature representations from these SSL models are then used for the downstream
ASR task. Our setup is very similar to [5], which was one of the first works
to explore speech-based SSL pre-trained models as upstream feature extractors
for raw speech. This pre-training and feature extraction paradigm has been
relatively under-studied, and our work contributes considerably in exploring
this better. Beyond achieving considerable improvements over our baselines,
contradictory to [3] where the analysis was done on a combined data setup, we
do the first of its kind analysis on the _continued pre-training_ setup, which
is more feasible in a real-world setting. Extensive experiments reveal that
source pre-trained data matters for the target _continued pre-training_ ,
significantly affecting the final downstream ASR task performance.

##  2 Related Work

A comprehensive study of SSL techniques on various Spoken Language Processing
(SLP) downstream tasks can be found in [2]. With research in this field
gaining attention, it has been of utmost importance to analyze the properties
of these learned features. We need to investigate the generalizability of
these features towards domain, language, and expand their capabilities to
build systems beyond English. Researchers have recently made reasonable
efforts to analyze the intrinsic properties of features learned through SSL at
various layers of the deep neural networks [6, 7]. [3] demonstrates that
wav2vec 2.0 tends to get biased towards the pre-training data, and that domain
similarity with the unlabeled data helps the final downstream ASR task. Along
similar lines, in this work, we try to analyze the effect of domain, multi-
linguality, and dataset size of the unlabeled data on our low-resource
downstream ASR task. Different from [3], we build on the _continued pre-
training_ paradigm [8] which we acknowledge is more practical in a real-world
setting than pre-training from scratch by combining the data.

##  3 Proposed Methodology

###  3.1 Model Architecture

Our experiments, including the baselines, use either Filter-Bank or wav2vec
2.0 models as the upstream feature extractors. We use the conformer-based [9]
Encoder-Decoder model as our downstream, which takes features from our
upstream as input and trains to learn the task of ASR.

###  3.2 Upstream

Previous to the self-supervised pre-training era, FBank-Pitch was among the
most common low-level feature extractor choices for building efficient ASR
systems [10, 11]. Thus, as shown in Fig.1, the 80-dimensional FBank-Pitch
features extracted from raw audio are directly fed to the downstream model for
training on the ASR task. With the recent dawn of SSL, the use of SSL pre-
trained models trained using Masked Acoustic Modelling (MAM) has shown to be a
better alternative to low-level features like FBank-Pitch [5]. As seen in
Fig.1, our setup is similar to [5], where either FBank-pitch features or
features from pre-trained SSL models are fed into the downstream model for the
ASR task. We resort to wav2vec 2.0 (LARGE) as the feature extractor for our
model, and to ensure a fair comparison with the FBank-Pitch feature extractor,
we do not fine-tune our upstream model parameters while training on ASR. We
acknowledge that fine-tuning the upstream might benefit the ASR task and
remains part of our future work. It is to be noted that Context
representations from the transformer encoder are used as the upstream features
from the wav2vec 2.0 models. The SSL pre-training details of wav2vec 2.0 are
beyond this paper’s scope, and we refer our readers to [12] for more details.

###  3.3 Downstream

As shown in Fig.1, our downstream ASR training is based on a joint
CTC/attention-based encoder-decoder architecture [13] and we conduct all our
downstream experiments using the ESPnet toolkit [14]. Our encoder is made up
of 12 conformer encoder blocks. While transformer models are well known for
capturing global information and data interactions, conformer blocks
effectively capture local as well as global information [9]. Our downstream
decoder uses 6 transformer blocks, and we train the models with a $0.4$ weight
to CTC.

Fig. 1: Illustration of the End-to-End ASR with various input speech
representations.

##  4 Experiments

###  4.1 Datasets

We experiment with two ASR datasets covering 4 Indian Languages. We use the
MSR (Microsoft Research) dataset, which was released as a part of the Low
Resource Speech Recognition Challenge for Indian Languages [15]. The dataset
consists of 50 hours of speech data with transcriptions for Gujarati, Tamil,
and Telugu amounting to 150 hours. For Hindi, we use the GramVaani dataset
[16], which was released as a part of the Interspeech GramVaani ASR Challenge
2022. It is a corpus of 1108 hours of real-world, spontaneous telephone speech
recordings in multiple dialects of the Hindi language. Of this 1108 hours of
data, 1000 hours is unlabeled, 100 hours is the labeled training data, and 5
hours of development data. Blind test data of 3 hours has been released for
evaluation. The audio files were recorded at different sampling rates ranging
from 8 kHz to 48 kHz in the mp3 format. For all the upstream pre-training /
continued pre-training of the wav2vec-2.0 models, we use the 991 hours of the
GramVaani unlabeled data for pre-training/continued pre-training and the rest
9 hours as a validation set. The 100-hour labeled data is used for the
downstream ASR training.

###  4.2 Baselines

As mentioned in section 3.3, all of the experiments have been carried out with
the same downstream model configuration, the structural details of which have
been mentioned in section 4.3. The baseline models differ from the rest of the
systems in terms of the upstream used. For the first baseline system, we use
80-dimensional FBank-Pitch features for training the downstream encoder-
decoder model. As the second baseline system, we pre-train a wav2vec-2.0 model
from scratch on the GramVaani unlabeled data and use it as the upstream
feature extractor to train the downstream encoder-decoder model.

###  4.3 Experimental Setup

Upstream Configuration. The wav2vec 2.0 models used as the upstream feature
extractors, have been pre-trained on data from different languages, domains,
and datasets of various sizes. We use wav2vec 2.0 LARGE models (24 transformer
layers) for all our experiments. Following are the descriptions of the SSL
models used, and the datasets they have been pre-trained on:

LL : The model is pre-trained on 60k hours of the Libri-Light dataset which is
a corpus of English audiobook data [12].

LL+CV+SF : The model is pre-trained on pooled multi-domain data sourced from
Libri-Light, CommonVoice [17], Switchboard [18] and Fisher datasets.

XLSR-128 : The model is pre-trained on 436k hours of unlabeled speech data
from 128 different languages [19].

IndicW2V : The model is pre-trained on 17k hours of multilingual data from 40
Indian languages [20]. Also, the data used has been sourced from varied
domains, including education, news, technology, and finance.

The 1024-dimensional representations extracted from these models, are passed
through a linear layer to obtain 80-dimensional features, which are then fed
to the downstream model for the End-to-End ASR training.

Downstream Configuration. Our downstream model is a conformer-based encoder-
decoder model. Our encoder has 12 conformer blocks with eight attention heads
and outputs a $512$-dimensional representation for each frame which it passes
to the decoder. Our decoder has 6 transformer blocks with eight attention
heads. While decoding, we analyze the effect of various upstreams independent
of the language model. During inference, we perform beam search decoding with
a beam size of 20. All the optimal hyper-parameters have been found via grid-
search.

Continued Pre-training (CP). To better adapt the above mentioned SSL pre-
trained models to the downstream tasks, we allow these models to continue pre-
training on the GramVaani unlabeled data. First introduced in [8], _continued
pre-training_ (CP) has proven to be quite effective for unsupervised domain
adaptation in NLP. All the pre-training and fine-tuning experiments have been
conducted using the Fairseq [21] and ESPnet toolkits [14] respectively, with 4
A-100 GPUs used for pre-training and 2 A-100 GPUs used for fine-tuning. All
the hyper-parameters for pre-training have been borrowed from the original
setting in [12, 21].

(a) Feature extractor _before_ continued pre-training

(b) Feature extractor _after_ continued pre-training

Fig. 2: (a), (b) shows Layer-wise attention weights for downstream fine-tuning with different wav2vec 2.0 feature extractors. “L0” is the convolutional feature encoder. Table 1: Performance of various E2E-ASR models with different upstream feature extractors Feature Extractor |  | Hindi |  | Gujarati |  | Tamil |  | Telugu
---|---|---|---|---|---|---|---|---
|  | dev | test |  | dev | test |  | dev | test |  | dev | test
Baseline |  |  |  |  |  |  |  |  |  |  |  |
1\. Fbank-Pitch |  | 34.2 | 33.7 |  | 27.8 | 35.3 |  | 30.1 | 29.8 |  | 32.8 | 32.9
2\. W2V-GV |  | 32.4 | 32.3 |  | 25.6 | 34.4 |  | 29.9 | 29.3 |  | 32.0 | 31.9
Pre-Trained Models |  |  |  |  |  |  |  |  |  |  |  |
3\. LL |  | 35.0 | 34.4 |  | 22.1 | 29.8 |  | 28.6 | 28.4 |  | 29.1 | 30.1
4\. LL+CV+SF |  | 34.3 | 34.2 |  | 22.0 | 29.5 |  | 28.2 | 27.9 |  | 28.5 | 28.9
5\. XLSR-128 |  | 32.7 | 32.5 |  | 21.7 | 28.5 |  | 28.1 | 27.7 |  | 28.3 | 28.8
6\. IndicW2V |  | 33.6 | 33.1 |  | 21.8 | 29.0 |  | 28.3 | 28.2 |  | 29.5 | 30.1
Continued Pre-Trained (CP) Models |  |  |  |  |  |  |  |  |  |  |  |
7\. LL |  | 29.7 | 29.8 |  | 22.2 | 29.5 |  | 29.0 | 28.4 |  | 29.1 | 29.6
8\. LL+CV+SF |  | 29.1 | 28.9 |  | 22.8 | 30.6 |  | 29.6 | 29.6 |  | 30.8 | 31.0
9\. XLSR-128 |  | 27.3 | 27.1 |  | 24.5 | 32.8 |  | 30.6 | 30.7 |  | 31.4 | 32.1
10\. IndicW2V |  | 31.4 | 31.5 |  | 24.0 | 32.8 |  | 31.0 | 30.6 |  | 31.6 | 32.2
Table 2: Performance of various E2E-ASR models with different upstream feature extractors using 10h labeled data Feature Extractor |  | Hindi
---|---|---
|  | dev | test
Baselines |  |  |
1\. Fbank-Pitch |  | 75.7 | 74.5
2\. W2V-GV |  | 50.4 | 50.3
3\. XLSR-128 |  | 56.8 | 55.9
4\. XLSR-128 (Continued pre-trained) |  | 46.2 | 46.0

##  5 Results and Analysis

Table 1 reports the %WER results for all our experiments across the 4 Indian
languages. W2V-GV indicates our second baseline system, where we pre-train the
wav2vec 2.0 model from scratch on the GramVaani unlabeled data and use it as
the upstream feature extractor. XLSR-128 as our upstream feature extractor
proves to be our best setup on Hindi. It outperforms our FBank-Pitch baselines
by 1.5%, and 1.2% WER on the dev and test sets. However, after _continued pre-
training_ XLSR-128 as a feature extractor outperforms the FBank-Pitch
baselines by 6.9%, and 6.6% WER.

Table 2 reports the %WER results with the amount of fine-tuning data limited
to 10 hours, sampled from the 100h GramVaani labeled data. XLSR-128 with
_continued pre-training_ which was the best performing feature extractor from
Table 1 outperforms our FBank-Pitch baseline by an absolute 29.5% WER and our
W2V-GV baseline by 4.2% WER, thus proving the benefits of SSL and continued
pre-training in extreme low-resource labeled data training regimes. We next
analyze how language, domain and dataset size factors affect the usefulness of
features obtained from pre-trained and continued pre-trained SSL models.

Feature Extractor’s layer importance: Since we use the weighted sum of
features from all the layers of our upstream model for our downstream ASR
fine-tuning, we try to analyze the importance given to each layer by our ASR
task, for the model fine-tuned on the Hindi dataset. Fig.2(a) and Fig.2(b)
show a pictorial representation of attention weights for each layer, for each
upstream model, before and after CP respectively. As we see, before CP our
task pays scattered attention across all layers based on the information
contained in these respective layers. For example, in lines with the
observation made by [6], that representation from layers 16-21 of a
wav2vec-2.0 LARGE model has the highest phonetic identity for IndicW2V which
has seen the highest amount of Hindi data even prior to CP. ASR fine-tuning
therefore draws the most information from these layers. Contrary to this, for
LL and LL+CV+SF, which were pre-trained entirely on a different language, our
task draws no information from these layers. However, for all the models after
CP, our downstream ASR task gives more attention to these layers than before,
which results in the model learning more language-specific phonetic
properties.

Upstream data volume and domains: As shown in Table 1, XLSR-128, which is pre-
trained on more than 436K hours of data, outperforms all the other pre-trained
models. After continued pre-training, a similar trend is observed for the
Hindi labeled data. Compared to other pre-trained models, XLSR-128 is pre-
trained on huge amounts of unlabeled data, giving a better initialization
before continued pre-training. Also from Table 1, LL+CV+SF performs better
than LL, and XLSR-128 performs better than IndicW2V across all the languages.
From this we conclude that models pre-trained on more domains learn better
domain-agnostic features.

Effectiveness of SSL features in low-resource data regimes: From the 100hours
GramVaani labeled data, we randomly sample 10hours of data for fine-tuning. As
shown in Table 2, the continued pre-trained XLSR-128 model results in an
average absolute improvement of 29% WER from our FBank-Pitch baseline on the
dev and the test set. From Table 1 we observe that using SSL features
benefitted relatively low resource languages like Gujarati, Tamil, and Telugu
(50h labeled data) compared to Hindi (100h labeled data). Therefore, using SSL
representations for low-resource fine-tuning proves to be more effective than
the FBank-Pitch features.

Continued pre-training on a different language doesn’t help: For multilingual
models such as XLSR-128, we observe an average hit of 3.5%, 2.8%, and 3.2% of
WER on Gujarati, Tamil, and Telugu data respectively, on the dev and test sets
after continued pre-training on the unlabeled Hindi data. We observe a similar
trend with IndicW2V. We hypothesize that for _multilingual models, continued
pre-training on a different language leads to catastrophic forgetting of the
in-domain knowledge_ [22]. As mentioned in [19], among the 128 languages,
XLSR-128 is pre-trained on 37 hours of Gujarati, 118 hours of Tamil, and 62
hours of Telugu. When we perform continued pre-training on a different
language, in the process of adapting to the new domain, the model fails to
retain the knowledge from its multilingual pre-training, leading to
catastrophic forgetting.

##  6 Conclusion

In this paper, we analyse the effects of domain, language, dataset size,
multilinguality and phonetic information sharing in SSL pre-trained models for
the downstream ASR fine-tuning setup. We study the impact of these factors
both with and without continued pre-training of the SSL models. We think our
work will be useful to the speech community, guiding the use of openly
available pre-trained models and promoting efficient model re-use. As part of
future work, we would like to devise better language and domain adaptation
methods for low-resource pre-training regimes.

## References

  * [1] Chiu et al.,  “State-of-the-art speech recognition with sequence-to-sequence models,”  in IEEE ICASSP 2018, pp. 4774–4778.
  * [2] Yang et al,  “Superb: Speech processing universal performance benchmark,”  in Interspeech 2021.
  * [3] Wei-Ning Hsu et al.,  “Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training,”  in Interspeech 2021, 2021, pp. 721–725.
  * [4] Awni Hannun,  “The history of speech recognition to the year 2030,”  arXiv preprint arXiv:2108.00084, 2021.
  * [5] Xuankai Chang, Takashi Maekaku, Pengcheng Guo, Jing Shi, Yen-Ju Lu, Aswin Shanmugam Subramanian, Tianzi Wang, Shu-wen Yang, Yu Tsao, Hung-yi Lee, et al.,  “An exploration of self-supervised pretrained representations for end-to-end speech recognition,”  arXiv preprint arXiv:2110.04590, 2021.
  * [6] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu,  “Layer-wise analysis of a self-supervised speech representation model,”  arXiv preprint arXiv:2107.04734, 2021.
  * [7] Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah,  “What all do audio transformer models hear? probing acoustic representations for language delivery and its structure,”  arXiv preprint arXiv:2101.00387, 2021.
  * [8] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith,  “Don’t stop pretraining: Adapt language models to domains and tasks,”  in ACL 2020.
  * [9] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [10] Andy T Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee,  “Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,”  in ICASSP 2020-2020. IEEE, 2020, pp. 6419–6423.
  * [11] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [12] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460, 2020.
  * [13] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi,  “Hybrid ctc/attention architecture for end-to-end speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
  * [14] Shinji Watanabe et al.,  “ESPnet: End-to-End Speech Processing Toolkit,”  in Interspeech 2018, pp. 2207–2211.
  * [15] Brij Mohan Lal Srivastava et al.,  “Interspeech 2018 Low Resource Automatic Speech Recognition Challenge for Indian Languages,”  in Proc. 6th Workshop on Spoken Language Technologies for Under-Resourced Languages (SLTU 2018), 2018, pp. 11–14.
  * [16] Bhanushali et al,  “Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi,”  in Proc. Interspeech 2022, 2022, pp. 3548–3552.
  * [17] Ardila et al.,  “Common voice: A massively-multilingual speech corpus,”  arXiv preprint arXiv:1912.06670, 2019.
  * [18] John J Godfrey, Edward C Holliman, and Jane McDaniel,  “Switchboard: Telephone speech corpus for research and development,”  in Acoustics, Speech, and Signal Processing, IEEE International Conference on. IEEE Computer Society, 1992, vol. 1, pp. 517–520.
  * [19] Arun Babu et al.,  “Xls-r: Self-supervised cross-lingual speech representation learning at scale,”  arXiv, vol. abs/2111.09296, 2021.
  * [20] Tahir Javed, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh M Khapra,  “Towards building asr systems for the next billion users,”  arXiv preprint arXiv:2111.03945, 2021.
  * [21] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli,  “fairseq: A fast, extensible toolkit for sequence modeling,”  in NAACL-HLT 2019 Demonstrations.
  * [22] Prakhar Kaushik, Adam Kortylewski, Alex Gain, and Alan Yuille,  “Understanding catastrophic forgetting and remembering in continual learning with optimal relevance mapping,”  in Fifth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems, 2021.


# VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams

Liao Wang1,3∗ Kaixin Yao1,3∗ Chengcheng Guo1 Zhirui Zhang1 Qiang Hu1 Jingyi
Yu1 Lan Xu1† Minye Wu2†
1 ShanghaiTech University 2 KU Leuven 3 NeuDim

###### Abstract

Neural Radiance Fields (NeRFs) excel in photorealistically rendering static
scenes. However, rendering dynamic, long-duration radiance fields on
ubiquitous devices remains challenging, due to data storage and computational
constraints. In this paper, we introduce VideoRF, the first approach to enable
real-time streaming and rendering of dynamic radiance fields on mobile
platforms. At the core is a serialized 2D feature image stream representing
the 4D radiance field all in one. We introduce a tailored training scheme
directly applied to this 2D domain to impose the temporal and spatial
redundancy of the feature image stream. By leveraging the redundancy, we show
that the feature image stream can be efficiently compressed by 2D video
codecs, which allows us to exploit video hardware accelerators to achieve
real-time decoding. On the other hand, based on the feature image stream, we
propose a novel rendering pipeline for VideoRF, which has specialized space
mappings to query radiance properties efficiently. Paired with a deferred
shading model, VideoRF has the capability of real-time rendering on mobile
devices thanks to its efficiency. We have developed a real-time interactive
player that enables online streaming and rendering of dynamic scenes, offering
a seamless and immersive free-viewpoint experience across a range of devices,
from desktops to mobile phones. Our project page is available at
https://aoliao12138.github.io/VideoRF/.

Figure 1: Our proposed VideoRF views dynamic radiance field as 2D feature
video streams combined with deferred rendering. This technique facilitates
hardware video codec and shader-based rendering, enabling smooth high-quality
rendering across diverse devices.

††* Authors contributed equally to this work. ${\dagger}$ The corresponding
authors are Minye Wu<EMAIL_ADDRESS>and Lan Xu
(xulan1@shanghaitech.edu.cn).

##  1 Introduction

Photorealistic Free-Viewpoint Video (FVV) of dynamic scenes offers an
immersive experience in virtual reality and telepresence. Work involving
Neural Radiance Fields (NeRFs) has shown great potential in creating
photorealistic Free-Viewpoint Videos (FVVs). However, there are still
challenges in smoothly delivering and rendering FVVs using NeRFs on commonly
used devices, similar to the ease of watching online videos. The difficulty
lies in reducing the data capacity for transmitting and storing long sequences
and ensuring a low, mobile-compatible computational load.

Neural Radiance Field (NeRF) [40] surpasses traditional 3D reconstruction
methods in photorealistic novel view synthesis. Several works extend NeRF to
dynamic scenes by maintaining a canonical space and matching it implicitly
[43, 49, 11] or explicitly [35] to align with each frame’s live space.
However, their dependence on canonical space limits effectiveness in sequences
with large motions or topological changes. Other methods introduce new
representations like 4D feature grids [19] or temporal voxel features [12],
achieving impressive results on the scenes with topological transformations.
Yet, [12] struggles with representing longer sequences owing to model capacity
constraints, while [19] encounters streaming challenges due to large storage
needs. Recent efforts [66, 55, 27] focus on compressing dynamic frames for
streaming, but their computational intensity limits mobile applicability.
Concurrently, on mobile devices, real-time rendering of static scenes has been
achieved by baking NeRF into mesh templates [7, 75, 53] or texture assets
[51]. However, these techniques fall short for dynamic scenes as their per-
frame representation becomes too bulky for real-time loading. Moreover, while
NeRF compression methods [52, 57, 66] can be employed, they introduce decoding
or rendering overheads unsuitable for mobile platforms. Despite the existence
of NeRF solutions for dynamic scenes and mobile optimization, a cohesive
approach that effectively addresses both still poses a difficulty.

In this paper, we propose VideoRF – a novel neural modeling approach that
enables real-time streaming and rendering of human-centric dynamic radiance
fields on common mobile devices (see Fig. 1). Our key idea is to view the 4D
feature volumes reconstructed from a dynamic scene as a 2D feature image
stream, which is friendly to video codec. Each feature image records the
densities and appearance features of one frame. We hence propose a rendering
pipeline for this representation. In this rendering pipeline, VideoRF uses a
set of mapping tables to connect the 3D space with the 2D feature images. This
allows for O(1) density and feature retrieval from the images for every 3D
sample point in the space. To reduce the computational complexity of
rendering, we adopt the deferred shading model [51] with a global tiny MLP to
decode the integrated feature vectors into pixel colors. All rendering
operations here are low-cost and compatible with the fragment shader model,
making it possible to implement this rendering pipeline on various devices
that have GPUs.

Second, we present a sequential training scheme to effectively generate the 2D
feature image stream. Specifically, VideoRF adaptively groups frames by
analyzing motion from the sequential data to ensure temporal stability for
mapping table generation. We deploy 3D and 2D Morton sorting techniques to
improve spatial consistency in the mapping. VideoRF contains spatial and
temporal continuity regularizers that are applied on the mapped 2D feature
images using the mapping table. This training strategy enforces the temporal
and spatial redundancy to the 2D feature image stream. We show that our
feature image stream with spatiotemporal sparsity can be efficiently
compressed by off-the-shelf video codecs and reaches high compression rates.

Moreover, based on these findings, we build a cross-platform player based on
VideoRF that can play FVVs in real-time, supported by video hardware
accelerators. In this way, users can interactively drag, rotate, pause, play,
fast-forward, rewind, and jump to specific frames, providing a viewing
experience as seamless as watching online videos on various devices, including
smartphones, tablets, laptops, and desktops which was unseen before.

To summarize, our contributions include:

  * •

We propose VideoRF, a novel approach to enable real-time dynamic radiance
field decoding, streaming and rendering on mobile devices.

  * •

We present an efficient and compact representation, which represents 4D
radiance field into 2D feature stream with low rendering complexity to support
hardware video codec and shader rendering.

  * •

We introduce a training scheme to directly impose spatial-temporal consistency
on our 2D feature stream for efficient compression.

##  2 Related work

Novel View Synthesis for Dynamic Scenes. Dynamic scenes pose a greater
challenge in achieving realistic view synthesis results due to moving objects.
One approach is to reconstruct the dynamic scene and render the geometry from
new viewpoints. The conventional RGB [24, 8, 50, 38, 31, 37, 80] or RGB-D [10,
22, 70, 42, 71, 20, 21] solutions have been extensively investigated. Mesh-
based neural network [68, 62, 5] techniques are effective for compact data
storage and can record view-dependent texture [68, 5], but such methods
heavily rely on geometry, especially perform poorly for topologically complex
scenarios.

Many methods extend NeRF into the dynamic view synthesis settings. Some
methods [49, 32, 69, 79, 16, 11, 44, 64, 30, 34, 77, 33] handle spatial change
directly conditions on time and [46, 47, 15, 73, 82] use feature latent code
to represent time information. However, they do not support the streaming of
long sequences due to the limited representation. Others learn spatial offsets
from the live scene to the canonical radiance field by using explicit voxel
matching [35], skeletal poses [36, 45, 26], deformed hashtable [25] and
deformed volume [23, 60, 81, 72, 61, 67]. While these methods can successfully
render dynamic scenes with photo-realistic quality, their heavy reliance on
the canonical space makes these methods vulnerable to long sequences, large
motions, and topological alterations. Recently, several methods using 4D
planes [14, 54, 2, 19, 74], voxel grid [12], Fourier representation [65],
residual layers [39] or dynamic MLP maps [48] to represent dynamic scene.
However, the inference computational complexity makes it difficult for them to
implement streaming and decoding on mobile devices. Additionally, several
methods such as [55, 66, 27] have managed to facilitate the streaming of
dynamic radiance fields by reducing the capacity of each frame, but the lack
of general hardware acceleration makes it challenging to represent dynamic
scenes in real-time on mobile phones.
Cross Device Neural Radiance Field Rendering. Recent works have demonstrated
that static neural radiance fields can be rendered on mobile devices. They
achieve this by directly converting the neural radiance field to mesh [59] or
using traditional texture to represent the scene. [7, 78, 75] have endeavored
to augment the capabilities of neural rendering by employing mesh templates
and feature texture, [53, 1] facilitating the rendering process through
fragment shader to achieve surface-like rendering with similar ideas. [63]
represents neural radiance features encoded on a two-layer duplex mesh for
better rendering quality.

Also, technologies [51, 2, 3] embody the principle of utilizing implicit
neural representations to handle radiance rendering on mobiles. Although all
these methods are fairly effective at compressing static scenes and enabling
rendering on mobile devices, they still do not support streaming for dynamic
scenes, particularly in the case of long sequences of dynamic scenes, they
will still occupy a large amount of storage space.

Figure 2: Demonstration of our rendering. For each 3D sample point, its
density $\sigma$ and feature $\mathbf{f}$ are fetched from the 2D feature
image through the mapping table $\mathbf{M}$. Each point feature is first
volumetrically accumulated to get the ray feature $\tilde{\mathbf{f}}$ and
pass MLP $\Phi$ to decode the ray color.

NeRF Acceleration and Compression. NeRF demonstrates exceptional performance
in generating images from arbitrary viewpoints, but it suffers from slow
rendering efficiency. Some methods focus on integrating a compact structure
with a simplified MLP, which reduces the complexity of MLP calculations in
traditional NeRFs. Key strategies have been explored, including the employment
of voxel grid [56, 27], octrees [76, 13, 65], tri-planes [4], hashing encoding
[41], codebook [57, 28], tensor decomposition [6, 58] to accomplish this.
Using explicit structure makes training and rendering faster, but it also
means that the 3D structure takes up more storage space.

Therefore, while methods such as CP-decomposition [6], rank reduction [58],
and vector quantization [57] manage to attain modest levels of data
compression, their application remains confined to static scenes.
Additionally, some approaches [66, 52, 9] opt for post-processing techniques,
initially utilizing baseline methods to train a decent NeRF representation,
then special encoding and decoding schemes are employed to reduce the storage
space. However, these kinds of methods are not hardware-friendly, the
inference process is computationally demanding and cannot support real-time
streaming on mobile devices. In contrast, our VideoRF uses codec tailored
training, hardware-friendly decoding and shader rendering, which enables real-
time dynamic radiance field for long sequences with large motion.

##  3 VideoRF Representation

To facilitate dynamic radiance field rendering on mobile devices, we adopt a
representation that aligns with the video codec format and maintains low
computational complexity for shader-based rendering. We propose turning the 3D
volume representation into a 2D formulation, coupled with a highly efficient
rendering pipeline.

In our approach, each frame of the radiance field is represented as a feature
image $\mathbf{I}$ where the first channel stores density and the remaining
$h$ channels store feature. As depicted in Fig. 2, given a 3D vertex position
$\mathbf{x}$, we retrieve its density $\sigma$ and feature $\mathbf{f}$ using
the equation:

$\sigma,\mathbf{f}=\mathbf{I}[\mathbf{M}(\mathbf{x})],$ (1)

where $\mathbf{M}$ is the 3D-to-2D mapping table from Sec. 4.1. This mapping
table not only effectively excludes empty space to reduce storage but also
specifics mapping from each non-empty 3D vertex to a corresponding 2D pixel.
The lookup operation is highly efficient, with a time complexity of O(1),
facilitating rapid and convenient queries. It’s worth noting that our sequence
of feature images is in a 2D format, which is friendly to the video codec
hardware.

Figure 3: Overview of our video codec-friendly training. First, we apply our
grid-based coarse training [56] to generate per-frame occupancy grid
$\mathbf{O}^{t}$. Then, during baking, we adaptively group each frame and
create a mapping table $\mathbf{M}$ for each group. Next, we sequentially
train each feature image $\mathbf{I}^{t}$ through our spatial, temporal and
photometric loss. Finally, feature images are compressed into the feature
video streaming to the player.

For rendering, inspired by [51, 18], we use a deferred rendering model. We
first accumulate the features along the ray:

$\begin{split}\tilde{\mathbf{f}}(\mathbf{r})&=\sum_{k=1}^{n_{s}}T_{k}(1-\exp(-\sigma_{k}\delta_{k}))\mathbf{f}_{k},\\\
T_{k}&=\operatorname{exp}\left(-\sum_{j=1}^{k-1}\sigma_{j}\delta_{j}\right),\end{split}\vspace{-20pt}$
(2)

where $n_{s}$ is the number of sample points along the ray $\mathbf{r}$.
$\sigma_{k}$, $\mathbf{f}_{k}$, $\delta_{k}$ denotes the density, feature of
the samples, and the interval between adjacent samples respectively. The view-
dependent color of the ray is then computed using a tiny global MLP $\Phi$
shared across the frames as:

$\tilde{C}(\mathbf{r})=\operatorname{sigmoid}\left(\Phi\left(\tilde{\mathbf{f}}(\mathbf{r}),\mathbf{d}\right)\right),$
(3)

where $\mathbf{d}$ is the view direction of the ray after positional encoding
[40]. In this way, the computational burden is significantly reduced as each
ray requires only a single MLP decoding which can be implemented in a shader
for real-time rendering on mobile devices.

##  4 Video Codec-friendly Training

In this section, we propose a training scheme, as depicted in Fig. 3, to
achieve a high compression rate by maintaining spatial and temporal
consistency. For spatial aspect, we incorporate 3D-2D Morton sorting to
preserve 3D continuity and apply a spatial consistency loss directly on 2D
feature images to enforce spatial coherence. On the temporal front, we employ
adaptive grouping, which allows frames within a group to share mappings,
thereby reducing temporal disruptions, and temporal consistency loss, which
serves to further reinforce temporal coherence. These consistencies are
crucial which provide a stable and predictable layout for video codecs.

###  4.1 Baking and Map Generation

Coarse stage pre-training. Given multiview images of each frame, we first
adopt an off-the-shelf approach [56] to generate the explicit density grid
$\mathbf{V}_{\sigma}^{t}$ for each frame $t$ independently. We then create a
per-frame occupancy grid $\mathbf{O}^{t}$ by masking in voxels whose opacity
exceeds a certain threshold $\gamma$. This coarse stage sets the foundation
for our subsequent adaptive grouping in the baking stage.
Adaptive group. Generating an independent mapping table for each frame $t$
could disrupt temporal continuity, leading to increased storage requirements
after video codec compression. This issue arises because the same 3D point in
adjacent frames might map to vastly different 2D positions on the feature
image. On the other hand, uniformly applying a single mapping table across all
frames could introduce significant spatial redundancy. Considering that a 3D
point may only be occupied at sparse intervals, it could remain underutilized
for most of the duration, resulting in inefficiency. To balance these factors,
we divide the sequence into Groups of Frames (GOFs), maintaining a fixed
resolution for our 2D feature image and adaptively determining the number of
frames in each group. For a set of consecutive frames
$\left\\{i,i+1,\ldots,i+n\right\\}$, we identify the maximum frame number
$\alpha$ such that the number of occupied voxels in the union from $i$ to
$\alpha$ does not exceed our pixel limit $\theta$:

$\operatorname{argmax}_{\alpha}\operatorname{g}\left(\bigcup_{j=i}^{\alpha}\mathbf{O}^{j}\right)\leqslant\theta,$
(4)

where $\operatorname{g}()$ means the number of occupied grids in the union
occupancy grid. Then, we will set the frame $i$ to frame $\alpha$ as a GOF and
frame $\alpha+1$ to be the start frame of a new GOF. In this way, all the
frames in the group share the same mapping. The same 3D position within the
group will be mapped to the same 2D pixel location which keeps temporal
consistency and saves the storage.
3D Morton sorting. After the union occupancy grid is achieved, we apply Morton
sorting to record its 3D spatial continuity. Morton ordering, or Z-ordering,
interleaves the binary representations of spatial coordinates, ensuring that
spatially proximate entities remain adjacent in linear space. As shown in Fig.
4(a), we apply 3D Morton sorting to the vertices based on their position
coordinates of the union occupancy grid. Vertices with a density below the
threshold $\gamma$ are excluded. This process effectively maintains 3D spatial
consistency in the ordering.

Figure 4: Illustration of our mapping table generation. We first perform 3D
Morton sorting (a) on each nonempty vertice and group it into chunks $C_{i}$
(b). Next, we lay out each chunk into each block $B_{i}$ of the feature image,
arranged in 2D Morton order (c) within it.

2D block partitioning and 2D Morton sorting. To preserve the 3D spatial
continuity within a 2D framework, we employ 2D Morton sorting and partition
the feature images into blocks. This approach aligns with the block-wise
compression of frames in video codecs, where blocks with local smoothness lead
to more efficient storage. Specifically, we first divide the feature image
into $N$ $8\times 8$ blocks, denoted as $B_{i}$, and correspondingly group the
sorted vertices into $N$ $8\times 8$ chunks, denoted as $C_{i}$, as
illustrated in Fig. 4(b). For each pixel $\mathbf{p}$ in a block $B_{i}$, we
sort its relative position $(u,v)$ in 2D Morton order. As shown in Fig. 4(c),
each chunk $C_{i}$ is then mapped to a block $B_{i}$, arranged in 2D Morton
order within the block, to form the mapping table $\mathbf{M}$. This 2D Morton
ordering ensures that sorted values in the 3D Morton ordering are positioned
closely within each feature image block, facilitating efficient compression
during the transformation process.

###  4.2 Fine-grained Sequential Training

With the aid of our mapping table, we can sequentially train our feature
images within each group through spatial consistency loss applied to 2D
feature images and temporal consistency loss between frames.
Spatial consistency loss. In video encoding, regions with homogeneous
characteristics demonstrate high compression efficiency. This efficiency stems
from the minimal variation in pixel values within these areas, leading to a
significant reduction in high-frequency components post Discrete Cosine
Transform (DCT) within the video codec. Consequently, these regions retain a
higher proportion of low-frequency components. Furthermore, the quantized
coefficients in such homogeneous regions are more likely to contain a greater
number of zero values, facilitating a more compact data representation and
reducing the overall data volume. In order to enhance the homogeneity of our
2D feature image, we introduce a total variance loss,
$\mathcal{L}_{\mathrm{spatial}}$, during our fine-grained training stage. For
each channel of the feature image $\mathbf{I}$, we enforce its local
smoothness by:

$\mathcal{L}_{\mathrm{spatial}}=\frac{1}{|\mathcal{P}|}\sum_{\begin{subarray}{c}\mathbf{p}\in\mathcal{V}\end{subarray}}\left(\Delta_{u}(\mathbf{p})+\Delta_{v}(\mathbf{p})\right),$
(5)

where $\mathcal{P}$ is the pixels of the feature image,
$\Delta_{u}(\mathbf{p})$ shorthand for Manhattan distance between the feature
value at pixel $\mathbf{p}:=(u,v)$ and the feature value at pixel $(u+1,v)$
normalized by the resolution, and analogously for $\Delta_{v}(\mathbf{p})$. By
increasing the spatial sparsity, the storage of feature videos after video
encoding is decreased at the same quality.
Temporal consistency loss. A naive per-frame training scheme will disrupt
temporal continuities by failing to incorporate inter-frame information and
resulting in a high bitrate. This is because the residuals between frames are
stored after entropy encoding. To optimize storage efficiency, we focus on
minimizing the differences in the feature space between the adjacent frames to
reduce the entropy. During our sequential training, we enhance inter-frame
similarities by regularizing the current feature image with its predecessor,
except for the initial frame of each adaptive group. This is achieved by
applying

$\mathcal{L}_{\mathrm{temporal}}=\|\mathbf{I}^{t}-\mathbf{I}^{t-1}\|_{1},$ (6)

for each frame $t$ in the group, ensuring small residuals between consecutive
feature images. By employing temporal smoothness in this manner, we can
further mitigate entropy throughout the video codec process, thereby
facilitating storage conservation.
Training objective. Our total loss function is formulated as:

$\mathcal{L}_{\text{total
}}=\mathcal{L}_{\mathrm{rgb}}+\lambda_{s}\mathcal{L}_{\mathrm{spatial}}+\lambda_{t}\mathcal{L}_{\mathrm{temporal}},$
(7)

where $\lambda_{s}$ and $\lambda_{t}$ are the weights for our regular terms
and $\mathcal{L}_{\mathrm{rgb}}$ is the photometric loss,

$\mathcal{L}_{\text{rgb}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{c}(\mathbf{r})-\hat{\mathbf{c}}(\mathbf{r})\|^{2},$
(8)

where $\mathcal{R}$ is the set of training pixel rays;
$\mathbf{c}(\mathbf{r})$ and $\mathbf{\hat{c}}(\mathbf{r})$ are the ground
truth color and predicted color of a ray $\mathbf{r}$ respectively.

Figure 5: Our VideoRF method generates results for inward-facing, 360∘ video
sequences featuring human-object interactions with large motion. The images in
the last row illustrate our ability to implement variable bitrate in these
sequences.

###  4.3 VideoRF Player

Finally, we implement a companion VideoRF player to stream and render dynamic
radiance fields on mobile devices. Given our feature stream, we quantize it
into uint8 format and use H.264 for video codec. Note that the total
resolution of feature images is under a 4K color image, ensuring seamless
real-time streaming and decoding even on mobile devices.

For the rendering part, we implement it via a fragment shader. For fast
raymarching, we employ a multi-resolution hierarchy of occupancy grids for
each group to skip empty space at different levels. We leverage matrix
multiplication in the shader to simulate our tiny MLP calculation. Different
from [51], our method stores the mapping table in a 2D-to-3D format as an RGB
image to save storage redundancy. Meanwhile, to speed up querying values for
each sampled point, we adopt a parallel approach to expand a 2D feature image
into a 3D volume in the player. These techniques increase the overall speed
while ensuring compact storage.

The VideoRF player marks a significant milestone, as it first enables users to
experience real-time rendering of dynamic radiance fields of any length.
Within this player, users can drag, rotate, pause, play, fast
forward/backward, seek dynamic scenes, or switch between different resolutions
like watching online video, offering an extraordinary high-quality free-
viewpoint viewing experience. This capability also extends across a wide range
of devices, from smartphones and tablets to laptops and desktops, broadening
the accessibility and applicability of dynamic radiance fields.

##  5 Experimental Results

In this section, we evaluate our VideoRF on a variety of challenging
scenarios. We use the PyTorch Framework to train the model on a single NVIDIA
GeForce RTX3090. Our new captured dynamic datasets contain around 80 views at
the resolution of 1920 ${\times}$ 1080 at 30 fps. To ensure the robustness of
the algorithm, we also use the ReRF dataset and the HumanRF dataset for result
demonstration in Fig. 5. We can render images for immersive, 360∘ video
sequences that capture human interactions with objects, especially for large
motion and long duration sequences. Please refer to the supplementary video
for more video results.

Figure 6: Qualitative comparison against dynamic scene reconstruction methods
and per frame static reconstruction methods.

###  5.1 Comparison

To the best of our knowledge, our approach is the first real-time dynamic
radiance field approach that can decode and render on mobile devices.
Therefore, we compare to existing dynamic neural rendering methods, including
ReRF [66], HumanRF [19], TiNeuVox [12] and per-frame static reconstruction
methods like MeRF [51]. We use ReRF [66] dataset and HumanRF [19] Actor3
sequence 1 for a fair comparison, both in storage memory and image quality. As
shown in Fig. 6, though TiNeuVox [12] has small storage, it faces the
challenge of intensifying blurring effects as the frame count rises. MeRF [51]
suffers from the large storage which is not suitable for dynamic scenes when
streaming and decoding for playing. HumanRF [19] though capable of
representing dynamic scenes, also faces difficulties in streaming and
rendering on mobiles due to its computation and storage load.

best second-best
---
Dataset | Method |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS $\downarrow$ |  Size(MB)$\downarrow$
ReRF |  ReRF [66] | 31.84 | 0.974 | 0.042 | 0.645
MeRF [51] | 31.12 | 0.975 | 0.030 | 7.529
HumanRF [19] | 28.82 | 0.900 | 0.069 | 2.800
TiNeuVox[12] | 22.70 | 0.923 | 0.083 | 0.337
Ours | 32.01 | 0.976 | 0.023 | 0.658
Actors-HQ |  ReRF [66] | 28.33 | 0.836 | 0.296 | 0.554
MeRF [51] | 27.22 | 0.807 | 0.271 | 7.164
HumanRF [19] | 28.98 | 0.888 | 0.151 | 2.286
TiNeuVox [12] | 22.98 | 0.752 | 0.430 | 0.447
Ours | 28.46 | 0.838 | 0.278 | 0.426
Table 1: Qualitative comparison against dynamic scene reconstruction methods
and per frame static reconstruction methods.

We also conduct a quantitative comparison using metrics such as the peak
signal-to-noise ratio (PSNR), structural similarity index (SSIM), and the
Learned Perceptual Image Patch Similarity (LPIPS) as metrics shown in Tab. 1.
For a fair comparison in our experiments, in the ReRF dataset, we test the
first 200 frames on the scene Kpop. We use 6 and 39 as test views and the
others as training views. As for the HumanRF dataset, we apply the test
methods as referenced in HumanRF on the first 250 frames on Actor3, sequence
1. We utilize the values reported for HumanRF and TiNeuVox methods directly
from the HumanRF study. In the ReRF dataset, which includes large motion
scenarios, we match the rendering quality of the standard ReRF, while also
maintaining a compact data storage size. Meanwhile, in the HumanRF dataset
with relatively small motion, we achieve a rendering quality that is second
only to HumanRF, accomplished with the most efficient storage utilization.
It’s worth noting that, our method is the only method that can provide both
mobile and dynamic rendering tasks. The existing dynamic reconstruction
methods like TiNeuVox [12], HumanRF [19] and ReRF [66] all cannot effectively
utilize hardware encoding and decoding techniques, preventing them from being
displayed on mobile devices, especially for the dynamic scene of a long
sequence.

Figure 7: Rate distortion curve. The rate distortion curve illustrates the
efficiency of various components within our system. We use different
quantization factors to obtain the average rendering quality at different
capacities. Our full model stands out as the most compact, allowing for
flexible bitrate adjustments to meet diverse storage needs. Figure 8:
Qualitative evaluation of different variations in our method at 600KB.

###  5.2 Evaluation

Ablation study. We analyze the impact of spatial consistency, temporal
consistency, Morton sorting, block storage, and adaptive grouping on
compression and rendering. In the case of models that do not employ 3D and 2D
Morton sorting, spatial point data is sorted sequentially in row-major order
while employing 2D block-wise storage. For models not utilizing block storage,
a 3D spatial Morton sorting is applied, followed by storage in a row-major
format within the 2D feature space. For models without adaptive grouping, the
mapping table for each frame is calculated only based on the occupancy grid of
the current frame. As illustrated in Fig. 7, our full model exhibits the best
performance in terms of rendering quality and storage efficiency. This
demonstrates the effectiveness of our modules in reducing capacity by
maintaining consistency across time and space. While ReRF [66] exhibits
slightly better quality in the storage range of 700-800KB, our method
consistently demonstrates higher PSNR across most capacity ranges and supports
mobile rendering. Fig. 8 shows that under a 600KB storage limit, our complete
model yields more realistic results with less compression blur.
Cross device runtime analysis. We evaluate the runtime breakdown analysis of
VideoRF as detailed in Tab. 2. Our experimental setup includes a Desktop with
an i7-12700F CPU and NVIDIA RTX3090 GPU, a Laptop with an i5-1135G7 CPU and
Integrated GPU, a Tablet (iPad Pro) with an M2 chip, and a phone (iPhone 14
Pro) with an A16 Bionic chip. For Desktops and Laptops, we employ a Python
code base along with ModernGL. Meanwhile, for Tablet and Smartphone, the
VideoRF player is developed using Swift and Metal. It’s worth noting that the
video decoding part primarily relies on CPU performance, while the rendering
part mainly depends on GPU performance. These two parts operate asynchronously
and simultaneously. The other part mainly covers operations such as data
conversion between the GPU and CPU. Our results demonstrate that VideoRF
allows users to enjoy free-view videos at high frame rates on multiple
devices, providing an experience that rivals the smoothness of watching 2D
videos on platforms such as YouTube.

Devices | FPS | Video Decoding | Rendering | Others
---|---|---|---|---
Desktop | 116 | 5.548 ms | 3.602 ms | 3.072 ms
Laptop | 25 | 10.54 ms | 29.31 ms | 11.69 ms
Tablet | 40 | 3.822 ms | 23.86 ms | 1.150 ms
Phone | 23 | 13.38 ms | 40.82 ms | 3.003 ms
Table 2: Runtime analysis across different devices when processing the same HD
image with a resolution of 1920 $\times$ 1080.

Storage of different components analysis. We present the storage requirements
of each VideoRF component in Tab. 3. This encompasses the average file sizes
for several key elements: the feature images for model detail, 3D-to-2D
mapping table, occupancy images used to efficiently skip over empty spaces,
and MLP parameters for the neural network. Note that our model’s total average
size is a mere 669.78KB. This compact representation facilitates rapid
streaming across various devices.

Components | Size(KB)
---|---
Feature Images | 661.62
3D to 2D Mapping Table | 2.58
Occupancy Images | 2.18
MLP Parameters | 3.40
Total Size | 669.78
Table 3: Storage of different components. The result is averaged over a
sequence of Kpop scene from ReRF [66] dataset.

##  6 Discussion

Limitation. As the first trial to enable a real-time dynamic radiance field
approach capable of decoding and rendering on mobile devices, our approach
presents certain limitations. First, high-quality reconstruction requires a
complex system that captures multiple views, which is notably expensive.
Second, our training time increases with the number of frames, especially for
long sequences. Additionally, our method currently lacks support for temporal
interpolation, signifying a direction for future exploration.
Conclusion. We have presented VideoRF, a novel approach enabling real-time
streaming and rendering of dynamic radiance fields on mobile devices. Our
VideoRF innovatively processes feature volumes as 2D feature streams and
adopts deferred rendering to effectively leverage classical video codecs and
rasterization pipelines. Our video codec friendly training scheme is
implemented on 2D feature space to enhance spatial-temporal consistency for
compactness. Additionally, our tailored player supports seamless streaming and
rendering of dynamic radiance fields across a range of devices, from desktops
to smartphones. Our experiments demonstrate its capability for compact and
effective dynamic scene modeling. With the unique ability of real-time
rendering of dynamic radiance fields on mobile devices, we believe that our
approach marks a significant step forward in neural scene modeling and
immersive VR/AR applications.

## References

  * Božič et al. [2022] Aljaž Božič, Denis Gladkov, Luke Doukakis, and Christoph Lassner.  Neural assets: Volumetric object capture and rendering for interactive environments, 2022.
  * Cao and Johnson [2023] Ang Cao and Justin Johnson.  Hexplane: A fast representation for dynamic scenes.  _CVPR_ , 2023.
  * Cao et al. [2023] Junli Cao, Huan Wang, Pavlo Chemerys, Vladislav Shakhrai, Ju Hu, Yun Fu, Denys Makoviichuk, Sergey Tulyakov, and Jian Ren.  Real-time neural light field on mobile devices, 2023.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16123–16133, 2022.
  * Chen et al. [2018] Anpei Chen, Minye Wu, Yingliang Zhang, Nianyi Li, Jie Lu, Shenghua Gao, and Jingyi Yu.  Deep surface light fields.  _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , 1(1):1–17, 2018.
  * Chen et al. [2022] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Chen et al. [2023] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi.  Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures.  In _The Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Collet et al. [2015] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan.  High-quality streamable free-viewpoint video.  _ACM Transactions on Graphics (TOG)_ , 34(4):69, 2015.
  * Deng and Tartaglione [2023] Chenxi Lola Deng and Enzo Tartaglione.  Compressing explicit voxel grid representations: Fast nerfs become also small.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 1236–1245, 2023.
  * Dou et al. [2017] Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi.  Motion2fusion: Real-time volumetric performance capture.  _ACM Trans. Graph._ , 36(6):246:1–246:16, 2017\.
  * Du et al. [2021] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu.  Neural radiance flow for 4d view synthesis and video processing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * Fang et al. [2022] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian.  Fast dynamic radiance fields with time-aware neural voxels.  In _SIGGRAPH Asia 2022 Conference Papers_ , 2022.
  * Fridovich-Keil et al. [2022] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5501–5510, 2022.
  * Fridovich-Keil et al. [2023] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In _CVPR_ , 2023.
  * Gafni et al. [2021] Guy Gafni, Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8649–8658, 2021.
  * Gao et al. [2021] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2021.
  * Guo et al. [2022] Xiang Guo, Guanying Chen, Yuchao Dai, Xiaoqing Ye, Jiadai Sun, Xiao Tan, and Errui Ding.  Neural deformable voxel grid for fast optimization of dynamic view synthesis.  In _Proceedings of the Asian Conference on Computer Vision (ACCV)_ , 2022.
  * Hedman et al. [2021] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5855–5864, 2021.
  * Işık et al. [2023] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, and Matthias Nießner.  Humanrf: High-fidelity neural radiance fields for humans in motion.  _ACM Transactions on Graphics (TOG)_ , 42(4):1–12, 2023.
  * Jiang et al. [2022] Yuheng Jiang, Suyi Jiang, Guoxing Sun, Zhuo Su, Kaiwen Guo, Minye Wu, Jingyi Yu, and Lan Xu.  Neuralhofusion: Neural volumetric rendering under human-object interactions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6155–6165, 2022.
  * Jiang et al. [2023] Yuheng Jiang, Kaixin Yao, Zhuo Su, Zhehao Shen, Haimin Luo, and Lan Xu.  Instant-nvr: Instant neural volumetric rendering for human-object interactions from monocular rgbd stream.  In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 595–605, 2023.
  * Joo et al. [2018] Hanbyul Joo, Tomas Simon, and Yaser Sheikh.  Total capture: A 3d deformation model for tracking faces, hands, and bodies.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8320–8329, 2018.
  * Kappel et al. [2023] Moritz Kappel, Vladislav Golyanik, Susana Castillo, Christian Theobalt, and Marcus Magnor.  Fast non-rigid radiance fields from monocularized data, 2023.
  * Kim et al. [2012] Hansung Kim, Jean-Yves Guillemaut, Takeshi Takai, Muhammad Sarim, and Adrian Hilton.  Outdoor dynamic 3-d scene reconstruction.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 22(11):1611–1622, 2012.
  * Kirschstein et al. [2023] Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner.  Nersemble: Multi-view radiance field reconstruction of human heads.  _ACM Trans. Graph._ , 42(4), 2023.
  * Kwon et al. [2023] Youngjoong Kwon, Lingjie Liu, Henry Fuchs, Marc Habermann, and Christian Theobalt.  Deliffas: Deformable light fields for fast avatar synthesis, 2023.
  * Li et al. [2022a] Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan.  Streaming radiance fields for 3d video synthesis.  _Advances in Neural Information Processing Systems_ , 35:13485–13498, 2022a.
  * Li et al. [2023a] Lingzhi Li, Zhongshu Wang, Zhen Shen, Li Shen, and Ping Tan.  Compact real-time radiance fields with neural codebook, 2023a.
  * Li et al. [2022b] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall, Angjoo Kanazawa, and Christoph Lassner.  Tava: Template-free animatable volumetric actors.  In _European Conference on Computer Vision (ECCV)_ , 2022b.
  * Li et al. [2022c] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al.  Neural 3d video synthesis from multi-view video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5521–5531, 2022c.
  * Li et al. [2019] Zhengqi Li, Tali Dekel, Forrester Cole, Richard Tucker, Noah Snavely, Ce Liu, and William T. Freeman.  Learning the depths of moving people by watching frozen people.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4516–4525, 2019.
  * Li et al. [2021] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6498–6508, 2021.
  * Li et al. [2023b] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely.  Dynibar: Neural dynamic image-based rendering, 2023b.
  * Lin et al. [2022] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Efficient neural radiance fields for interactive free-viewpoint video.  In _SIGGRAPH Asia Conference Proceedings_ , 2022.
  * Liu et al. [2022] Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Devrf: Fast deformable voxel radiance fields for dynamic scenes.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Luo et al. [2022] Haimin Luo, Teng Xu, Yuheng Jiang, Chenglin Zhou, Qiwei Qiu, Yingliang Zhang, Wei Yang, Lan Xu, and Jingyi Yu.  Artemis: Articulated neural pets with appearance and motion synthesis.  _ACM Trans. Graph._ , 41(4), 2022.
  * Luo et al. [2020] Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf.  Consistent video depth estimation.  _ACM Transactions on Graphics (TOG)_ , 39(4):71–1, 2020.
  * Lv et al. [2018] Zhaoyang Lv, Kihwan Kim, Alejandro Troccoli, Deqing Sun, James M. Rehg, and Jan Kautz.  Learning rigidity in dynamic scenes with a moving camera for 3d motion field estimation.  In _Computer Vision – ECCV 2018_ , pages 484–501, Cham, 2018. Springer International Publishing.
  * Mihajlovic et al. [2023] Marko Mihajlovic, Sergey Prokudin, Marc Pollefeys, and Siyu Tang.  ResFields: Residual neural fields for spatiotemporal signals.  _arXiv preprint arXiv:2309.03160_ , 2023.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European conference on computer vision_ , pages 405–421. Springer, 2020.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Trans. Graph._ , 41(4):102:1–102:15, 2022\.
  * Newcombe et al. [2015] Richard A. Newcombe, Dieter Fox, and Steven M. Seitz.  Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 343–352, 2015.
  * Park et al. [2021a] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 5865–5874, 2021a.
  * Park et al. [2021b] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  _ACM Trans. Graph._ , 40(6), 2021b.
  * Peng et al. [2021a] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In _ICCV_ , 2021a.
  * Peng et al. [2021b] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In _CVPR_ , 2021b.
  * Peng et al. [2023a] Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Implicit neural representations with structured latent codes for human body modeling.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(8):9895–9907, 2023a.
  * Peng et al. [2023b] Sida Peng, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Representing volumetric videos as dynamic mlp maps.  In _CVPR_ , 2023b.
  * Pumarola et al. [2021] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10318–10327, 2021.
  * Ranftl et al. [2016] René Ranftl, Vibhav Vineet, Qifeng Chen, and Vladlen Koltun.  Dense monocular depth estimation in complex dynamic scenes.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4058–4066, 2016.
  * Reiser et al. [2023] Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, and Peter Hedman.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _SIGGRAPH_ , 2023.
  * Rho et al. [2023] Daniel Rho, Byeonghyeon Lee, Seungtae Nam, Joo Chan Lee, Jong Hwan Ko, and Eunbyung Park.  Masked wavelet representation for compact neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 20680–20690, 2023.
  * Rojas et al. [2023] Sara Rojas, Jesus Zarzar, Juan C. Pérez, Artsiom Sanakoyeu, Ali Thabet, Albert Pumarola, and Bernard Ghanem.  Re-rend: Real-time rendering of nerfs across devices.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3632–3641, 2023.
  * Shao et al. [2023] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu.  Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering, 2023.
  * Song et al. [2023] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger.  Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields.  _IEEE Transactions on Visualization and Computer Graphics_ , 29(5):2732–2742, 2023.
  * Sun et al. [2022] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5449–5459, 2022.
  * Takikawa et al. [2022] Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler.  Variable bitrate neural fields.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–9, 2022.
  * Tang et al. [2022] Jiaxiang Tang, Xiaokang Chen, Jingbo Wang, and Gang Zeng.  Compressible-composable nerf via rank-residual decomposition.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Tang et al. [2023] Jiaxiang Tang, Hang Zhou, Xiaokang Chen, Tianshu Hu, Errui Ding, Jingdong Wang, and Gang Zeng.  Delicate textured mesh recovery from nerf via adaptive surface refinement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 17739–17749, 2023.
  * Tretschk et al. [2021] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt.  Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 12959–12970, 2021.
  * Tretschk et al. [2024] Edith Tretschk, Vladislav Golyanik, Michael Zollhöfer, Aljaz Bozic, Christoph Lassner, and Christian Theobalt.  Scenerflow: Time-consistent reconstruction of general dynamic scenes.  In _International Conference on 3D Vision (3DV)_ , 2024.
  * Waechter et al. [2014] Michael Waechter, Nils Moehrle, and Michael Goesele.  Let there be color! large-scale texturing of 3d reconstructions.  In _European conference on computer vision_ , pages 836–850. Springer, 2014.
  * Wan et al. [2023] Ziyu Wan, Christian Richardt, Aljaž Božič, Chao Li, Vijay Rengarajan, Seonghyeon Nam, Xiaoyu Xiang, Tuotuo Li, Bo Zhu, Rakesh Ranjan, and Jing Liao.  Learning neural duplex radiance fields for real-time view synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8307–8316, 2023.
  * Wang et al. [2021] Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, and Jingyi Yu.  ibutter: Neural interactive bullet time generator for human free-viewpoint rendering.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 4641–4650, 2021.
  * Wang et al. [2022] Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu.  Fourier plenoctrees for dynamic radiance field rendering in real-time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13524–13534, 2022.
  * Wang et al. [2023a] Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu.  Neural residual radiance fields for streamably free-viewpoint videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 76–87, 2023a.
  * Wang et al. [2023b] Yiming Wang, Qin Han, Marc Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu.  Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2023b.
  * Wood et al. [2000] Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle.  Surface light fields for 3d photography.  In _Proceedings of the 27th annual conference on Computer graphics and interactive techniques_ , pages 287–296, 2000.
  * Xian et al. [2021] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim.  Space-time neural irradiance fields for free-viewpoint video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9421–9431, 2021.
  * Xu et al. [2020] Lan Xu, Zhuo Su, Lei Han, Tao Yu, Yebin Liu, and Lu Fang.  Unstructuredfusion: Realtime 4d geometry and texture reconstruction using commercial rgbd cameras.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 42(10):2508–2522, 2020.
  * Xu et al. [2021] Lan Xu, Wei Cheng, Kaiwen Guo, Lei Han, Yebin Liu, and Lu Fang.  Flyfusion: Realtime dynamic scene reconstruction using a flying depth camera.  _IEEE Transactions on Visualization and Computer Graphics_ , 27(1):68–82, 2021.
  * Xu et al. [2023a] Yuelang Xu, Lizhen Wang, Xiaochen Zhao, Hongwen Zhang, and Yebin Liu.  Avatarmav: Fast 3d head avatar reconstruction using motion-aware neural voxels.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023a.
  * Xu et al. [2023b] Yuelang Xu, Hongwen Zhang, Lizhen Wang, Xiaochen Zhao, Huang Han, Qi Guojun, and Yebin Liu.  Latentavatar: Learning latent expression code for expressive neural head avatar.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023b.
  * Xu et al. [2023c] Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou.  4k4d: Real-time 4d view synthesis at 4k resolution.  2023c.
  * Yariv et al. [2023] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall.  Bakedsdf: Meshing neural sdfs for real-time view synthesis.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , New York, NY, USA, 2023. Association for Computing Machinery.
  * Yu et al. [2021] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  Plenoctrees for real-time rendering of neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5752–5761, 2021.
  * Yu et al. [2023] Heng Yu, Joel Julin, Zoltán Á. Milacski, Koichiro Niinuma, and László A. Jeni.  Dylin: Making light field networks dynamic.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12397–12406, 2023.
  * Zhang et al. [2022] Anlan Zhang, Chendong Wang, Bo Han, and Feng qian.  Yuzu:neural-enhanced volumetric video streaming.  2022\.
  * Zhang et al. [2021] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu.  Editable free-viewpoint video using a layered neural representation.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–18, 2021.
  * Zhao et al. [2022] Fuqiang Zhao, Yuheng Jiang, Kaixin Yao, Jiakai Zhang, Liao Wang, Haizhao Dai, Yuhui Zhong, Yingliang Zhang, Minye Wu, Lan Xu, and Jingyi Yu.  Human performance modeling and rendering via neural animated mesh.  _ACM Trans. Graph._ , 41(6), 2022.
  * Zhao et al. [2023] Xiaochen Zhao, Lizhen Wang, Jingxiang Sun, Hongwen Zhang, Jinli Suo, and Yebin Liu.  Havatar: High-fidelity head avatar via facial model conditioned neural radiance field.  _ACM Trans. Graph._ , 2023.
  * Zheng et al. [2023] Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and Yebin Liu.  Avatarrex: Real-time expressive full-body avatars.  _ACM Transactions on Graphics (TOG)_ , 42(4), 2023.

Supplementary Material

##  7 Training Details for VideoRF

###  7.1 Coarse Stage Pre-training and Baking

Given a long-duration multi-view sequence, we initially adopt the approach
from DVGO [56] to generate an explicit density volume grid
$\mathbf{V}_{\sigma}$ and color feature grids $\mathbf{V}_{c}$ representation
for each frame $t$. Following ReRF [66], we employ a global MLP $\Phi_{c}$
during this coarse stage training. This MLP comprises a single hidden layer
with 129 channels, and we set the color feature dimension at $h=12$.
Throughout the training, we incrementally upscale the volume grid, from
$(125\times 125\times 125)\rightarrow(150\times 150\times
150)\rightarrow(200\times 200\times 200)\rightarrow(250\times 250\times 250)$,
after reaching the training step 2000, 4000 and 6000, respectively. For loss
calculation, we utilize both the photometric MSE loss and the total variation
loss on $\mathbf{V}_{\sigma}$, expressed as:

$\mathcal{L}_{\text{rgb}_{\text{{coarse}}}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{c}(\mathbf{r})-\hat{\mathbf{c}}(\mathbf{r})\|^{2},$
(9)
$\mathcal{L}_{\text{TV}_{\text{coarse}}}=\frac{1}{\left|\mathbf{V}_{\sigma}\right|}\sum_{\mathbf{v}\in\mathbf{V}_{\sigma}}\sqrt{\Delta_{x}^{2}\mathbf{v}+\Delta_{y}^{2}\mathbf{v}+\Delta_{z}^{2}\mathbf{v}},$
(10)
$\mathcal{L}_{\text{coarse}}=\mathcal{L}_{\text{rgb}_{\text{{coarse}}}}+\lambda_{\text{TV}}\mathcal{L}_{\text{TV}_{\text{coarse}}},$
(11)

where $\lambda_{\text{TV}}=0.000016$. Here, $\mathcal{R}$ represents the set
of training pixel rays, with $\mathbf{c}(\mathbf{r})$ and
$\mathbf{\hat{c}}(\mathbf{r})$ denoting the actual and predicted colors of a
ray $\mathbf{r}$, respectively. $\Delta_{x,y,z}^{2}\mathbf{v}$ signifies the
squared difference in the voxel’s density value. Notably, the total variation
loss is activated only during the training iterations from 1000 to 12000. For
optimization, we utilize the Adam optimizer for training 16000 iterations with
a batch size of 10192 rays. The learning rates for $\mathbf{V}_{\sigma}$,
$\mathbf{V}_{c}$ and global MLP are 0.1, 0.11 and 0.002, respectively.

Once we obtain the density grid $\mathbf{V}_{\sigma}^{t}$ for each frame $t$
in the coarse training phase, we generate a per-frame occupancy grid
$\mathbf{O}^{t}$ by retaining voxels with a density above the threshold
$\gamma=0.003$. During our adaptive grouping stage, we set the pixel limit to
$\theta=512\times 512$.

###  7.2 Fine-grained Sequential Training

After creating the mapping tables, we proceed to fine-grained sequential
training within each group. At this stage, we also introduce a global tiny MLP
$\Phi_{f}$ designed for efficient rendering on mobile platforms. This minimal
MLP $\Phi_{f}$ consists of only one hidden layer with 16 channels, and we
maintain the color feature dimension $h$ at 12. Similar to the coarse stage,
we progressively upscale the volume grid during training, moving from
$(125\times 125\times 125)$ to $(150\times 150\times 150)$, then to
$(200\times 200\times 200)$, and finally to $(250\times 250\times 250)$,
corresponding to the training steps at 2000, 4000, and 6000, respectively. For
loss calculations, we employ both the photometric MSE loss and the total
variation loss on the density volume $\mathbf{V}_{\sigma}$, as well as spatial
consistency loss and temporal consistency loss on the feature image
$\mathbf{I}$:

$\mathcal{L}_{\text{rgb}_{\text{{fine}}}}=\sum_{\mathbf{r}\in\mathcal{R}}\|\mathbf{c}(\mathbf{r})-\hat{\mathbf{c}}(\mathbf{r})\|^{2},$
(12)
$\mathcal{L}_{\text{TV}_{\text{fine}}}=\frac{1}{\left|\mathbf{V}_{\sigma}\right|}\sum_{\mathbf{v}\in\mathbf{V}_{\sigma}}\sqrt{\Delta_{x}^{2}\mathbf{v}+\Delta_{y}^{2}\mathbf{v}+\Delta_{z}^{2}\mathbf{v}}$
(13)
$\mathcal{L}_{\mathrm{spatial}}=\frac{1}{|\mathcal{P}|}\sum_{\begin{subarray}{c}\mathbf{p}\in\mathcal{V}\end{subarray}}\left(\Delta_{u}(\mathbf{p})+\Delta_{v}(\mathbf{p})\right),$
(14)
$\mathcal{L}_{\mathrm{temporal}}=\|\mathbf{I}^{t}-\mathbf{I}^{t-1}\|_{1},$
(15)
$\mathcal{L}_{\text{fine}}=\mathcal{L}_{\mathrm{rgb}_{\text{fine}}}+\lambda_{\text{TV}}\mathcal{L}_{\text{TV}_{\text{fine}}}+\lambda_{\text{s}}\mathcal{L}_{\mathrm{spatial}}+\lambda_{\text{t}}\mathcal{L}_{\mathrm{temporal}},$
(16)

where $\lambda_{\text{TV}}=0.000016$, $\lambda_{\text{s}}=0.0001$ and
$\lambda_{\text{t}}=0.0001$. The total variation loss is specifically
activated during training iterations 1000 to 12000. We continue to use the
Adam optimizer for 16000 iterations with a batch size of 10192 rays. The
learning rates for $\mathbf{V}_{\sigma}$, $\mathbf{V}_{c}$, and the global MLP
are set to 0.1, 0.11, and 0.002, respectively.

Figure 9: Our VideoRF facilitates dynamic radiance field rendering on
ubiquitous devices, from desktops to mobile phones.

##  8 Implementation Details for VideoRF Player

During the baking process in our training scheme, we generate a mapping table.
For streaming, we save it in a 2D-to-3D format as an RGB image to conserve
bitrate. Then, to enable rapid rendering, we first use this mapping table to
recover a 3D volume. We employ a compute shader for efficient processing,
segmenting the 512$\times$512 mapping table into 16$\times$16 workgroups, each
handling a 32$\times$32 pixel section. The highly parallel architecture of
compute shaders enables us to efficiently convert 2D features back into a 3D
volume. Meanwhile, we adopt a multi-resolution occupancy grid to bypass empty
3D spaces at various levels. This approach significantly reduces unnecessary
network inferences during the ray marching process. The largest occupancy grid
is derived from max-pooling the full-resolution binary mask. Each subsequent
grid is designed to be half the resolution of its predecessor. For instance,
considering our full-resolution binary mask is of size
288$\times$288$\times$288, our multi-resolution occupancy grids follow suit
with sizes of 144$\times$144$\times$144, 72$\times$72$\times$72,
36$\times$36$\times$36, 18$\times$18$\times$18, and 9$\times$9$\times$9.

##  9 Additional Experiments

best second-best
---
Method | Metric | 20 | 50 | 100 | 250 | 500 | 1000
HumanRF |  $\downarrow$ LPIPS | 0.120 | 0.138 | 0.135 | 0.151 | 0.155 | 0.160
$\uparrow$ PSNR | 31.02 | 30.26 | 30.25 | 28.98 | 29.50 | 29.19
$\uparrow$ SSIM | 0.893 | 0.888 | 0.896 | 0.888 | 0.885 | 0.881
TiNeuVox |  $\downarrow$ LPIPS | 0.352 | 0.298 | 0.406 | 0.430 | 0.436 | 0.452
$\uparrow$ PSNR | 27.51 | 26.62 | 24.13 | 22.98 | 22.30 | 21.28
$\uparrow$ SSIM | 0.782 | 0.791 | 0.760 | 0.752 | 0.751 | 0.747
NDVG |  $\downarrow$ LPIPS | 0.240 | 0.281 | 0.354 | 0.435 | 0.453 | 0.481
$\uparrow$ PSNR | 28.76 | 25.83 | 23.13 | 21.17 | 20.05 | 17.83
$\uparrow$ SSIM | 0.841 | 0.812 | 0.763 | 0.731 | 0.724 | 0.692
HyperNeRF |  $\downarrow$ LPIPS | 0.233 | 0.250 | 0.275 | 0.322 | 0.374 | 0.388
$\uparrow$ PSNR | 25.75 | 26.53 | 25.96 | 24.85 | 23.29 | 23.04
$\uparrow$ SSIM | 0.827 | 0.818 | 0.800 | 0.777 | 0.758 | 0.761
NeuralBody |  $\downarrow$ LPIPS | 0.288 | 0.333 | 0.354 | 0.368 | 0.396 | 0.429
$\uparrow$ PSNR | 27.51 | 25.88 | 27.18 | 25.30 | 24.81 | 25.68
$\uparrow$ SSIM | 0.804 | 0.777 | 0.739 | 0.762 | 0.745 | 0.668
TAVA |  $\downarrow$ LPIPS | 0.261 | 0.303 | 0.341 | 0.410 | 0.467 | 0.504
$\uparrow$ PSNR | 28.47 | 26.93 | 25.83 | 24.28 | 23.13 | 22.21
$\uparrow$ SSIM | 0.820 | 0.801 | 0.782 | 0.749 | 0.721 | 0.704
MeRF |  $\downarrow$ LPIPS | 0.278 | 0.276 | 0.259 | 0.271 | 0.272 | 0.263
$\uparrow$ PSNR | 28.24 | 28.19 | 27.24 | 27.22 | 27.31 | 27.68
$\uparrow$ SSIM | 0.783 | 0.791 | 0.815 | 0.807 | 0.805 | 0.814
ReRF |  $\downarrow$ LPIPS | 0.297 | 0.296 | 0.297 | 0.296 | 0.292 | 0.294
$\uparrow$ PSNR | 28.69 | 28.51 | 28.55 | 28.33 | 28.12 | 27.73
$\uparrow$ SSIM | 0.834 | 0.828 | 0.827 | 0.836 | 0.836 | 0.841
Ours |  $\downarrow$ LPIPS | 0.276 | 0.285 | 0.283 | 0.278 | 0.274 | 0.275
$\uparrow$ PSNR | 29.14 | 28.79 | 28.81 | 28.46 | 28.50 | 28.32
$\uparrow$ SSIM | 0.840 | 0.835 | 0.830 | 0.838 | 0.840 | 0.844

Table 4: Quantitative comparison on long-duration sequence. We evaluate on the
Actor 3, Sequence 1 of the Actors-HQ Dataset. Figure 10: Qualitative
comparison on the long-duration sequence against recent dynamic scene
reconstruction methods and per frame static reconstruction methods.

As illustrated in Fig. 9, our method can enable dynamic radiance field
rendering on a wide range of devices, including desktops (an i7-12700F CPU and
NVIDIA RTX3090 GPU), laptops (an i5-1135G7CPU and Integrated GPU), tablets
(iPad Pro, an M2 chip) and mobile phones (iPhone 14 Pro, an A16 Bionic chip).
Long-duration dynamic scenes. Following the approach in HumanRF [19], we
assess performance on a long-duration sequence (Actor3, sequence1, 1000
frames) from the Actors-HQ dataset. We compare our method with ReRF [66] and
MeRF [51] through per-frame static reconstruction in Fig. 10. Our method keeps
a small storage to enable streaming while maintaining a high rendering
quality. We adopt the testing methods outlined in HumanRF. The performance
metrics for HumanRF [19], TiNeuVox [12], NDVG [17], HyperNeRF [44], NeuralBody
[46], TAVA [29] are directly sourced from the HumanRF publication. As shown in
Fig. 10 and Tab. 4, our approach demonstrates its capability to sustain high
photorealism which is only second to HumanRF throughout long-duration
sequences. Note that, our VideoRF is the only method to enable rendering
dynamic scenes on mobile platforms.