# BakedAvatar: Baking Neural Fields for Real-Time Head Avatar Synthesis

Hao-Bin Duan<EMAIL_ADDRESS>0009-0002-1200-3236 State Key Laboratory of
Virtual Reality Technology and Systems, Beihang UniversityBeijingChina100191 ,
Miao Wang<EMAIL_ADDRESS>0000-0002-4102-8582 State Key Laboratory of
Virtual Reality Technology and Systems, Beihang University, and Zhongguancun
LaboratoryBeijingChina100191 ,  Jin-Chuan Shi<EMAIL_ADDRESS>0009-0003-4899-2205 State Key Laboratory of Virtual Reality Technology and
Systems, Beihang UniversityBeijingChina100191 ,  Xu-Chuan Chen
0009-0006-7072-4305 State Key Laboratory of Virtual Reality Technology and
Systems, Beihang UniversityBeijingChina and  Yan-Pei Cao<EMAIL_ADDRESS>ARC Lab, Tencent PCGBeijingChina

###### Abstract.

Synthesizing photorealistic 4D human head avatars from videos is essential for
VR/AR, telepresence, and video game applications. Although existing Neural
Radiance Fields (NeRF)-based methods achieve high-fidelity results, the
computational expense limits their use in real-time applications. To overcome
this limitation, we introduce _BakedAvatar_ , a novel representation for real-
time neural head avatar synthesis, deployable in a standard polygon
rasterization pipeline. Our approach extracts deformable multi-layer meshes
from learned isosurfaces of the head and computes expression-, pose-, and
view-dependent appearances that can be baked into static textures for
efficient rasterization. We thus propose a three-stage pipeline for neural
head avatar synthesis, which includes learning continuous deformation,
manifold, and radiance fields, extracting layered meshes and textures, and
fine-tuning texture details with differential rasterization. Experimental
results demonstrate that our representation generates synthesis results of
comparable quality to other state-of-the-art methods while significantly
reducing the inference time required. We further showcase various head avatar
synthesis results from monocular videos, including view synthesis, face
reenactment, expression editing, and pose editing, all at interactive frame
rates on commodity devices. Source codes and demos are available on our
project page: https://buaavrcg.github.io/BakedAvatar.

Head Avatar Synthesis, Face Reenactment, Neural Radiance Fields

††submissionid: 891††copyright: acmlicensed††journal: TOG††journalyear:
2023††journalvolume: 42††journalnumber: 6††article: 225††publicationmonth:
12††price: 15.00††doi: 10.1145/3618399††ccs: Computing methodologies Image-
based rendering Figure 1. In _BakedAvatar_ , the learned neural fields of a
head avatar are baked into (a) deformable layered meshes (and corresponding
textures). This enables (b) face reenactment, (c) view synthesis, expression
editing, and pose editing applications. It supports (d) real-time rendering on
a variety of common devices with a standard polygon rasterization pipeline,
and (e) outperforms alternatives in rendering speed while maintaining
comparable or superior rendering quality.

##  1\. Introduction

The digitization of photorealistic 4D (or dynamic 3D) human heads has been a
significant area of research in the fields of computer graphics and vision due
to its importance in various applications such as VR/AR, telepresence, and
video games. Given the complexity of human head shape, appearance, and
movement, the precise replication of head avatar details from monocular videos
presents a considerable challenge. Recently, the advent of neural rendering
methods, particularly Neural Radiance Fields (NeRF) (Mildenhall et al., 2020),
has substantially improved the quality of novel view synthesis. Subsequent
applications of NeRF have accomplished 4D head avatar reconstruction by
employing morphable facial models (Paysan et al., 2009; Cao et al., 2013; Li
et al., 2017) with either direct expression and pose conditioning or learned
linear blendable bases (Gafni et al., 2021; Gao et al., 2022). Other
techniques involve the deformation of motion grids, meshes, or implicit
functions (Xu et al., 2023a; Zielonka et al., 2023; Zheng et al., 2022); or
the learning of warpings to a canonical space (Park et al., 2021a, b).

Despite yielding photorealistic head avatar reconstructions, NeRF-based
methods come with extensive computational demands. This is largely due to the
dense ray-marching and implicit scene representation, which requires large
neural networks to evaluate hundreds of samples along each ray for every pixel
rendered. Consequently, their real-time application on commodity devices is
significantly limited. Recent advancements have sought to address this issue
by converting a trained NeRF into explicit or hybrid representation for fast
inference, such as multi-plane images (MPIs) (Wizadwongsa et al., 2021),
sparse voxel grids (Hedman et al., 2021), meshes (Chen et al., 2023a; Yariv et
al., 2023), and tri-planes (Reiser et al., 2023). The conversion process,
often called “baking”, essentially pre-computes implicit representation and
stores them in data structures that can be queried with minimal computation.
However, these methods primarily focus on speeding up the inference of static
scenes and do not readily extend to the more complex challenge of
reconstructing dynamic objects.

The primary goal of this paper is to develop a representation that is
optimized for 4D head avatar rendering, capable of running in real-time even
on commodity devices such as mobiles. This introduces several constraints,
including efficient implementation across devices, moderate storage demands,
and the capability to animate based on user input expression or pose
conditions. One might consider a mesh-based representation, the best supported
primitive in the modern graphics pipeline. However, conventional mesh-based
solutions primarily record appearance and compute shading at object surfaces,
which significantly differs from volume-based NeRF. Although there are methods
(Oechsle et al., 2021; Wang et al., 2021; Yariv et al., 2021) that resort to
surface-based reconstruction, we find them unsatisfactory for human head
avatars, as pure surfaces struggle to capture semi-transparent and fine-scale
structures such as hair. Furthermore, recovering detailed surface geometry
would necessitate a highly intricate mesh, which could undermine the rendering
performance.

Building upon the observations discussed previously, we present _BakedAvatar_
, a novel representation to achieve person-specific photorealistic head avatar
synthesis harnessing rasterization, aiming to match the volume rendering
quality provided by NeRF. While NeRF inherently demands a large number of
sample points along the ray to produce impeccable results, recent advancements
(Neff et al., 2021; Attal et al., 2023; Deng et al., 2022; Chen et al., 2021a;
Lin et al., 2022) have demonstrated the feasibility of a substantial reduction
in sampling points, as long as the locations of key sampling points are
available. For instance, GRAM (Deng et al., 2022) employs a learned manifold
that guides the radiance sampling, defining implicit isosurfaces and utilizing
only their intersections with camera rays for ray integration. The crux of our
proposed method is to similarly learn a manifold, but one that closely
envelops the human head surface, allowing us to significantly minimize the
number of sampling points. We then proceed to extract multiple mesh proxies
from this learned manifold, enabling direct calculation of ray-manifold
intersections. Our proposed representation shares similarities with MPIs
(Tucker and Snavely, 2020), but is warped around the human head. This unique
configuration accommodates larger deformations than conventional MPIs can
handle.

We then animate the proposed layered mesh representation using a morphable
model. Specifically, our approach adopts FLAME (Li et al., 2017), a parametric
head model that offers control over expression and pose. By learning a FLAME
deformation field, we are able to export the derived blendshapes and skinning
weights as vertex attributes. Furthermore, we propose to bake expression-,
pose- and view-dependent appearance as a combination of linearly blendable
texture bases. In addition, we develop a lightweight appearance decoder that
runs efficiently during the shading process. Once trained from monocular
videos, our representation enables real-time synthesis of 4D head avatars on
consumer-grade devices.

To summarize, our major contributions include:

  * •

We present _BakedAvatar_ , a novel representation for 4D head avatars that
bakes neural fields into deformable layered meshes and corresponding textures,
designed for a rasterization pipeline.

  * •

We propose a multi-layer mesh proxy that approximates the volumetric rendering
of human heads. These meshes are rigged with FLAME-based morphing whose
weights are extracted from a learned deformation field.

  * •

We bake expression-, pose- and view-dependent appearance into linear blendable
texture bases, and compose them with a light-weighted appearance decoder
running in pixel shaders.

  * •

Experimental results show that our method synthesizes head avatars of
comparable quality to state-of-the-art (SOTA) methods while being
significantly faster. We demonstrate that our representation achieves real-
time rendering on commodity devices including laptops, tablets, and mobile
phones, while enabling interactive expression and pose editing, as well as
face reenactment.

Figure 2. Overview of our three-stage pipeline. In the first stage, we learn
three implicit fields in canonical space for FLAME-based deformation,
isosurface geometry, as well as multiple radiance bases and a position feature
that are combined by the light-weighted appearance decoder. In the second
stage, we bake these neural fields as deformable multi-layer meshes and
multiple static textures. In the third stage, we use differential
rasterization to fine-tune the baked textures.

##  2\. Related Work

###  2.1. 4D Head Avatar Synthesis

Extensive research has been conducted on synthesizing 4D human head avatars.
Image-based approaches reconstruct human heads from portraits or monocular
videos without any 3D representation. These methods include encoder-decoder
models (Kim et al., 2018; Thies et al., 2019; Zakharov et al., 2020), warping
fields (Siarohin et al., 2019), and generative adversarial networks with
explicit control (Wang et al., 2023). Although Deep Video Portraits (Kim et
al., 2018; Wen et al., 2020) can run on commodity devices almost in real-time,
image-based methods still face challenges with view inconsistencies. Mesh-
based methods output 3D human head meshes based on portraits or videos along
with geometry prior (Li et al., 2017), mitigating the above weaknesses. Some
reconstruct a morphable head model without textures from a portrait (Feng et
al., 2021a; Zielonka et al., 2022), while others utilize a hybrid
representation to model the morphable head model and the texture space
(Grassal et al., 2022; Khakhulin et al., 2022; Bai et al., 2023). Although
morphable models are convenient to animate and stable in the classical
rendering pipeline, they require a trade-off between efficiency and quality.
Recently, NeRF-based methods have gained attention. Deformable NeRFs have been
explored for dynamic scenes (Pumarola et al., 2021; Park et al., 2021a, b; Liu
et al., 2022), which can be applied to human heads. To achieve accurate
animations, NeRFs are usually combined with 3D Morphable Models (Gafni et al.,
2021; Gao et al., 2022; Zheng et al., 2022; Xu et al., 2023a) or a generative
framework (Wu et al., 2022; Xu et al., 2023b; Yu et al., 2023). However, these
methods require dense ray sampling and evaluation of large neural networks to
predict attributes such as vertex displacement, radiance, density, surface
position, etc. The significant computational requirements make real-time
rendering prohibitive on consumer-grade devices, especially for mobiles. More
recent works are gradually approaching interactive rendering on desktop GPUs,
with methods such as mesh-guided deformation (Zielonka et al., 2023) and real-
time inversion of a trained 3D portrait generator (Trevithick et al., 2023).
Our approach takes a further step that directly renders multi-layer meshes,
achieving real-time framerates even on mobile devices.

###  2.2. Hybrid NeRF Representation for Accelerated Inference

Minimizing the computational expense of NeRFs for rapid inference using
explicit or hybrid scene representations is an active research direction. One
solution is to split large Multi-Layer Perceptrons (MLPs) in NeRF into small
chunks and cache them independently (Garbin et al., 2021) or store them in
smaller MLPs (Reiser et al., 2021). Sparse voxel structures such as octrees
can efficiently represent a scene by skipping empty spaces (Liu et al., 2020),
encoding appearance using baked spherical harmonics (Yu et al., 2021;
Fridovich-Keil et al., 2022), or storing features which can be decoded with a
tiny deferred shading network to produce specular appearance (Hedman et al.,
2021). Image-based representations like MPIs (Wizadwongsa et al., 2021) or
triplane decomposed volumes (Reiser et al., 2023), can store a scene with less
memory. Mesh-based representation is rarely used for NeRFs as shading only
occurs on the surface. Still, recent work has converted NeRFs to meshes for
rasterization pipeline with an optimized polygon soup (Chen et al., 2023a) or
signed distance fields (Yariv et al., 2023). Neumesh (Yang et al., 2022) also
bakes NeRF to a triangle mesh for editing. The exploration of hybrid mesh-
volume representation (Guo et al., 2023) is also underway. Dynamic scenes can
be parameterized using Fourier-PlenOctrees (Wang et al., 2022) or 4D voxel
deformation fields (Liu et al., 2022); however, these approaches necessitate
substantial memory consumption and are unsuitable for animatable head avatars.
Our work focuses on real-time inference of an animatable head avatar by baking
neural fields into deformable layered meshes and corresponding textures
without using large MLPs or ray marching-based radiance samplings.

##  3\. Method

Our main idea is to learn multi-layer meshes that approximate volume rendering
in NeRFs. These layered meshes should possess the ability to deform in
response to head pose and expression variations while also accurately
capturing dynamic changes in appearance caused by micro-expressions. Further,
the neural fields associated with a head avatar can be baked into the
deformable layered meshes and corresponding textures for real-time rendering.
To this end, we develop a three-stage method to train the person-specific head
avatar. First, we reconstruct the 4D head avatar with implicit fields designed
for baking (Sec. 3.1). Second, we extract the deformable layered meshes and
bake the radiance field into static textures for real-time rendering (Sec.
3.2). Finally, we fine-tune this representation using differential
rasterization (Sec. 3.3). The pipeline of our method is illustrated in Fig. 2.

###  3.1. Learning Deformable Manifold and Expression-, Pose-, View-Dependent
Appearance

To acquire multi-layer meshes and enable real-time 4D head avatar synthesis
from monocular videos, we first reconstruct the head avatar with an implicit
representation. Specifically, we learn three continuous fields: a manifold
field for multi-layer geometry, a forward deformer for FLAME-based
deformation, and a radiance field for dynamic appearance conditioned on
expression, pose, and view.

####  3.1.1. Canonical Manifold Field

We learn proxy layered surfaces to constrain radiance sampling points, with
the surfaces represented as predefined level sets of a manifold field. The
manifold field is established in canonical space, which is shared by all
expressions and poses to streamline the training process. In accordance with
GRAM (Deng et al., 2022), we employ an MLP to implement the manifold field,
taking a canonical position $\mathbf{x}_{c}$ as input and generating a scalar
level $s$, expressed as

(1) $\mathcal{M}:\mathbf{x}_{c}\in\mathbb{R}^{3}\to s\in\mathbb{R}.$

Radiance accumulation occurs on the learned layered surfaces, which are given
by canonical space intersections of the ray and implicit isosurfaces
$\mathcal{S}=\\{S_{i}\\}$ of predefined level sets $\Gamma=\\{l_{i}\\}$. Let
$\mathbf{\hat{x}}_{c}$ be a canonical intersection point, the $i$-th layer
surface is denoted as
$S_{i}=\\{\mathbf{\hat{x}}_{c}|\mathcal{M}(\mathbf{\hat{x}}_{c})=l_{i}\\}$.
Undesirable intersections among various layers are substantially avoided, as
they are defined as isosurfaces of the manifold level sets, which assist in
extracting well-structured meshes as described in Sec. 3.2. We implement the
canonical manifold using a coordinate-based MLP and adopt geometry
initialization (Atzmon and Lipman, 2020) to prevent training divergence.
Furthermore, we employ a low position encoding frequency to promote smooth
geometry, facilitating the extraction of simpler meshes.

####  3.1.2. FLAME-Driven Forward Deformer

To achieve real-time manipulation of the layered meshes based on head
expression and pose changes, we adopt the FLAME deformation model (Li et al.,
2017), which allows us to compute deformations with per-vertex blendshape and
linear blend skinning (LBS) operations on the extracted meshes. Unlike
backward deformation used in dynamic NeRFs (Pumarola et al., 2021; Park et
al., 2021a, b), the forward deformer transforms a canonical isosurface to a
deformed one that intersects with camera rays. Following IMavatar (Zheng et
al., 2022), the blendshape and LBS weights of a point $\mathbf{x}_{c}$ in
canonical space are predicted with a deformation field
$\mathcal{D}:\mathbf{x}_{c}\in\mathbb{R}^{3}\to\mathcal{E},\mathcal{P},\mathcal{W}$,
where $\mathcal{E}\in\mathbb{R}^{n_{e}\times 3}$ are $n_{e}=50$ expression
blendshapes, $\mathcal{P}\in\mathbb{R}^{n_{p}\times 9\times 3}$ are $n_{p}=4$
pose correctives, and $\mathcal{W}\in\mathbb{R}^{n_{j}}$ are $n_{j}=5$ LBS
bone weights, all at canonical points. The location of each deformed point
$\mathbf{x}_{d}$ from canonical point $\mathbf{x}_{c}$ is computed using the
FLAME deformation formula:

(2)
$\mathbf{x}_{d}=LBS(\mathbf{x}_{c}+B_{E}(\psi;\mathcal{E})+B_{P}(\theta;\mathcal{P}),J(\mathbf{x}_{c}+B_{E}(\psi;\mathcal{E})),\theta,\mathcal{W}),$

where $\psi$ and $\theta$ represent the FLAME expression and pose coefficients
for the current frame, while $B_{E}(\cdot)$ and $B_{P}(\cdot)$ compute the
expression and pose offsets using the corrective blendshape bases
$\mathcal{P}$ and $\mathcal{E}$. $J(\cdot)$ serves as the joint regressor.

As we learn a deformation field in canonical space, we follow the iterative
root finding scheme in SNARF (Chen et al., 2021b) to retrieve the canonical
correspondence of ray sampling points in the deformed space. More details can
be found in the appendix.

Figure 3. The light-weighted appearance decoder $P_{\text{app}}$ consists of a
global MLP and a spatial MLP. The global MLP takes inputs of global expression
and pose conditions and produces the weights of spatial MLP. The spatial MLP
takes inputs of position texture, normal, and view direction, which vary
spatially in screen space, and produces normalized blending weights. These
weights are then combined with multiple radiance texture bases to generate
color and alpha values of the current mesh layer.

####  3.1.3. Expression-, Pose-, and View-Dependent Radiance Field

The appearance of a head avatar is highly dynamic and complex, influenced by
factors such as viewing direction, pose and expression, etc. To model the
expression-, pose, and view-dependent effects, the radiance field takes input
from FLAME expression coefficients $\psi\in\mathbb{R}^{n_{e}}$ and rotation of
jaw and eye joints $\theta\in\mathbb{R}^{3\times 3}$. Additionally, we
condition the radiance field with the spatially varying normal
$\mathbf{n}\in\mathbb{R}^{3}$ and view direction
$\mathbf{v}\in\mathbb{R}^{3}$, both in the camera space. The radiance field
predicts RGB color $\mathbf{c}\in\mathbb{R}^{3}$ and occupancy
$\alpha\in[0,1]$ in canonical space, which is denoted as

(3)
$\mathcal{R}:(\mathbf{x}_{c},\mathbf{v},\mathbf{n},\mathbf{\theta},\mathbf{\psi})\to(\mathbf{c},\alpha)\in\mathbb{R}^{4}.$

To ensure real-time inference performance and avoid evaluating large MLPs, we
represent this radiance field using multiple linear blendable bases, where
each basis can be easily baked into a static texture for efficient querying.
Concretely, assuming there are $n_{T}$ texture bases, the radiance field then
predicts bases’ colors
$\\{\mathbf{c}_{1},\mathbf{c}_{2},\cdots,\mathbf{c}_{n_{T}}\\}$ and
occupancies
$\\{\mathbf{\alpha}_{1},\mathbf{\alpha}_{2},\cdots,\mathbf{\alpha}_{n_{T}}\\}$,
as well as a shared $d_{p}$-dimensional position feature
$\mathbf{f}_{p}\in\mathbb{R}^{d_{p}}$ in canonical space, denoted as

(4)
$\mathcal{R}_{\text{basis}}:\mathbf{x}_{c}\in\mathbb{R}^{3}\to(\\{\mathbf{c}^{i}\\},\\{\mathbf{\alpha}^{i}\\},\mathbf{f}_{p})\in\mathbb{R}^{4\times
n_{T}+d_{p}}.$

The position feature contains high-frequency textural information which can be
recorded in a static texture. We use an appearance decoder $P_{\text{app}}$ to
project the global and spatial varying conditions
$(\mathbf{f}_{p},\mathbf{v},\mathbf{n},\mathbf{\theta},\mathbf{\psi})$ into
softmax normalized weights $\mathbf{w}\in\mathbb{R}^{n_{T}}$ for blending
multiple radiance bases. As shown in Fig. 3, the decoder is implemented as a
tiny hyper-network that consists of two parts: $P_{\text{global}}$, mapping
global condition to the weights of $P_{\text{spatial}}$, and
$P_{\text{spatial}}$, predicting blending weights from spatial varying
conditions:

(5)
$\begin{cases}P_{\text{global}}:(\mathbf{\theta},\mathbf{\psi})\to\Theta_{\text{spatial}}\\\
P_{\text{spatial}}:(\mathbf{f}_{p},\mathbf{v},\mathbf{n};\Theta_{\text{spatial}})\to\mathbf{w}\in\mathbb{R}^{n_{T}}.\end{cases}$

The hyper-network efficiently reduces the computation related to global
conditions, as the global MLP requires evaluation only once per frame, while
the spatial MLP is evaluated at each ray-manifold intersection. The final
radiance is derived as a linear combination of radiance from all bases:

(6) $(\mathbf{c},\mathbf{\alpha})=\sum_{i\in
n_{T}}\mathbf{w}_{i}(\mathbf{c_{i}},\mathbf{\alpha}_{i}).$

####  3.1.4. Training Strategy

We employ non-rigid ray marching to identify the canonical manifold
isosurfaces and collect the radiance and occupancy at their intersections
along the ray using alpha compositing:

(7)
$C(r)=\sum_{j=1}^{M}\prod_{k<j}(1-\alpha(\mathbf{x}_{j}))\alpha(\mathbf{x}_{j})\mathbf{c}(\mathbf{x}_{j})$

where $\alpha(\mathbf{x}_{j})$ is the opacity at position $\mathbf{x}_{j}$,
and $\mathbf{c}(\mathbf{x}_{j})$ is the color at position at position
$\mathbf{x}_{j}$. Note that this formulation is different from the original
rendering formula in NeRF, as we are using multiple surfaces instead of
sampling points in a volume. During training, we randomly sample a number of
rays, and the geometry, deformation, and appearance fields are jointly
optimized in an end-to-end manner. This process is supervised by image-based
losses and pseudo deformation ground truth extracted from the off-the-shelf
expression capture module DECA (Feng et al., 2021a). We initially apply a
photometric loss on the rendered color of each ray $r$ where a ray-manifold
intersection is found:

(8) $\mathcal{L}_{\text{rgb}}=\frac{1}{|R|}\sum_{r\in
R}\|\mathbf{C}_{r}-\mathbf{C}_{r}^{\text{GT}}\|_{1},$

where $R$ is the set of rays intersected with the isosurfaces $\mathcal{S}$,
$\mathbf{C}_{r}$ is the rendered color, $\mathbf{C}_{r}^{\text{GT}}$ is the
corresponding ground truth color.

We also apply a weighted mask loss on all manifold layers, encouraging outer
layers to match the ground truth mask with higher weights and gradually
reducing the weight towards inner layers:

(9)
$\mathcal{L}_{\text{mask}}=\frac{1}{n_{T}}\sum_{i}^{n_{T}}\frac{exp(-\gamma\|l_{i}\|)}{|\overline{R}_{i}|}\sum_{r\in\overline{R}_{i}}CE(-s^{i}_{r},O_{r})$

where $\overline{R}$ is the set of non-surface rays, $O_{r}$ is the ground
truth mask, $s^{i}_{r}$ is the predicted manifold scalar at ray intersection
with the $i$-th isosurface, $CE(\cdot)$ is the binary cross entropy loss,
$l_{i}$ is the predefined level, and $\gamma=10$ is the attenuation factor.
The ground truth mask is obtained with an off-the-shelf foreground estimator
(Ke et al., 2022).

Following (Zheng et al., 2022), we supervise the deformation network with
pseudo FLAME ground truth, and encourage a lower Manhattan distance between
isosurfaces and FLAME surface for the skin region $R^{\text{skin}}$:

(10) $\displaystyle\mathcal{L}_{\text{flame}}$
$\displaystyle=\frac{1}{n_{T}}\sum_{i}^{n_{T}}\Big{[}\sum_{r\in
R_{i}}(\lambda_{e}\|\mathcal{E}_{r}-\mathcal{E}_{r}^{\text{GT}}\|_{2}+\lambda_{r}\|\mathcal{P}_{r}-\mathcal{P}_{r}^{\text{GT}}\|_{2}$
$\displaystyle+\lambda_{w}\|\mathcal{W}_{r}-\mathcal{W}_{r}^{\text{GT}}\|_{2})+\sum_{r\in
R^{\text{skin}}_{i}}\lambda_{d}\|\mathbf{x}_{r}-\mathbf{x}_{r}^{\text{GT}}\|_{2}\Big{]},$

where $\mathcal{E}_{r}$, $\mathcal{P}_{r}$, and $\mathcal{W}_{r}$ represent
the predicted values of the deformation network, $\mathbf{x}_{r}$ is the
position of the ray-isosurface intersection point, while
$\mathcal{E}^{\text{GT}}_{r}$, $\mathcal{P}^{\text{GT}}_{r}$,
$\mathcal{W}^{\text{GT}}_{r}$ and $\mathbf{x}^{\text{GT}}_{r}$ denote the
pseudo ground truth weights and positions defined by the nearest FLAME
vertices, $\lambda_{e}=1000$, $\lambda_{r}=1000$, $\lambda_{w}=0.1$,
$\lambda_{d}=1$ are coefficients.

In summary, the overall loss function is:

(11)
$\mathcal{L}_{\text{train}}=\lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}}+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}+\lambda_{\text{flame}}\mathcal{L}_{\text{flame}},$

where $\lambda_{\text{rgb}}=1$, $\lambda_{\text{mask}}=10$,
$\lambda_{\text{flame}}=1$ are loss weights.

Figure 4.  Visualization of extracted meshes under the given pose and
expression, shown from outer layers (on the left) to inner layers (on the
right). It can be seen that these multiple layers of meshes generally conform
to the shape of the head. They also offer the essential intersection points
needed to approximate the effects of volumetric rendering.

###  3.2. Computing Deformable Layered Meshes and Textures

Following the first stage, we bake the continuous implicit fields to layered
meshes and multiple textures for real-time rendering. We begin with mesh
extraction, using Marching Cubes (Lorensen and Cline, 1987) to extract
isosurfaces in canonical space for each predefined level. Since there are
excess triangle faces in the smooth area, and the back region of the head is
not visible during rendering, we apply a selective mesh simplification scheme
using Quadric Edge Collapse Decimation (Garland and Heckbert, 1997) to improve
real-time inference performance. Specifically, we first reduce the face count
to a target number and then further simplify faces with normal $\mathbf{n}$
that satisfy
$\arccos(\mathbf{n}\cdot\mathbf{n}_{\text{view}}^{T})>\theta_{max}$ to a
target percentage, where $\mathbf{n}_{\text{view}}$ is set to the Z+ axis (the
direction facing the head). Fig. 4 visualizes the representative deformable
mesh layers. More details are provided in Sec. 5.3.3 and Sec. A.4.

We bake deformation parameters onto extracted mesh vertex attributes to
perform mesh morphing at inference time. For each mesh vertex in canonical
space, we evaluate the deformation field to obtain the corresponding FLAME
weights, which we store as vertex attributes. We also evaluate the normalized
gradients of the manifold field as vertex normals. We apply UV unwrapping
using xaltas (jpcy, 2023) to acquire texture coordinates for each vertex and
densely sample all texels for baking radiance bases and position features from
$R_{\text{basis}}$. Specifically, for each texel, we uniformly sample 16
points and find their corresponding positions in canonical space. We then
evaluate the radiance field to obtain a smooth estimation of colors
$\\{\mathbf{c}_{1},\mathbf{c}_{2},\dots,\mathbf{c}_{n_{T}}\\}$ and occupancies
$\\{\alpha_{1},\alpha_{2},\dots,\alpha_{n_{T}}\\}$ for $n_{T}$ radiance bases,
as well as the position feature $\mathbf{f}_{p}\in\mathbb{R}^{d_{p}}$. We
store these as $n_{T}$ RGBA textures and a deep texture with $d_{p}$ channels,
all quantized to the range [0, 255] and stored as 8-bit unsigned integers.

Figure 5. Visualization of the rendered color and depth, as well as the
appearance and geometry of isosurfaces, from outer layers to inner layers.

###  3.3. Fine-tuning Mesh Textures

To minimize the errors introduced during the baking stage, we include a fine-
tuning stage using differential rasterization to directly optimize mesh
assets. As optimizing vertex positions of the discretized mesh is prone to
local minima and may introduce undesirable intersections between manifold
levels, we fix the mesh geometry and only optimize textures during fine-
tuning. Fine-tuning textures helps eliminate artifacts at texture seams
introduced in the UV parameterization process and refines appearance details
that may not be well-reconstructed in the first stage. We render all mesh
layers from inner levels to outer levels respectively, where the light-
weighted MLP is evaluated at all rendered pixels to blend radiance texture
bases, and finally combine them with standard alpha composition to produce the
final color. We also optimize the camera translation and FLAME pose of each
frame during the fine-tuning process, alleviating the errors of extracted
camera extrinsic in datasets. We apply the same RGB loss from the first stage,
and a VGG perceptual loss (Johnson et al., 2016) since we now render the whole
image instead of randomly sampled pixels:

(12)
$\mathcal{L}_{\text{vgg}}=\|F_{\text{vgg}}(\mathbf{C})-F_{\text{vgg}}(\mathbf{C}^{\text{GT}})\|,$

where $\mathbf{C}$ and $\mathbf{C}^{\text{GT}}$ are the rendered and ground
truth images respectively, $F_{\text{vgg}}(\cdot)$ outputs features from the
first four convolution layers of a pre-trained VGG network. The total loss for
fine-tuning is:

(13)
$\mathcal{L}_{\text{finetune}}=\lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}}+\lambda_{\text{vgg}}\mathcal{L}_{\text{vgg}},$

where $\lambda_{\text{rgb}}=1$, $\lambda_{\text{vgg}}=0.1$.

###  3.4. Real-Time Rendering

We implement an interactive viewer with Javascript and WebGL2 on an HTML
webpage. Our final representation for real-time rendering consists of four
components: multi-layer meshes with vertex attributes (i.e., UVs, normals,
FLAME weights), position and radiance texture atlases for each mesh, vertices
and joint regressor of the FLAME template mesh, and weights of the appearance
decoder. We compute blendshape offsets and LBS weights of each vertex from
global expression and pose parameters in the vertex shader. The spatial MLP
runs in the fragment shader, which calculates the blending coefficients of
multiple radiance bases for all shaded pixels of each mesh layer in parallel.
Specifically, the real-time rendering process includes three steps: (1) using
expression and pose parameters at the current frame to calculate the global
pose correctives and LBS bone transformations from the FLAME template mesh;
(2) running the global MLP to produce dynamic weights of the spatial MLP,
which are stored in a texture and then passed to the fragment shader; (3)
drawing the layered meshes from inner to outer layers to ensure correct
ordering in alpha composition (see Fig. 5). By harnessing the optimized
rasterization pipeline of modern GPUs, our interactive viewer renders high-
quality, dynamic head avatars in real time.

Table 1. Quantitative Comparisons against SOTA methods.

|  L1$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  KD$\downarrow$ |  FPS$\uparrow$
---|---|---|---|---|---|---
BiLayer (2020) |  $0.0765$ |  $15.39$ |  $0.830$ |  $0.172$ |  $5.77$ |  $3.39$
NerFace (2021) |  $0.0219$ |  $25.68$ |  $0.888$ |  $0.112$ |  $5.44$ |  $0.15$
IMavatar (2022) |  $0.0242$ |  $24.18$ |  $0.881$ |  $0.118$ |  $5.13$ |  $0.02$
PointAvatar (2023) |  $0.0180$ |  $27.42$ |  $0.907$ |  $0.060$ |  $5.51$ |  $8.53$
Ours |  $0.0147$ |  $28.66$ |  $0.917$ |  $0.057$ |  $4.97$ |  $804$
NHA (2022) |  $0.0204$ |  $23.85$ |  $0.902$ |  $0.078$ |  $6.08$ |  $12.30$
Ours (w/o cloth) |  $0.0108$ |  $30.93$ |  $0.943$ |  $0.045$ |  $4.97$ |  $831$

##  4\. Implementation Details

We select 8 manifold layers for manifold learning, with predefined levels
sampled evenly from $-0.2$ to $0$, and use $n_{T}=16$ and $d_{p}=8$ for
appearance learning. In the baking stage, we set the target number of faces of
each mesh layer to $10,000$, and further simplify backward-facing triangles
with $\theta_{max}=120^{\circ}$. Position feature and radiance bases are
sampled and stored in $1024\times 1024$ sized images. During fine-tuning, we
utilize nvdiffrast (Laine et al., 2020) for differential rasterization on the
layered meshes, with training frames scaled to $256\times 256$ and $512\times
512$ in the training and fine-tuning stages, respectively. Network
architecture details are provided in the supplementary material. We implement
the training procedure in PyTorch (Paszke et al., 2019), using Adam (Kingma
and Ba, 2015) optimizer with an initial learning rate set to $3\times
10^{-4}$, and decays by a factor of $0.5$ every 20k iterations. It takes about
a day to train our model for a subject on a single RTX 3090 GPU. We recognize
that optimizing training time (Müller et al., 2022; Chen et al., 2023b) is
beyond the scope of this paper, as our primary objective is to achieve real-
time inference and rendering.

##  5\. Experiments

We conduct experiments using monocular videos of 8 subjects gathered from
public datasets of PointAvatar (Zheng et al., 2023), NHA (Grassal et al.,
2022), NerFace (Gafni et al., 2021), and two Youtube videos (ceZw5ZhIqwE and
-mIaZPqoOYw). The training dataset contains 1,000-4,000 frames, while the test
dataset includes frames with novel expressions and poses. We assess our method
by examining the image quality of self-reenactment and controlled head avatar
synthesis, and the rendering speed on diverse devices, along with an ablation
study.

Figure 6.  Comparisons of self-reenactment results. BiLayer (Zakharov et al.,
2020) fails to faithfully preserve the identity in the results. NerFace (Gafni
et al., 2021) and IMavatar (Zheng et al., 2022) generate blurry results due to
the smooth nature of coordinate-based MLP. While NHA (Grassal et al., 2022)
generates sharp textures, it struggles to reconstruct semi-transparent thin
structures such as hair. PointAvatar (Zheng et al., 2023) generates volumetric
effects well but fails to capture details of the mouth interior. In contrast,
our approach excels in capturing finer details, particularly in complex
structures like hair and mouth. Ground truth videos _Stabenow©Senate
Democrats_ and _Obama©The Obama White House_ in the last two rows are from
public domain.

###  5.1. Comparison of Quality and Speed

Table 2. Quantitative comparison of results after each stage.

|  L1$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  KD$\downarrow$
---|---|---|---|---|---
Stage-1 (Training) |  $0.0148$ |  $28.14$ |  $0.919$ |  $0.085$ |  $4.94$
Stage-2 (Baking) |  $0.0170$ |  $27.87$ |  $0.915$ |  $0.087$ |  $5.02$
Stage-3 (Fine-tuning) |  $0.0147$ |  $28.66$ |  $0.917$ |  $0.057$ |  $4.97$

Figure 7.  Novel view synthesis results by changing yaw (left) and pitch
(right) angles. The bottom right figure is from _Stabenow©Senate Democrats_.
Figure 8. Expression and pose editing results. For each subject, the first
column shows a source frame, followed by the editings of expressions (columns
2 and 3) and poses (columns 4 and 5).

Given a training portrait video, we execute self-reenactment driven by a
testing video featuring the same subject and evaluate the results against SOTA
methods. Specifically, we compare our approach with representative methods,
including Bi-Layer (Zakharov et al., 2020), a one-shot image-based technique;
NerFace (Gafni et al., 2021), a NeRF-based method; Neural Head Avatars (NHA)
(Grassal et al., 2022), which employs an explicit 3D morphable mesh; IMavatar
(Zheng et al., 2022), which utilizes a neural implicit surface; PointAvatar
(Zheng et al., 2023), a point clouds-based approach.

To evaluate image quality, we report $L_{1}$-distance, PSNR, SSIM, and LPIPS
(Zhang et al., 2018) metrics. For expression accuracy of face reenactment, we
measure the Keypoint Distance (KD) between the detected facial landmarks and
the ground truth. We also report the average inference speed of all methods,
measured on a desktop with an RTX 3090 GPU at $512\times 512$ resolution. Our
method’s frame rate is measured in the interactive WebGL renderer, while other
methods implemented on PyTorch (Paszke et al., 2019) use a batch size of 1
during inference, simulating interactive face reenactment.

Tab. 1 lists the quantitative metrics for self-reenactment results. Note that
since NHA (Grassal et al., 2022) does not generate the cloth part, it is
excluded from general comparisons. In summary, our method not only
effortlessly surpasses the maximum screen refresh rate for rendering speed but
also achieves the highest rendering quality when compared to SOTA methods.
Fig. 6 illustrates the qualitative comparison of all methods for various
subjects. The results indicate that our approach excels in capturing finer
details, particularly in complex structures like hair and mouth, surpassing
the performance of the other techniques.

We present visual quality metrics at each phase in Tab. 2 to gain insights
into how the baking and fine-tuning steps impact this aspect. Notably, after
the second baking phase, a decrease in visual quality metrics is observed.
This decline is due to errors stemming from the discretization of geometric
and appearance attributes, such as the resolution involved in the marching
cubes operation, the simplification of mesh faces, and texture seam issues
arising from UV parameterization. However, these errors are rectified during
the third fine-tuning phase, resulting in quality metrics that actually exceed
those recorded after the initial training stage. Improvement in this stage is
attributed to the elimination of low-frequency bias inherent in coordinate-
based MLPs, and the direct optimization of extracted textures that enhance
high-frequency textural details. The fine-tuning phase also eradicates texture
seams observed in the baking stage and specifically fine-tunes the LPIPS
metrics by rendering complete images rather than sampled rays.

Figure 9. Cross-identity reenactment results. We use the expression and pose
sequences extracted from a driving video to animate different subjects.

To evaluate the rendering performance of our method, we report the average
frame rates (FPS) at different resolutions on commodity devices in Tab. 3.
Leveraging optimized triangle-based rasterization and a lightweight appearance
decoder in pixel shaders, our approach delivers high-framerate rendering even
on devices with limited computational power. To the best of our knowledge,
_BakedAvatar_ is the first NeRF-based approach to achieve real-time animatable
head avatars with SOTA quality on mobiles.

Table 3.  Rendering speed (FPS) of our method on commodity devices. The tested
devices include a gaming laptop with an RTX 3060 Mobile GPU, a tablet (iPad
Pro 2021), and a mobile phone (OnePlus 11). For the laptop, plugged and
unplugged scenarios are tested. The tablet and mobile devices had a maximum
FPS cap of 60.

Resolution | Laptop (plugged/unplugged) | Tablet | Mobile
---|---|---|---
$256\times 256$ |  $381$/$205$ | $60$ | $53$
$512\times 512$ |  $239$/$189$ | $59$ | $41$
$1024\times 1024$ |  $114$/$102$ | $27$ | $25$
$2048\times 2048$ |  $42$/$33$ | $9$ | $9$

###  5.2. Controlled Head Avatar Synthesis

Since _BakedAvatar_ targets real-time rendering of reenactable head avatars,
it is crucial to evaluate the synthesis quality of manually controlled novel
expression, pose and view. Fig. 7 illustrates the results of novel view
synthesis. We conduct editing by adding an offset to the FLAME expression and
pose coefficients that represent novel facial expressions not covered during
training. Fig. 8 demonstrates the expression and pose editing results for
source frames, indicating that _BakedAvatar_ learns a generalized dynamic
geometry and appearance representation of the head avatar. Additionally, we
show cross-identity face reenactment results in Fig. 9, where driving
expression and pose come from different subjects. More results are provided in
the supplementary video.

Table 4. Ablation of rendering quality and speed.

|  | L1$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FPS$\uparrow$
---|---|---|---|---|---|---
Mesh Layer |  $1$ | $0.0188$ | $25.47$ | $0.913$ | $0.0757$ | $781$
$2$ | $0.0179$ | $25.88$ | $0.918$ | $0.0695$ | $435$
$4$ | $0.0172$ | $26.34$ | $0.921$ | $0.0663$ | $223$
Texture Basis |  $1$ | $0.0189$ | $25.28$ | $0.910$ | $0.0723$ | $573$
$2$ | $0.0177$ | $25.93$ | $0.919$ | $0.0673$ | $220$
$4$ | $0.0173$ | $26.19$ | $0.920$ | $0.0662$ | $183$
$8$ | $0.0168$ | $26.59$ | $0.923$ | $0.0645$ | $156$
Mesh Prec. |  $\times 1/2$ | $0.0173$ | $26.04$ | $0.921$ | $0.0680$ | $144$
$\times 2$ | $0.0159$ | $27.00$ | $0.927$ | $0.0629$ | $89$
$\times 4$ | $0.0158$ | $27.43$ | $0.925$ | $0.0648$ | $65$
Texture Res. |  $256^{2}$ | $0.0177$ | $26.05$ | $0.919$ | $0.0687$ | $134$
$512^{2}$ | $0.0167$ | $26.54$ | $0.923$ | $0.0644$ | $127$
w/o fine-tuning | $0.0185$ | $26.16$ | $0.917$ | $0.0937$ | $-$
w/o cam&pose opt. | $0.0185$ | $25.80$ | $0.914$ | $0.0720$ | $-$
w/o VGG loss | $0.0167$ | $26.53$ | $0.924$ | $0.0762$ | $-$
Ours (full) | $0.0163$ | $27.13$ | $0.925$ | $0.0643$ | $114$

Figure 10. Ablation study results. Given a source frame (column 1), we
visually assess the effect of mesh layers (columns 2 and 3), radiance bases
(columns 4 to 6), fine-tuning (column 7), camera and pose optimization (column
8), and perceptual loss (column 9) on self-reenactment results.

###  5.3. Ablation Studies

We report the rendering quality and speed of our full model and other design
choices. FPS is measured on an RTX 3060 Mobile GPU.

####  5.3.1. Mesh Layers

We compare the quality and speed with various numbers of mesh layers. For a
single mesh layer, our method is the same as traditional morphable mesh
representation, with an expression- and pose-conditioned neural texture. The
results, presented in Tab. 4 and Fig. 10, indicate that increasing the number
of mesh layers significantly improves quality. Nevertheless, this improvement
is accompanied by slower rendering times. Our multi-layer meshes effectively
approximate volumetric effects and achieve better rendering quality for
complex structures.

####  5.3.2. Radiance Bases

As our expression- and pose-dependent appearance decouples dynamic appearance
as multiple radiance texture bases, we compare the impact on rendering quality
with various numbers of bases. When the number of bases is set to $1$, the
appearance is identical to a fully static diffuse color representation. As
shown in Tab. 4 and Fig. 10, the single basis fails to represent dynamic
appearance caused by expression and pose changes, and increasing the number of
texture bases gradually improves quality.

####  5.3.3. Mesh and Texture Complexity

We further analyze how the precision of extracted meshes and textures affects
the final rendering quality and speed. The results from different
configurations of mesh (expressed as multiples of the target triangle count)
and texture extracting setups are listed in Tab. 4. As we encourage smooth
geometry in Sec. 3.1.1, the precision of the mesh extraction process does not
significantly affect the synthesis quality of our representation. Thus we can
utilize a simple mesh with a moderate number of triangles so as to be rendered
with lower computation cost.

Figure 11. Failure cases. (a) Artifact occurs when rendering the backside of
the head avatar. (b) _BakedAvatar_ does not specifically model eyeballs, which
may lead to inaccurate eye rendering. Figure 12.  Details of the mouth region
with extrapolated expressions that are not well covered by the training set.
Although such expression extrapolation is feasible, the visual quality might
be inferior due to the difficulty of accurate reconstruction of geometry and
deformation around the mouth interior and teeth. The right figure is from
_Obama©The Obama White House_.

####  5.3.4. Fine-tuning

We assess the quality enhancement of our comprehensive model compared to
alternative design choices during the fine-tuning phase, as presented in Table
4 and Figure 10. Specifically, we compare our complete method with our
approach excluding the fine-tuning phase, without camera and pose tuning, and
without the incorporation of perceptual loss. The absence of fine-tuning
results in the omission of high-frequency appearance details. Furthermore, the
inclusion of camera and pose tuning, as well as perceptual loss, contributes
to improved learning of intricate structures.

##  6\. Conclusion

We present _BakedAvatar_ , a new representation for synthesizing person-
specific 4D head avatars. Our key idea is to bake the learned deformation,
geometry, and appearance neural fields into layered meshes and multiple
texture bases, combining with an efficient pose-, expression- and view-
dependent appearance representation that can be evaluated with lightweight
computation. We demonstrate that our method is able to achieve SOTA-quality
synthesis results while running at real-time frame rates with the
rasterization pipeline, enabling interactive applications such as facial
reenactment and expression editing, even on mobile devices with limited
computation resources.

##### Limitations and Future work

The capability of _BakedAvatar_ is limited by the number of mesh layers and
the detail quality of mesh textures, and may not be able to achieve full
volumetric and refraction effects faithfully, such as thin hair strands and
distortion of eyeglasses. Our methods can extrapolate to unseen head poses and
facial expressions that are not covered by the training data, nevertheless,
the synthesis outcome for extreme views or expressions may be of inferior
quality. Fig. 11 shows failure examples. In particular, visual artifacts of
complex novel pose and expression around the mouth region might occur, as
demonstrated in Fig. 12. This is possibly caused by the mismatch between the
learned FLAME weights of extracted meshes and the ground truth, as it is
challenging to learn deformation accurately due to appearance and geometry
complexity in the mouth region. This can be addressed with a more accurate
reconstruction of deformation and a non-linear representation of the dynamic
appearance, which we leave as future work.

During the dataset pre-processing steps, we adopt a FLAME tracking pipeline
from (Zheng et al., 2022), which initially estimates FLAME shape and
expression parameters with DECA (Feng et al., 2021b), and then optimizes them
with detected facial keypoints, leveraging loss functions based on assumptions
of temporal sequences. While the derived FLAME parameters are sufficient for
implicit field reconstruction, flickering artifacts can be observed for
specific identities (Fig. 6, Row 1-2) in the facial reenactment task. This
issue is likely to be attributed to the temporal inconsistencies in the
detected facial keypoints of consecutive frames. Our attempts indicate that
slightly smoother motion can be obtained with a better facial landmark
detector (Bazarevsky et al., 2019) and higher weights on temporal consistency
loss. Still, better results might be achieved with a photometric tracker, or
temporal consistent optimization of the FLAME parameter sequences during
training. However, as the focus of this paper is not on facial tracking
techniques, we leave the improvements as future work.

Our method can be further extended in several directions. To facilitate
relighting applications, the appearance representation can be further divided
into distinct elements like environment lighting and albedo/specular maps,
integrating these with a physics-based lighting model. To enhance the
precision and realism of iris movements, incorporating specific modeling for
eyeballs could be beneficial. Expanding our technique to generate head avatars
that are person-agnostic could also be an interesting direction to explore.

###### Acknowledgements.

The authors thank the anonymous reviewers for their insightful feedback and
also appreciate the consent given by authors from other publications to use
video datasets of the featured individuals. This work was supported by the
National Natural Science Foundation of China (Project Number: 61932003) and
the Fundamental Research Funds for the Central Universities.

## References

  * (1)
  * Attal et al. (2023) Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhöfer, Johannes Kopf, Matthew O’Toole, and Changil Kim. 2023\.  HyperReel: High-Fidelity 6-DoF Video With Ray-Conditioned Sampling. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16610–16620.
  * Atzmon and Lipman (2020) Matan Atzmon and Yaron Lipman. 2020.  Sal: Sign agnostic learning of shapes from raw data. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 2565–2574.
  * Bai et al. (2023) Ziqian Bai, Feitong Tan, Zeng Huang, Kripasindhu Sarkar, Danhang Tang, Di Qiu, Abhimitra Meka, Ruofei Du, Mingsong Dou, Sergio Orts-Escolano, Rohit Pandey, Ping Tan, Thabo Beeler, Sean Fanello, and Yinda Zhang. 2023.  Learning Personalized High Quality Volumetric Head Avatars From Monocular RGB Videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16890–16900.
  * Bazarevsky et al. (2019) Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019\.  Blazeface: Sub-millisecond neural face detection on mobile gpus.  _arXiv preprint arXiv:1907.05047_ (2019).
  * Cao et al. (2013) Chen Cao, Yanlin Weng, Shun Zhou, Yiying Tong, and Kun Zhou. 2013.  Facewarehouse: A 3d facial expression database for visual computing.  _IEEE Transactions on Visualization and Computer Graphics_ 20, 3 (2013), 413–425.
  * Chen et al. (2021a) Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. 2021a.  Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 14124–14133.
  * Chen et al. (2023b) Xu Chen, Tianjian Jiang, Jie Song, Max Rietmann, Andreas Geiger, Michael J Black, and Otmar Hilliges. 2023b.  Fast-SNARF: A fast deformer for articulated neural fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2023).
  * Chen et al. (2021b) Xu Chen, Yufeng Zheng, Michael J Black, Otmar Hilliges, and Andreas Geiger. 2021b.  Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 11594–11604.
  * Chen et al. (2023a) Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2023a.  MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16569–16578.
  * Deng et al. (2022) Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. 2022\.  Gram: Generative radiance manifolds for 3d-aware image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10673–10683.
  * Feng et al. (2021a) Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021a.  Learning an Animatable Detailed 3D Face Model from In-the-Wild Images.  _ACM Transactions on Graphics (ToG), Proc. SIGGRAPH_ 40, 4 (Aug. 2021), 88:1–88:13.
  * Feng et al. (2021b) Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021b.  Learning an Animatable Detailed 3D Face Model from In-The-Wild Images.  _ACM Transactions on Graphics, (Proc. SIGGRAPH)_ 40, 8.  https://doi.org/10.1145/3450626.3459936
  * Fridovich-Keil et al. (2022) Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2022\.  Plenoxels: Radiance fields without neural networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 5501–5510.
  * Gafni et al. (2021) Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. 2021.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8649–8658.
  * Gao et al. (2022) Xuan Gao, Chenglai Zhong, Jun Xiang, Yang Hong, Yudong Guo, and Juyong Zhang. 2022\.  Reconstructing personalized semantic facial nerf models from monocular video.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022), 1–12.
  * Garbin et al. (2021) Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021\.  Fastnerf: High-fidelity neural rendering at 200fps. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 14346–14355.
  * Garland and Heckbert (1997) Michael Garland and Paul S Heckbert. 1997.  Surface simplification using quadric error metrics. In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_. 209–216.
  * Grassal et al. (2022) Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. 2022.  Neural head avatars from monocular RGB videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18653–18664.
  * Guo et al. (2023) Yuan-Chen Guo, Yan-Pei Cao, Chen Wang, Yu He, Ying Shan, Xiaohu Qie, and Song-Hai Zhang. 2023.  VMesh: Hybrid Volume-Mesh Representation for Efficient View Synthesis.  _arXiv preprint arXiv:2303.16184_ (2023).
  * Hedman et al. (2021) Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. 2021\.  Baking neural radiance fields for real-time view synthesis. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5875–5884.
  * Johnson et al. (2016) Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016.  Perceptual losses for real-time style transfer and super-resolution. In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_. Springer, 694–711.
  * jpcy (2023) jpcy. 2023.  xatlas.  https://github.com/jpcy/xatlas
  * Ke et al. (2022) Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson WH Lau. 2022.  Modnet: Real-time trimap-free portrait matting via objective decomposition. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 1140–1147.
  * Khakhulin et al. (2022) Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov. 2022.  Realistic One-shot Mesh-based Head Avatars. In _European Conference of Computer vision (ECCV)_.
  * Kim et al. (2018) Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt. 2018.  Deep Video Portraits.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 163\.
  * Kingma and Ba (2015) Diederik Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _International Conference on Learning Representations (ICLR)_. San Diega, CA, USA.
  * Laine et al. (2020) Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020\.  Modular Primitives for High-Performance Differentiable Rendering.  _ACM Transactions on Graphics_ 39, 6 (2020).
  * Li et al. (2017) Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. Graph._ 36, 6 (2017), 194–1.
  * Lin et al. (2022) Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. 2022.  Efficient Neural Radiance Fields for Interactive Free-viewpoint Video. In _SIGGRAPH Asia 2022 Conference Papers_. 1–9.
  * Liu et al. (2022) Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022.  Devrf: Fast deformable voxel radiance fields for dynamic scenes.  _Advances in Neural Information Processing Systems_ 35 (2022), 36762–36775.
  * Liu et al. (2020) Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. 2020.  Neural sparse voxel fields.  _Advances in Neural Information Processing Systems_ 33 (2020), 15651–15663.
  * Lorensen and Cline (1987) William E Lorensen and Harvey E Cline. 1987.  Marching cubes: A high resolution 3D surface construction algorithm.  _ACM siggraph computer graphics_ 21, 4 (1987), 163–169.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  Nerf: Representing scenes as neural radiance fields for view synthesis. In _European conference on computer vision_.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ 41, 4 (2022), 1–15.
  * Neff et al. (2021) Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. 2021.  DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 45–59.
  * Nguyen et al. (2022) Thanh Thi Nguyen, Quoc Viet Hung Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, Thien Huynh-The, Saeid Nahavandi, Thanh Tam Nguyen, Quoc-Viet Pham, and Cuong M Nguyen. 2022\.  Deep learning for deepfakes creation and detection: A survey.  _Computer Vision and Image Understanding_ 223 (2022), 103525.
  * Oechsle et al. (2021) Michael Oechsle, Songyou Peng, and Andreas Geiger. 2021\.  Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5589–5599.
  * Park et al. (2021a) Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. 2021a.  Nerfies: Deformable neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5865–5874.
  * Park et al. (2021b) Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. 2021b.  HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields.  _ACM Trans. Graph._ 40, 6, Article 238 (dec 2021).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Paysan et al. (2009) Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009.  A 3D Face Model for Pose and Illumination Invariant Face Recognition. In _2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance_. 296–301.  https://doi.org/10.1109/AVSS.2009.58
  * Pumarola et al. (2021) Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021.  D-nerf: Neural radiance fields for dynamic scenes. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10318–10327.
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 14335–14345.
  * Reiser et al. (2023) Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. 2023\.  Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–12.
  * Siarohin et al. (2019) Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019\.  First Order Motion Model for Image Animation. In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019\.  Deferred neural rendering: Image synthesis using neural textures.  _Acm Transactions on Graphics (TOG)_ 38, 4 (2019), 1–12.
  * Trevithick et al. (2023) Alex Trevithick, Matthew Chan, Michael Stengel, Eric Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. 2023\.  Real-time radiance fields for single-image portrait view synthesis.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–15.
  * Tucker and Snavely (2020) Richard Tucker and Noah Snavely. 2020.  Single-view View Synthesis with Multiplane Images. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Wang et al. (2022) Liao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao, Yanshun Zhang, Yingliang Zhang, Minye Wu, Jingyi Yu, and Lan Xu. 2022.  Fourier plenoctrees for dynamic radiance field rendering in real-time. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13524–13534.
  * Wang et al. (2023) Lizhen Wang, Xiaochen Zhao, Jingxiang Sun, Yuxiang Zhang, Hongwen Zhang, Tao Yu, and Yebin Liu. 2023.  StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video. In _ACM SIGGRAPH 2023 Conference Proceedings_.
  * Wang et al. (2021) Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. 2021\.  NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction.  _NeurIPS_ (2021).
  * Wen et al. (2020) Xin Wen, Miao Wang, Christian Richardt, Ze-Yin Chen, and Shi-Min Hu. 2020.  Photorealistic audio-driven video portraits.  _IEEE Transactions on Visualization and Computer Graphics_ 26, 12 (2020), 3457–3466.
  * Wizadwongsa et al. (2021) Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021\.  Nex: Real-time view synthesis with neural basis expansion. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8534–8543.
  * Wu et al. (2022) Yue Wu, Yu Deng, Jiaolong Yang, Fangyun Wei, Qifeng Chen, and Xin Tong. 2022\.  Anifacegan: Animatable 3d-aware face image generation for video avatars.  _Advances in Neural Information Processing Systems_ 35 (2022), 36188–36201.
  * Xu et al. (2023a) Yuelang Xu, Lizhen Wang, Xiaochen Zhao, Hongwen Zhang, and Yebin Liu. 2023a.  AvatarMAV: Fast 3D Head Avatar Reconstruction Using Motion-Aware Neural Voxels. In _ACM SIGGRAPH 2023 Conference Proceedings_.
  * Xu et al. (2023b) Yuelang Xu, Hongwen Zhang, Lizhen Wang, Xiaochen Zhao, Huang Han, Qi Guojun, and Yebin Liu. 2023b.  LatentAvatar: Learning Latent Expression Code for Expressive Neural Head Avatar. In _ACM SIGGRAPH 2023 Conference Proceedings_.
  * Yang et al. (2022) Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. 2022.  Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In _European Conference on Computer Vision_. Springer, 597–614.
  * Yariv et al. (2021) Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. 2021\.  Volume rendering of neural implicit surfaces.  _Advances in Neural Information Processing Systems_ 34 (2021), 4805–4815.
  * Yariv et al. (2023) Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, and Ben Mildenhall. 2023.  BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis. In _ACM SIGGRAPH 2023 Conference Proceedings_ (Los Angeles, CA, USA) _(SIGGRAPH ’23)_. Association for Computing Machinery, New York, NY, USA, Article 46, 9 pages.  https://doi.org/10.1145/3588432.3591536
  * Yu et al. (2021) Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021\.  Plenoctrees for real-time rendering of neural radiance fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5752–5761.
  * Yu et al. (2023) Wangbo Yu, Yanbo Fan, Yong Zhang, Xuan Wang, Fei Yin, Yunpeng Bai, Yan-Pei Cao, Ying Shan, Yang Wu, Zhongqian Sun, et al. 2023\.  NOFA: NeRF-based One-shot Facial Avatar Reconstruction. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–12.
  * Zakharov et al. (2020) Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. 2020.  Fast bi-layer neural synthesis of one-shot realistic head avatars. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_. Springer, 524–540.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zheng et al. (2022) Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. 2022.  Im avatar: Implicit morphable head avatars from videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13545–13555.
  * Zheng et al. (2023) Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. 2023.  PointAvatar: Deformable Point-based Head Avatars from Videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zielonka et al. (2022) Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2022\.  Towards Metrical Reconstruction of Human Faces. In _European Conference on Computer Vision (ECCV)_. Springer International Publishing.
  * Zielonka et al. (2023) Wojciech Zielonka, Timo Bolkart, and Justus Thies. 2023\.  Instant Volumetric Head Avatars. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4574–4584.

##  Appendix A Appendix

###  A.1. Network Architecture

In the training stage, we first learn all fields as coordinate-based implicit
MLPs. Specifically, we use a $6$ layer MLP with $256$ hidden neurons for the
manifold network, and a $3$ layer MLP with $128$ hidden neurons for the
deformer network, both with Softplus activation. Since the radiance network
outputs multiple radiance bases, we design it as a trunk of a $4$ layer MLP, a
head that outputs the shared position feature $f_{p}$, and several separate
heads of a $2$ layer MLP that outputs radiance $c_{i}$ and occupancy
$\alpha_{i}$ of each texture basis, all with $512$ hidden neurons and ReLU
activation (except for the last layer), as illustrated in Fig. 13. Positional
encoding with frequency numbers $5,12$ are used for the manifold and radiance
network respectively, while the deformer network does not use positional
encoding.

For the appearance decoder, we use a $3$-layer MLP of $64$ hidden neurons with
PReLU activation for the global MLP and a $2$-layer MLP with $16$ hidden
neurons for the spatial MLP. After the training and fine-tuning stage, the
global MLP weights of the appearance decoder are exported, together with the
meshes and textures.

Figure 13. Network structure of the canonical radiance field. There are
multiple heads for outputting $n_{T}$ radiance bases ($n_{T}=16$ in the main
paper), and a head for outputting the shared position feature $f_{p}$.

###  A.2. Training of the Forward Deformer

In Section 3.1.2, we learn the FLAME blendshapes and linear blend skinning
weights in canonical space. As the FLAME template has a closed mouth state,
causing a sharp transition of LBS weights that can not be learned well by MLP,
we adopt the approach from PointAvatar (Zheng et al., 2023) by introducing a
custom canonical space with a half-opened mouth state. Specifically, we first
warp the points in the custom canonical space to the original FLAME space,
then use FLAME expression and pose coefficients to warp them to the deformed
space with learned blendshapes and LBS weights. When exporting the extracted
multi-layer meshes at the baking stage in Section 3.2 of the main paper, we
transform all vertex positions and normals from canonical space to the
original FLAME template space, removing the first warping procedure at
inference time.

###  A.3. Implementation details of real-time rendering

Table 5. Total polygon count and uncompressed storage size (MiB) under various mesh precision and texture resolution settings. Mesh Precision | Vertices | Triangles | Storage under tex. size
---|---|---|---
$256^{2}$ |  $512^{2}$ |  $1024^{2}$
$\times 1/2$ | $13544$ | $19017$ |  $41$ |  $84$ |  $248$
$\times 1$ (Ours) | $28882$ | $44298$ |  $58$ |  $107$ |  $266$
$\times 2$ | $61401$ | $100492$ |  $90$ |  $135$ |  $302$
$\times 4$ | $128540$ | $220570$ |  $161$ |  $206$ |  $374$

In the baking stage outlined in Section 3.2, we employ a mesh simplification
scheme to reduce the complexity of the extracted mesh to a desired number of
triangles. To assess the complexity of the resulting meshes, we provide the
total count of vertices and triangles for the extracted layered meshes, along
with their storage requirements at different mesh precision and texture
resolutions, as shown in Table 5.

During the texture export process, we utilize quantization-aware training to
address quantization errors that may arise when converting continuous color
values into 8-bit integers, by clamping and rounding values during the forward
pass while retaining their floating-point gradient in the backward pass. Upon
completion of the fine-tuning process, we export two expanded RGBA textures:
the position texture and multiple radiance texture bases, by tiling the multi-
channel image as a grid of images on a large texture atlas, which allows
better compression with PNG and avoiding exceeding the texture unit limit in
the WebGL implementation.

To ensure the correct rendering order for the alpha composition result, as
described in Section 3.4 of the main paper, we draw the multi-layer meshes
from the inner layers to the outer layers. However, the rasterization order of
triangles within the same mesh layer is not deterministic, which may lead to
incorrect alpha composition results. To deal with the ordering problem of
transparent triangles in the same layer, each mesh layer is rendered in two
passes: the first pass acquires the depth of forefront triangles by only
writing to the depth buffer, and the second pass draws pixels with depth less
or equal than the current value in the depth buffer using depth test. This
approach effectively occludes each mesh layer with minimal additional
computational overhead.

###  A.4. Details of Performance Measurement

The performance of our representation is measured with the WebGL2 interactive
renderer using browsers under each device. For the desktop and laptop
computers, we close all other applications and disable the frame rates limit
and vertical synchronization in Chrome by setting:

> \----disable-gpu-vsync ----disable-frame-rate-limit

We measure the average frame rates of 6 avatar subjects when rendering a
segment of the facial reenactment sequence. We set the camera field-of-view to
$14^{\circ}$ to ensure the head avatar roughly covers the entire rendering
canvas. When evaluating baseline methods implemented in the PyTorch framework,
we measured their average inference time on a desktop computer equipped with
an NVIDIA RTX 3090 GPU. We use a batch size of $1$ with gradient computation
disabled to simulate the facial reenactment application with real-time user
inputs, and exclude all irrelevant Python operations.

To compare the energy efficiency of different methods, we provide the
performance-to-watt ratios for our method and other approaches in Table 6. Our
representation demonstrates higher energy efficiency compared to alternative
head avatar synthesis methods. Consequently, it enables real-time head avatar
synthesis on devices with highly limited computational resources, such as
tablets and mobile phones. The primary factor contributing to the reduced
computational cost of our method is the baked radiance representation, which
eliminates the need for evaluating large neural networks during inference.
Additionally, our approach employs rasterization-based rendering instead of
the more computationally expensive ray-marching procedure. While Neural Head
Avatar (Grassal et al., 2022) and PointAvatar (Grassal et al., 2022) also
utilize rasterization-based rendering, their runtime performance remains
constrained by the size of their neural networks.

Table 6. Energy Efficiency Comparison. FPS is measured under $512\times 512$ resolution.  Method | Watts$\downarrow$ | FPS$\uparrow$ | FPS/W$\uparrow$
---|---|---|---
BiLayer (2020) | $350$ | $3.39$ | $0.00968$
NerFace (2021) | $350$ | $0.15$ | $0.00042$
IMavatar (2022) | $350$ | $0.02$ | $0.00005$
PointAvatar (2023) | $350$ | $8.53$ | $0.02437$
NHA (2022) | $350$ | $12.30$ | $0.03514$
Ours (Desktop) | $350$ | $804$ | $2.29714$
Ours (Laptop) | $105$ | $239$ | $2.27619$
Ours (Tablet) | $17$ | $59$ | $3.47058$
Ours (Mobile) | $13$ | $41$ | $3.15384$

###  A.5. Ethical Considerations

In the research presented within this paper, we introduce novel techniques
capable of reconstructing high-fidelity dynamic 3D avatars from monocular
videos and performing expression transfer in real-time across various devices,
including mobile platforms. This method’s capacity to synthesize a realistic
rendering of individuals, with detailed control over expression and head pose,
is not devoid of significant ethical considerations. Specifically, these
techniques harbor the potential for malevolent utilization, such as
fabricating authentic-seeming portrayals of individuals within contexts they
have never engaged in, consequently misleading viewers and possibly resulting
in defamation or harm to the depicted persons. Additionally, the ability of
these techniques to extrapolate from existing images can engender potential
privacy concerns, by enabling the depiction of individuals in unauthorized or
invasive scenarios. While the primary aim of this technology serves to foster
positive applications, such as enhancing connections through mixed-reality
video conferencing or the development of virtual human guides, we cannot
overlook its potential for misuse. The eradication of technologies akin to
deep-fakes (Nguyen et al., 2022) may be beyond our reach, yet we aspire that
our contributions will augment the understanding of human avatar synthesis,
thereby informing the development of countermeasures against any unscrupulous
exploitation of this technology.