# DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars

Tobias Kirschstein Simon Giebenhain Matthias Nießner
Technical University of Munich

###### Abstract

DiffusionAvatars synthesizes a high-fidelity 3D head avatar of a person,
offering intuitive control over both pose and expression. We propose a
diffusion-based neural renderer that leverages generic 2D priors to produce
compelling images of faces. For coarse guidance of the expression and head
pose, we render a neural parametric head model (NPHM) from the target
viewpoint, which acts as a proxy geometry of the person. Additionally, to
enhance the modeling of intricate facial expressions, we condition
DiffusionAvatars directly on the expression codes obtained from NPHM via
cross-attention. Finally, to synthesize consistent surface details across
different viewpoints and expressions, we rig learnable spatial features to the
head’s surface via TriPlane lookup in NPHM’s canonical space.
We train DiffusionAvatars on RGB videos and corresponding tracked NPHM meshes
of a person and test the obtained avatars in both self-reenactment and
animation scenarios. Our experiments demonstrate that DiffusionAvatars
generates temporally consistent and visually appealing videos for novel poses
and expressions of a person, outperforming existing approaches.

(a) Avatar creation |  (b) Avatar animation
---|---

Figure 1: Given a set of multi-view videos and corresponding tracked meshes,
we build a DiffusionAvatar of a person. Our method translates expressions of a
morphable model into realistic facial appearances of a person while also
providing control over the viewpoint.

##  1 Introduction

A significant problem in computer vision and graphics is the creation of
photorealistic animatable avatars. Ideally, these avatars permit consistent
video renderings while providing free control over desired expression, pose,
and viewpoint. A digital 3D head avatar may then be used in various scenarios
such as VR/AR applications, immersive teleconferencing, gaming, movie
animation, language learning, or as a virtual assistant.

Unfortunately, creating a 3D avatar from which one can render photorealistic
images from arbitrary viewpoints is challenging. The task becomes even more
difficult when the avatar has to simultaneously offer precise control over
poses and expressions, for instance given by a driving sequence from a
different video or controlled manually. Technically, this leads to a 4D
photometric reconstruction problem which is typically underconstrained. This
presents animation systems with a challenging task that traditionally involves
extensive efforts by 3D artists.

Many existing methods have tackled the challenge of facial image synthesis via
2D neural networks [29, 69, 24, 54, 66, 48]. In particular, Diffusion Models
have recently been demonstrated to produce visually appealing imagery [21, 55,
57, 84, 81]. 2D models are great at generating photo-realistic images but
generally do not offer the degree of control and temporal consistency needed
for a high-quality 3D head avatar. On the other side, methods that reconstruct
an animatable 3D representation of the head [90, 19, 88, 87] provide much
better consistency but in many cases the resulting renderings do not have the
same photo-realism as 2D models.

To this end, we introduce DiffusionAvatars, a novel approach for 3D head
avatar creation, by combining the strong image synthesis capabilities of 2D
diffusion models with the view consistency of a detailed 3D head
representation. As a controllable 3D representation, we leverage the recent
neural parametric head model (NPHM) [17] due to its detailed prediction of
human head geometry, which can give us better geometric cues than established
mesh-based 3DMMs [35, 51]. Inspired by neural textures [68], we further base
face synthesis on person-specific learnable features rigged to the model’s
surface via NPHM’s canonical space. Such surface-mapped features can
effectively compensate for imperfect geometry and improve consistency across
views by giving the neural renderer cues about corresponding surface points.
To avoid learning a neural renderer from scratch, we base our architecture on
a pre-trained latent diffusion model (LDM) [55] and convert it into an image-
to-image translation model by following the ControlNet [84] paradigm to
condition on the rasterized images of NPHM meshes. That way, we not only
inherit a powerful image synthesis backbone but also benefit from the learned
facial prior obtained from large 2D image datasets, which facilitates
generalization to unseen expressions. Finally, we insert cross-attention
blocks into the pre-trained LDM to condition the diffusion-based neural
renderer on the expression codes of NPHM directly. This allows our model to
produce more expressive renderings. Intuitively, the direct conditioning helps
the diffusion model to distinguish subtle expression details while the
rasterized NPHM renderings encode the viewpoint and the overall shape of the
head. In summary, our contributions are as follows:

  * •

We present DiffusionAvatars, a diffusion-based neural renderer that leverages
ControlNet to create animatable 3D head avatars.

  * •

We design a method for rigging learnable spatial features to the surface of
the underlying NPHM via TriPlanes.

  * •

We propose direct expression conditioning via cross-attention to transfer
detailed expressions from NPHM to the synthesized 3D head avatar.

##  2 Related Work

###  2.1 3D Face Animation

It is natural to approach the task of 3D Face Animation with an actual 3D
representation. In the seminal work by [4], a 3D morphable model (3DMM) was
introduced, which paved the way towards reconstructing animatable 3D head
geometry from images or depth observations. More recent 3DMMs [35, 51, 70, 77,
7] still build upon this paradigm and have since been the drivers of face
animation oriented applications with most methods leveraging a 3DMM to gain
control over the expression and pose of the avatar. For example, Neural Head
Avatars [19] and ROME [30] finetune the mesh topology of FLAME [35] to obtain
more faithful and realistic mesh-based avatars. NeRFace [15], INSTA [90] and
RigNeRF [1] build a radiance field that is controlled by a 3DMM. Other 3D
representations such as points [88] or signed distance functions [87, 37] have
also been explored for avatar creation.
In our work, we do not aim to learn a complete 3D representation. Instead, we
utilize the 3D prior of the recent neural parametric head model (NPHM) [17]
with a 2D rendering network for high-quality image synthesis.

###  2.2 Face Synthesis in 2D

Ever since the impactful Pix2Pix [89] work, it has been a popular approach to
obtain control over synthesized images via 2D networks. In the face synthesis
domain, methods based on Generative Adversarial Networks (GANs) [18] have
received a lot of attention [29, 69, 24, 54, 66, 48]. To obtain better control
over head pose and camera viewpoint, a common technique is to combine powerful
2D image synthesis networks with a controllable 3D head proxy [8, 63, 65, 2,
45, 10, 75, 68, 36, 67]. Another line of work aims at directly generating
consistent videos by video-to-video translation [31, 72] or even video
stylization [78, 26, 14]. Most related to our method is Deferred Neural
Rendering [68] which allows facial animation by learning a neural renderer in
order to decode neural textures rigged to Basel Face Model [51]. While
2D-based approaches produce visually appealing frames, they often struggle
with synthesizing consistent images across both views and time or provide only
limited viewpoint control.
Our method aims to improve consistency by utilizing a more powerful implicit
3D representation as geometric proxy and provides better expression
generalization by utilizing a diffusion model as a synthesis backbone.

###  2.3 Controllable 2D Diffusion

Recently, Diffusion models emerged as powerful 2D image generation models.
While they demonstrate superior capabilities in generating high-fidelity and
diverse 2D content from text prompts [21, 55, 57], it has been a key challenge
to leverage their power for other tasks. Several works extend 2D diffusion
models to text-guided video generation by employing a multi-stage approach [5,
22, 73, 60], generating frames autoregressively [23] or stylizing an input
video [74, 16, 79, 76]. In the human domain, diffusion models have already
been applied to face editing [25], animation [81, 62] and bodies [64]. Other
works, such as ControlNet [84], IPAdapter [80] or T2I-Adapter [46], allow
fine-tuning a pre-trained diffusion model on additional controls such as
landmarks or depth cues, opening a wide space of possible applications for 2D
face editing. Most similar to our method, DiffusionRig [11] proposes a
diffusion model conditioned on rasterized FLAME meshes to provide 3D animation
control over an avatar created from a personal photo album. While the
synthesized images or videos of these approaches already show great
controllability and visual quality, they typically lack consistency due to the
absence of an underlying accurate 3D representation.

###  2.4 Diffusion for 3D Faces

Following a different approach, several works explore diffusion for generic 3D
generation [52, 40, 47]. This has also been extended to 3D head generation
[71, 20, 83], text-guided 3D head editing [49] or 3D body generation [82].
Some works also generate animatable 3D asset animatable for heads [3] and
bodies [6, 53, 27] by rigging the 3D asset to a parameterized FLAME [35] or
SMPL [41] mesh. Such generated 3D avatars are view consistent by design but
lack the visual quality of 2D-based diffusion models. Furthermore, animation
control is limited by the underlying mesh template. In our work, we employ a
more accurate, implicit 3D morphable head model for better expression control
and follow ControlNet [84] and IPAdapter [80] to build a diffusion-based
neural renderer for high-quality image synthesis.

Figure 2: Method overview: We decode an NPHM expression code $z_{exp}$ in two
ways to obtain a realistic image: We first extract an NPHM mesh and rasterize
it from the desired viewpoint in (a), giving us canonical coordinates, depths,
and normal renderings for the head mesh. In (b), the canonical coordinates
$x_{can}$ are used to look up spatial features in a TriPlanes structure,
rigging the features onto the mesh surface. Together with the rasterizer
output, these mapped features form the input for the ControlNet part of
DiffusionAvatar. The second route for the expression code goes through a
linear layer depicted in (c). It yields expression tokens that are
subsequently used in a newly added cross-attention layer inside the pre-
trained latent diffusion model. Intuitively, the rasterized inputs should
encode pose, shape and rough expression while the direct expression
conditioning hints at more detailed facial expressions. The final image is
synthesized in (d) by iteratively denoising Gaussian noise using the original
DDPM denoising schedule [21].

##  3 Method

Our goal is to create a temporally consistent 3D head avatar of a person with
explicit control over viewpoint and expression. We approach this task by
designing a diffusion-based neural renderer that decodes learnable features
rigged to the surface of an implicitly defined proxy geometry. In analogy to
Deferred Neural Rendering [68], we dub this approach Deferred Diffusion. We
begin by introducing the foundations of diffusion models (Sec. 3.1) and NPHM
(Sec. 3.2) that we use as proxy geometry before going into detail about our
method: In the rasterization (Sec. 3.3) and surface feature mapping (Sec. 3.4)
stage, we generate the input images for our diffusion-based neural renderer.
These renderings encode the desired viewpoint, head shape, and global
expression. For more fine-grained expression control, we add NPHM’s expression
codes as an additional input to the diffusion model (Sec. 3.5). Finally, our
pipeline leverages a pre-trained diffusion model’s image synthesis and
generalization capabilities for better quality.

###  3.1 Diffusion

We build our work upon latent diffusion models (LDM) [55], which operate in
the the latent space of an autoencoder. Formally, LDMs employ an encoder
$\mathcal{E}$ to map RGB images into a lower-dimensional latent space,
facilitating generative tasks [12]. In the following, any mention of an image
$x_{0}$ always refers to an RGB image $I$ that was mapped into the
autoencoder’s latent space via $x_{0}=\mathcal{E}(I)$ to obtain a latent
image.

In diffusion, the fixed forward process iteratively adds noise to the latent
image:

$\displaystyle q(x_{\tau}|x_{\tau-1})$
$\displaystyle=\mathcal{N}\left(x_{\tau};\sqrt{\alpha_{\tau}}x_{\tau-1},(1-\alpha_{\tau})\mathbf{I}\right)$
(1)

where $\tau=1...N$ indicates the denoising step and the forward process
variances $\alpha_{\tau}$ define the noise scheduling [21]. In practice, noisy
samples $x_{\tau}$ are obtained with the standard Gaussian reparameterization:

$x_{\tau}=\sqrt{\bar{\alpha}_{\tau}}x_{0}+\sqrt{1-\bar{\alpha}_{\tau}}\epsilon\qquad\epsilon\sim\mathcal{N}\left(0,\mathbf{I}\right)$
(2)

Typically, diffusion models are trained to predict the original noise
$\epsilon$ given the noisy sample $x_{\tau}$. We empirically find a different
method, known as the $v$-parameterization [57], to work better in our
scenario, where $v$ is defined as:

$\displaystyle v$
$\displaystyle=\sqrt{\bar{\alpha}_{\tau}}\epsilon-\sqrt{1-\bar{\alpha}_{\tau}}x_{0}$
(3)

This has two advantages. First, $v$-prediction ensures that the loss can
always guide the model to learn something meaningful, even when the input
already contains a lot of noise, leading to faster convergence [22]. Second,
it allows us to train the model also on pure noise inputs, which during
inference is the most challenging denoising step. To this end, we rescale the
noise schedule $\alpha$ to ensure zero signal-to-noise ratio inputs during
training as proposed by [39].

###  3.2 NPHM

NPHM [17] is a morphable head model that generates a signed distance field
(SDF) of a human head given an identity code $z_{id}$ and an expression code
$z_{exp}$. We employ COLMAP [59, 58] to obtain pointclouds
$\\{\mathcal{P}_{t}\\}$ for each timestep $t$ of a multi-view video sequence
of the person. Subsequently, we fit NPHM to each pointcloud $\mathcal{P}_{t}$
to obtain $z_{id}$ and $z^{t}_{exp}$. We use a variant of NPHM that uses a
backward deformation field instead of the originally proposed forward
deformation field as it simplifies the fitting process. Formally, we obtain
the NPHM fitting as follows:

$z_{id},z^{t}_{exp}=\operatorname*{arg\,min}_{z_{id},z_{exp}}\sum_{x\in\mathcal{P}_{t}}\left|\mathcal{F}_{id}(\mathcal{F}_{exp}(x))\right|$
(4)

where $\mathcal{F}_{id}$ is NPHM’s identity network implemented as a signed
distance field, and $\mathcal{F}_{exp}$ is the backward deformation network.
Note that we drop the dependency of $\mathcal{F}_{id}$ and $\mathcal{F}_{exp}$
on their respective latent codes for simplicity. The fitted SDF representation
can then be translated into a mesh $M_{t}$ via marching cubes [42].
Furthermore, for each vertex $x\in M_{t}$ of the extracted mesh, we can obtain
its canonical coordinates $x_{can}\in\mathbb{R}^{3+2}$ via the backward
deformation field:

$x_{can}=\mathcal{F}_{exp}(x)$ (5)

where the first 3 coordinates of $x_{can}$ represent the usual spatial
dimensions while the remaining 2 coordinates are ambient dimensions that can
help resolve topological issues in mouth and eye regions [50]. We utilize the
meshes ${M_{t}}$ in the subsequent rasterization stage, whereas the canonical
coordinates $x_{can}$ are necessary for the spatial feature lookup.

###  3.3 Rasterization

We employ nvdiffrast [34] to generate the actual input images for the
diffusion model. Let $M_{t}$ be a facial mesh of a person at timestep $t$ and
$\pi_{t}$ a camera pose. We then obtain a set of renderings
$R^{t}\in\mathbb{R}^{H\times W\times C}$ for this timestep using the
rasterizer $\mathcal{R}$:

$\displaystyle R^{t}=\mathcal{R}\left(M_{t},\pi_{t}\right)$ (6)

In our case, the channels of $R^{t}$ are normals, depths, and a canonical
coordinate rendering $R^{t}_{can}\in\mathbb{R}^{H\times W\times(3+2)}$ of the
NPHM mesh.

###  3.4 TriPlane Feature Mapping

It has been shown that tying learnable features to a surface helps neural
renderers to synthesize more detailed images, a technique known as neural
textures [68]. Since NPHM lacks a consistent UV space due to its implicit
nature, we propose a simple extension of neural textures: We tie learnable
features to the surface of the extracted mesh $M_{t}$ by querying a spatial
structure with the canonical coordinates $x_{can}$. We use TriPlanes [8] for
the 3 spatial dimensions and a regular 2D feature map for the ambient
dimensions:

$\displaystyle R^{t}_{\text{feat}}$
$\displaystyle=\textsc{TriPlane}\left(R^{t}_{can,0\text{-}3}\right)$ (7)
$\displaystyle R^{t}_{\text{feat\\_amb}}$
$\displaystyle=\textsc{AmbientMap}\left(R^{t}_{can,3\text{-}5}\right)$ (8)

These learnable feature maps form the 2D input for our diffusion-based neural
renderer, together with the other buffers $R^{t}$ from the rasterizer:

$\displaystyle R^{t}\leftarrow
R^{t},R^{t}_{\text{feat}},R^{t}_{\text{feat\\_amb}}$ (9)

###  3.5 Direct Expression Conditioning

Conditioning on renderings guides the head pose as well as coarse expressions.
However, subtle details may not be easy for the network to decode from
renderings. To facilitate the synthesis of detailed expressions, we add new
cross-attention layers to the U-Net, following IPAdapter [80]. Let $Z$ be the
intermediate feature map computed by an existing cross-attention operation in
the pre-trained LDM: $Z=\textsc{attention}(Q,K,V)$. We then perform direct
expression conditioning by adding another cross-attention layer:

$\displaystyle f^{t}_{exp}$ $\displaystyle=\textsc{exp}(z^{t}_{exp})$ (10)
$\displaystyle Z$ $\displaystyle\leftarrow
Z+\textsc{attention}(Q,W^{k}f^{t}_{exp},W^{v}f^{t}_{exp})$ (11)

where exp is our expression conditioning module that linearly maps
$z^{t}_{exp}$ into a sequence $f^{t}_{exp}\in\mathbb{R}^{4\times d}$ of 4
expression tokens forming the keys and values for cross-attention. In total,
we add 15 cross-attention layers to the LDM.

###  3.6 Deferred Diffusion

The final rendering is obtained by iteratively denoising full noise
$x_{T}\sim\mathcal{N}(0,I)$ with our diffusion-based neural renderer
$\mathcal{D}$ conditioned on the rasterized NPHM meshes $R^{t}$ and expression
tokens $f_{exp}$:

$\displaystyle
x_{\tau-1}\sim\mathcal{D}(x_{\tau})=\mathcal{S}(x_{\tau},f^{t}_{exp},\mathcal{C}(x_{\tau},R^{t}))$
(12)

where $\mathcal{C}$ is the ControlNet architecture and $\mathcal{S}$ a pre-
trained LDM with our expression conditioning modules inserted. The denoised
prediction $x_{\tau-1}$ is obtained via the sampling procedure of [21].

During training, we minimize the following loss:

$\displaystyle\mathcal{L}=\mathbb{E}_{\epsilon,\tau,x^{t}_{0},R^{t},f^{t}_{exp}}\left[\left\|\mathcal{D}\left(x^{t}_{\tau},R^{t},f^{t}_{exp}\right)-v\right\|_{2}\right]$
(13)

where $\epsilon$ is the sampled noise, $\tau$ is the denoising step and
$x^{t}_{0}=\mathcal{E}(I^{t})$ is the ground truth latent image. We use
$\mathcal{L}$ to optimize (i) the ControlNet $\mathcal{C}$, (ii) the
parameters of expression conditioning $W^{k},W^{v}$, and exp, and (iii) the
spatial feature maps TriPlane and AmbientMap.

NeRFace [15] |  DNR [68] |  DNR + GAN |  DiffusionRig [11] |  Ours |  GT
---|---|---|---|---|---

Figure 3: Qualitative results for self-reenactment. We compare against 3D
methods (NeRFace [15]) and methods employing 2D renderers (Deferred Neural
Rendering [68], DiffusionRig [11]). Note that the slightly gray background for
DiffusionRig is caused by their training scheme using $\epsilon$-prediction
(see Sec. 3.1). Our method consistently produces more expressive facial
performances while simultaneously providing more detailed renderings.

Source Actor |  |  NeRFace [15] |  DNR [68] |  DNR + GAN |  DiffusionRig [11] |  Ours
---|---|---|---|---|---|---

Figure 4: Qualitative results for avatar animation. Our method faithfully
transfers the source actor’s expression and consistently produces compelling
renderings, even for complex performances.

##  4 Experimental Results

###  4.1 Training

For our experiments, we employ the recently released multi-view video dataset
from NeRSemble [33]. The dataset contains 26 different multi-view sequences
per person captured with 16 cameras, featuring different facial expressions,
sentences and emotions. We use BackgroundMattingV2 [38] to remove the
background and a segmentation network to remove the torso [86]. We fit NPHM on
the video sequences of 5 different persons to create pairings of multi-view
videos and tracked meshes. In total, we use about 3300 timesteps per person.
We train one DiffusionAvatar for each person on these pairings with Adam [32]
using a learning rate of $1e-4$ for ControlNet and the expression conditioning
layers, and a learning rate of $1e-2$ for the TriPlane feature lookups. We use
a batch size of $B=8$ and spatial resolutions $H=W=512$. We employ random
cropping and resizing of the target images to avoid overfitting. The size of
our spatial feature maps is $512\times 512\times 16$ for each TriPlane and the
ambient map. In total, this yields 73 channels for the input renderings
$R_{t}$ fed to our diffusion-based renderer. The pre-trained LDM is a Stable
Diffusion v2.1 network [55]. Training takes roughly two days on a single RTX
A6000 for 100k gradient descent steps.

###  4.2 Experiment Setup

#### Tasks.

We measure the quality of a 3D avatar in terms of image quality, 3D
consistency and expression synthesis. To this end, we conduct experiments on
two tasks: Self-reenactment and avatar animation. For self-reenactment, we
hold out 2 complete sequences and the frontal camera of every sequence during
training to measure both image quality and view consistency. During testing,
we provide the holdout tracked meshes of the same person and render from the
novel view. For avatar animation, we first generate NPHM meshes using the same
identity code $z_{id}$ as the avatar and a sequence of expression codes
$\\{z^{t}_{exp}\\}$ from a different person. We then render a video of the
avatar from a novel view. To evaluate the realism of this expression transfer,
we conduct a user study where we show the animated avatars to participants.

#### Metrics.

We employ two paired-image metrics to measure the quality of individual
generated frames: Peak Signal-to-Noise Ratio (PSNR) and Learned Perceptual
Image Patch Similarity (LPIPS) [85]. Furthermore, to assess temporal
consistency, we employ the perceptual video metric JOD [43] to measure visual
similarity of a rendered video to its ground truth counterpart. Finally, we
make use of two face-specific metrics: Average Keypoint Distance (AKD)
measured in pixels with keypoints estimated from PIPNet [28], and cosine
similarity (CSIM) of identity embeddings based on ArcFace [9].

#### Baselines.

We compare our method against the following state-of-the-art 3D head avatar
creation methods:

NeRFace [15]. A NeRF-based method that reconstructs a radiance field
conditioned on expression codes. The original implementation is for monocular
videos and uses FLAME. We extend it to our multi-view training scenario and
provide NPHM expression codes for better comparison.

Deferred Neural Rendering (DNR) [68]. A screen-space decoder built on top of
Pix2Pix [89] that uses learnable neural textures rigged to a tracked Basel
Face Model [51]. We provide our tracked NPHM meshes as input for better
comparison. Additionally, we compare against a version of DNR that also uses
an adversarial loss for sharper images that we call DNR+GAN in the following.

DiffusionRig [11]. A recent diffusion-based method that uses albedo, normal,
and shaded color renderings of FLAME meshes tracked by DECA [13] as input to
control pose and expression. While their method was originally proposed to be
fine-tuned on a small personal photo album of 20 images, we fine-tune their
pre-trained model on all training frames of our dataset.

###  4.3 Self-Reenactment

In Tab. 1, we compare DiffusionAvatars to other state-of-the-art approaches on
the self-reenactment task. Results are averaged over avatars generated from
five different persons. Our method shows the overall best performance with the
biggest improvement in perceptual and face-specific metrics. The results
indicate that DiffusionAvatars generates sharper images (LPIPS) that are more
temporally consistent (JOD). Furthermore, our method better maintains the
person’s identity when rendered from a novel view (CSIM) than the baselines.
Finally, facial features such as the position of mouth corners and eyebrows
synthesized by our method match the ground truth expressions more closely
(AKD).
We further provide a qualitative comparison in Fig. 3. The baselines struggle
especially in facial regions that are poorly explained by the NPHM mesh, such
as the mouth interior. In contrast, our method can plausibly fill these
regions, providing a consistently realistic appearance.

Method | Self Reenactment |  | User Study
---|---|---|---
PSNR$\uparrow$ | LPIPS$\downarrow$ | JOD$\uparrow$ | AKD$\downarrow$ | CSIM$\uparrow$ |  | VQ$\uparrow$ | DF$\uparrow$
NeRFace [15] | 22.5 | 0.299 | 6.65 | 5.10 | 0.761 |  | 2.19 | 2.35
DiffusionRig [11] | 19.6 | 0.218 | 6.42 | 2.66 | 0.886 |  | 2.47 | 3.52
DNR [68] | 24.4 | 0.226 | 7.31 | 2.18 | 0.898 |  | 2.87 | 3.94
DNR+GAN [68] | 22.8 | 0.115 | 7.06 | 2.32 | 0.859 |  | 3.06 | 3.97
Ours | 24.5 | 0.088 | 7.53 | 1.86 | 0.912 |  | 4.02 | 4.14

Table 1: Quantitative Comparison: We report metrics for a self-reenactment
scenario on unseen expressions and unseen views. For our user study on avatar
animation, we collect 735 responses from 49 participants to measure Visual
Quality (VQ) and Driving Fidelity (DF) on a scale of 1-5.

###  4.4 Avatar Animation

Due to the task’s inherent lack of any ground truth, we perform a user study
to validate avatar animation. We collect 735 responses from 49 participants on
two questions: (i) Visual Quality (VQ): What is the overall quality of the
avatar animation? (ii) Driving Fidelity (DF): How closely does the animated
avatar match the expressions of the driving sequence? We present two videos
side-by-side: One sequence of a source actor performing various expressions,
and a video of another person’s avatar with the expressions transferred to it.
The results of our user study can be seen in Tab. 1. Our method achieves the
highest scores for both visual quality and similarity to the driving sequence.
The difference is particularly pronounced for the visual quality score. We
attribute this to the fact that users heavily penalize any visual artifacts
immediately obvious in video renderings.
These perceptual differences in the task of avatar animation are also apparent
in the qualitative comparison shown in Fig. 4.

###  4.5 Ablations

In the following, we examine the design choices of our method. All ablation
experiments are conducted on 3 avatars on the self-reenactment task, and the
averaged results are summarized in Tab. 2.

| PSNR$\uparrow$ | LPIPS$\downarrow$ | JOD$\uparrow$ | AKD$\downarrow$ | CSIM$\uparrow$
---|---|---|---|---|---
w/o diffusion | 25.1 | 0.133 | 7.69 | 1.96 | 0.892
FLAME | 24.2 | 0.083 | 7.36 | 2.46 | 0.900
w/o exp. cond. | 24.5 | 0.081 | 7.65 | 1.98 | 0.911
w/o LDM prior | 24.5 | 0.078 | 7.67 | 1.89 | 0.913
w/o spatial features | 24.9 | 0.078 | 7.77 | 1.87 | 0.918
spherical UV | 24.9 | 0.075 | 7.77 | 1.95 | 0.920
Ours | 25.3 | 0.074 | 7.85 | 1.91 | 0.918
Table 2: Ablation of architectural choices. Our full method utilizing TriPlane
feature mapping performs overall the best. The biggest quantitative
improvements stem from using NPHM instead of FLAME, employing direct
expression conditioning, and using a pre-trained LDM.

#### Effect of Diffusion.

We study whether diffusion models are necessary for our task. To this end, we
train our architecture without the pre-trained LDM to directly predict the
output in a single forward pass, similar to Deferred Neural Rendering. The
results indicate that such a method performs notably worse, especially in the
LPIPS metric, indicating that diffusion is important to obtain sharp
renderings.

#### Effect of NPHM vs FLAME.

We compare DiffusionAvatars with a version trained on renderings of a fitted
FLAME instead of NPHM. Since NPHM models facial geometry as an SDF, it
possesses a much higher representational power than FLAME. As a result, the
renderings are much closer to the actual desired appearance of the head. This
is also reflected by the results of our ablation study, where using FLAME
gives consistently worse performance. This is especially true for the AKD
metric, indicating that our diffusion-based neural renderer cannot reenact
certain complex expressions based on FLAME alone.

#### Effect of Expression Conditioning.

The overall performance worsens when direct expression conditioning is turned
off. We attribute this to the fact that expression conditioning helps to
distinguish subtle expression details that can hardly be inferred from the
NPHM mesh alone. It also helps DiffusionAvatars to synthesize expressions not
captured by NPHM, such as tongue movement, by enabling the network to
correlate such appearances with certain expression code combinations.

#### Effect of 2D LDM Prior.

We train our pipeline from scratch without resorting to a pre-trained LDM. In
this case, we remove the ControlNet module and condition the U-Net directly on
the NPHM renderings. This architecture, trained from scratch, performs
competitively but slightly worse. We see the biggest advantage of pre-training
in faster convergence and better coherence of the generated images.

#### Effect of Spatial Features.

We compare three spatial feature mapping approaches: (i) Our Triplanes, (ii) a
simpler approach based on spherical feature mapping, and (iii) not using any
spatial features at all. For spherical feature mapping, we create an
approximate UV map for each NPHM mesh by projecting the 3D canonical
coordinates for each vertex onto a sphere centered inside the head. Tab. 2
shows that using TriPlanes boosts most metrics, whereas the simpler spherical
mapping brings almost no benefit.

##  5 Limitations

DiffusionAvatars can create photo-realistic 3D head avatars with pose and
expression control. More challenges must be solved, however, before an avatar
can be used in production. For example, to put an avatar into a realistic
environment, it is necessary to have control over the lighting properties.
Currently, DiffusionAvatars bakes the lighting into the generated images.
Since we employ a detailed 3D head geometry underneath, one could imagine
directly modeling shadow effects as a composite of the synthesized images.
Further, our current architecture is not (yet) amenable to real-time
applications due to the comparatively compute-intensive denoising loop. Here,
recent advances in the distillation of diffusion models may provide a remedy
[61, 44].

##  6 Conclusion

This paper presents a novel method DiffusionAvatars, which creates a photo-
realistic 3D head avatar from multi-view videos using a tracked parametric
head model of a person. DiffusionAvatars can be animated by a source actor or
directly via NPHM. We use rendered NPHM meshes with rigged spatial features as
input to a diffusion-based neural renderer. We leverage the image synthesis
capabilities of a pre-trained latent diffusion model with ControlNet,
facilitating generalization to unseen expressions. While this pipeline is
already capable of synthesizing appealing images, we show that additionally
conditioning the neural renderer on NPHM’s expression codes further improves
the model’s ability to generate complex facial expressions. Our experiments
suggest that the proposed architecture can effectively synthesize high-quality
renderings and animate the 3D avatar with high fidelity. We believe our
approach demonstrates an exciting application for high-fidelity 3D avatar
creation.

### Acknowledgements

This work was supported by the ERC Starting Grant Scan2CAD (804724) and the
German Research Foundation (DFG) Research Unit “Learning and Simulation in
Visual Computing”. We would also like to thank Karla Weighart for proof-
reading and Angela Dai for the video voice-over.

## References

  * Athar et al. [2022] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu.  Rignerf: Fully controllable neural 3d portraits.  In _Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition_ , pages 20364–20373, 2022.
  * Bergman et al. [2022] Alexander Bergman, Petr Kellnhofer, Wang Yifan, Eric Chan, David Lindell, and Gordon Wetzstein.  Generative neural articulated radiance fields.  _Advances in Neural Information Processing Systems_ , 35:19900–19916, 2022.
  * Bergman et al. [2023] Alexander W Bergman, Wang Yifan, and Gordon Wetzstein.  Articulated 3d head avatar generation using text-to-image diffusion models.  _arXiv preprint arXiv:2307.04859_ , 2023.
  * Blanz and Vetter [2023] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3d faces.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pages 157–164. 2023.
  * Blattmann et al. [2023] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22563–22575, 2023.
  * Cao et al. [2023] Yukang Cao, Yan-Pei Cao, Kai Han, Ying Shan, and Kwan-Yee K Wong.  Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models.  _arXiv preprint arXiv:2304.00916_ , 2023.
  * Chai et al. [2023] Zenghao Chai, Tianke Zhang, Tianyu He, Xu Tan, Tadas Baltrusaitis, HsiangTao Wu, Runnan Li, Sheng Zhao, Chun Yuan, and Jiang Bian.  Hiface: High-fidelity 3d face reconstruction by learning static and dynamic details.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9087–9098, 2023.
  * Chan et al. [2022] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16123–16133, 2022.
  * Deng et al. [2019] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4690–4699, 2019.
  * Deng et al. [2020] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong.  Disentangled and controllable face image generation via 3d imitative-contrastive learning.  In _IEEE Computer Vision and Pattern Recognition_ , 2020.
  * Ding et al. [2023] Zheng Ding, Xuaner Zhang, Zhihao Xia, Lars Jebe, Zhuowen Tu, and Xiuming Zhang.  Diffusionrig: Learning personalized priors for facial appearance editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12736–12746, 2023.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Feng et al. [2021] Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart.  Learning an animatable detailed 3D face model from in-the-wild images.  2021\.
  * Fišer et al. [2017] Jakub Fišer, Ondřej Jamriška, David Simons, Eli Shechtman, Jingwan Lu, Paul Asente, Michal Lukáč, and Daniel Sỳkora.  Example-based synthesis of stylized facial animations.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–11, 2017.
  * Gafni et al. [2021] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8649–8658, 2021.
  * Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  _arXiv preprint arXiv:2307.10373_ , 2023.
  * Giebenhain et al. [2023] Simon Giebenhain, Tobias Kirschstein, Markos Georgopoulos, Martin Rünz, Lourdes Agapito, and Matthias Nießner.  Learning neural parametric head models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21003–21012, 2023.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Grassal et al. [2022] Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies.  Neural head avatars from monocular rgb videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18653–18664, 2022.
  * Gu et al. [2023] Jiatao Gu, Qingzhe Gao, Shuangfei Zhai, Baoquan Chen, Lingjie Liu, and Josh Susskind.  Learning controllable 3d diffusion models from single-view images.  _arXiv preprint arXiv:2304.06700_ , 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022a.
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  _arXiv:2204.03458_ , 2022b.
  * Hong et al. [2022] Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu.  Depth-aware generative adversarial network for talking head video generation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3397–3406, 2022.
  * Huang et al. [2023] Ziqi Huang, Kelvin C.K. Chan, Yuming Jiang, and Ziwei Liu.  Collaborative diffusion for multi-modal face generation and editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023.
  * Jamriška et al. [2019] Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sỳkora.  Stylizing video by example.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–11, 2019.
  * Jiang et al. [2023] Ruixiang Jiang, Can Wang, Jingbo Zhang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control.  _arXiv preprint arXiv:2303.17606_ , 2023.
  * Jin et al. [2021] Haibo Jin, Shengcai Liao, and Ling Shao.  Pixel-in-pixel net: Towards efficient facial landmark detection in the wild.  _International Journal of Computer Vision_ , 2021.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Khakhulin et al. [2022] Taras Khakhulin, Vanessa Sklyarova, Victor Lempitsky, and Egor Zakharov.  Realistic one-shot mesh-based head avatars.  In _European Conference on Computer Vision_ , pages 345–362. Springer, 2022.
  * Kim et al. [2018] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Nießner, Patrick Pérez, Christian Richardt, Michael Zollöfer, and Christian Theobalt.  Deep video portraits.  _ACM Transactions on Graphics (TOG)_ , 37(4):163, 2018.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirschstein et al. [2023] Tobias Kirschstein, Shenhan Qian, Simon Giebenhain, Tim Walter, and Matthias Nießner.  Nersemble: Multi-view radiance field reconstruction of human heads.  _ACM Trans. Graph._ , 42(4), 2023.
  * Laine et al. [2020] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila.  Modular primitives for high-performance differentiable rendering.  _ACM Transactions on Graphics_ , 39(6), 2020.
  * Li et al. [2017] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero.  Learning a model of facial shape and expression from 4D scans.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6):194:1–194:17, 2017.
  * Lin et al. [2022] C.Z. Lin, D.B. Lindell, E.R. Chan, and G. Wetzstein.  3d gan inversion for controllable portrait image animation.  In _ECCV Workshop on Learning to Generate 3D Shapes and Scenes_ , 2022\.
  * Lin et al. [2023a] Connor Z. Lin, Koki Nagano, Jan Kautz, Eric R. Chan, Umar Iqbal, Leonidas Guibas, Gordon Wetzstein, and Sameh Khamis.  Single-shot implicit morphable faces with consistent texture parameterization.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023a.
  * Lin et al. [2021] Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman.  Real-time high-resolution background matting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8762–8771, 2021.
  * Lin et al. [2023b] Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang.  Common diffusion noise schedules and sample steps are flawed.  _arXiv preprint arXiv:2305.08891_ , 2023b.
  * Liu et al. [2023] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object, 2023.
  * Loper et al. [2023] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pages 851–866. 2023.
  * Lorensen and Cline [1998] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_ , pages 347–353. 1998.
  * Mantiuk et al. [2021] Rafał K. Mantiuk, Gyorgy Denes, Alexandre Chapiro, Anton Kaplanyan, Gizem Rufo, Romain Bachy, Trisha Lian, and Anjul Patney.  Fovvideovdp: A visible difference predictor for wide field-of-view video.  _ACM Trans. Graph._ , 40(4), 2021.
  * Meng et al. [2023] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Mensah et al. [2023] Dann Mensah, Nam Hee Kim, Miika Aittala, Samuli Laine, and Jaakko Lehtinen.  A hybrid generator architecture for controllable face synthesis.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–10, 2023\.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Müller et al. [2023] Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner.  Diffrf: Rendering-guided 3d radiance field diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4328–4338, 2023.
  * Nitzan et al. [2022] Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or.  Mystyle: A personalized generative prior.  _arXiv preprint arXiv:2203.17272_ , 2022.
  * Pan et al. [2023] Mohit Mendiratta Pan, Mohamed Elgharib, Kartik Teotia, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, Christian Theobalt, et al.  Avatarstudio: Text-driven editing of 3d dynamic human head avatars.  _arXiv preprint arXiv:2306.00547_ , 2023.
  * Park et al. [2021] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  _arXiv preprint arXiv:2106.13228_ , 2021.
  * Paysan et al. [2009] Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.  A 3d face model for pose and illumination invariant face recognition.  In _2009 sixth IEEE international conference on advanced video and signal based surveillance_ , pages 296–301. Ieee, 2009.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv_ , 2022.
  * Qin et al. [2023] Bosheng Qin, Wentao Ye, Qifan Yu, Siliang Tang, and Yueting Zhuang.  Dancing avatar: Pose and text-guided human motion videos synthesis with image diffusion model.  _arXiv preprint arXiv:2308.07749_ , 2023.
  * Richardson et al. [2021] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Rössler et al. [2018] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  Faceforensics: A large-scale video dataset for forgery detection in human faces.  _arXiv preprint arXiv:1803.09179_ , 2018.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Schönberger and Frahm [2016] Johannes Lutz Schönberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Schönberger et al. [2016] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In _European Conference on Computer Vision (ECCV)_ , 2016.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  2023\.
  * Stypułkowski et al. [2023] Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zieba, Stavros Petridis, and Maja Pantic.  Diffused heads: Diffusion models beat gans on talking-face generation.  _arXiv preprint arXiv:2301.03396_ , 2023.
  * Sun et al. [2023] Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu.  Next3d: Generative neural texture rasterization for 3d-aware head avatars.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20991–21002, 2023.
  * Svitov et al. [2023] David Svitov, Dmitrii Gudkov, Renat Bashirov, and Victor Lempitsky.  Dinar: Diffusion inpainting of neural textures for one-shot human avatars.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7062–7072, 2023.
  * Tang et al. [2023] Junshu Tang, Bo Zhang, Binxin Yang, Ting Zhang, Dong Chen, Lizhuang Ma, and Fang Wen.  3dfaceshop: Explicitly controllable 3d-aware portrait generation.  _IEEE Transactions on Visualization and Computer Graphics_ , 2023\.
  * Tewari et al. [2020a] Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt.  Pie: Portrait image embedding for semantic control.  2020a.
  * Tewari et al. [2020b] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zöllhofer, and Christian Theobalt.  Stylerig: Rigging stylegan for 3d control over portrait images, cvpr 2020\.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2020b.
  * Thies et al. [2019] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  _ACM Transactions on Graphics 2019 (TOG)_ , 2019.
  * Tov et al. [2021] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  _ACM Transactions on Graphics (TOG)_ , 40(4):1–14, 2021.
  * Wang et al. [2022] Lizhen Wang, Zhiyua Chen, Tao Yu, Chenguang Ma, Liang Li, and Yebin Liu.  Faceverse: a fine-grained and detail-controllable 3d face morphable model from a hybrid dataset.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR2022)_ , 2022.
  * Wang et al. [2023a] Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4563–4573, 2023a.
  * Wang et al. [2021] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Wang et al. [2023b] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al.  Lavie: High-quality video generation with cascaded latent diffusion models.  _arXiv preprint arXiv:2309.15103_ , 2023b.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Wu et al. [2022] Yue Wu, Yu Deng, Jiaolong Yang, Fangyun Wei, Chen Qifeng, and Xin Tong.  Anifacegan: Animatable 3d-aware face image generation for video avatars.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Yan et al. [2023] Hanshu Yan, Jun Hao Liew, Long Mai, Shanchuan Lin, and Jiashi Feng.  Magicprop: Diffusion-based video editing via motion-aware appearance propagation.  _arXiv preprint arXiv:2309.00908_ , 2023.
  * Yang et al. [2020] Haotian Yang, Hao Zhu, Yanru Wang, Mingkai Huang, Qiu Shen, Ruigang Yang, and Xun Cao.  Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yang et al. [2022] Shuai Yang, Liming Jiang, Ziwei Liu, and Chen Change Loy.  Vtoonify: Controllable high-resolution portrait video style transfer.  _ACM Transactions on Graphics (TOG)_ , 41(6):1–15, 2022.
  * Yang et al. [2023] Shuai Yang, Yifan Zhou, Ziwei Liu, , and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  In _ACM SIGGRAPH Asia Conference Proceedings_ , 2023.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arXiv:2308.06721_ , 2023.
  * Zeng et al. [2023a] Bohan Zeng, Xuhui Liu, Sicheng Gao, Boyu Liu, Hong Li, Jianzhuang Liu, and Baochang Zhang.  Face animation with an attribute-guided diffusion model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 628–637, 2023a.
  * Zeng et al. [2023b] Yifei Zeng, Yuanxun Lu, Xinya Ji, Yao Yao, Hao Zhu, and Xun Cao.  Avatarbooth: High-quality and customizable 3d human avatar generation.  _arXiv preprint arXiv:2306.09864_ , 2023b.
  * Zhang et al. [2023a] Chi Zhang, Yiwen Chen, Yijun Fu, Zhenglin Zhou, Gang Yu, Billzb Wang, Bin Fu, Tao Chen, Guosheng Lin, and Chunhua Shen.  Styleavatar3d: Leveraging image-text diffusion models for high-fidelity 3d avatar generation.  _arXiv preprint arXiv:2305.19012_ , 2023a.
  * Zhang et al. [2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023b.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zheng et al. [2021] Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen.  General facial representation learning in a visual-linguistic manner.  _arXiv preprint arXiv:2112.03109_ , 2021.
  * Zheng et al. [2022] Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges.  Im avatar: Implicit morphable head avatars from videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13545–13555, 2022.
  * Zheng et al. [2023] Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J Black, and Otmar Hilliges.  Pointavatar: Deformable point-based head avatars from videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 21057–21067, 2023.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2223–2232, 2017.
  * Zielonka et al. [2023] Wojciech Zielonka, Timo Bolkart, and Justus Thies.  Instant volumetric head avatars.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4574–4584, 2023.

Appendix

##  Appendix A Novel View Synthesis Results

Method | PSNR$\uparrow$ | LPIPS$\downarrow$ | JOD$\uparrow$ | AKD$\downarrow$ | CSIM$\uparrow$
---|---|---|---|---|---
NeRFace [15] | 28.12 | 0.284 | 7.93 | 2.72 | 0.841
DiffusionRig [11] | 19.73 | 0.231 | 6.02 | 9.71 | 0.917
DNR [68] | 26.90 | 0.213 | 7.91 | 1.58 | 0.933
DNR+GAN [68] | 23.56 | 0.104 | 7.30 | 1.70 | 0.906
Ours | 26.93 | 0.067 | 8.28 | 1.16 | 0.939

Table 3: Quantitative Comparison: We report metrics for novel-view-synthesis.

Tab. 3 contains results for novel view synthesis. We measure the performance
from the unseen frontal view and average over the 24 train sequences for each
of our 5 avatars. DiffusionAvatars scores the best metrics except for PSNR,
which favors the overly blurry renderings of NeRFace. These results show that
our method also provides good 3D control.

##  Appendix B User Study Setup

In Fig. 5, we show the interface of our user study. We ask users to rate
visual quality and driving fidelity by providing them with the following
definitions:

#### Visual Quality.

The overall quality of the generated video: Is there any flickering, visual
artifacts, blurriness, etc., that feel unnatural?

#### Driving Fidelity.

How closely does the generated video follow the desired facial expressions:
Are the displayed expressions the same? Is the emotion displayed by the avatar
recognizable?

During the study, each user is presented with 15 randomly chosen pairs of
driving sequences (left) and avatar animations (right). The pairs were drawn
from 135 different avatar animations generated by the baselines and our
method. In total, we received 735 responses for each question from 49
participants.

Figure 5: Setup of our User Study. We ask participants to evaluate the avatar
animation task in which an avatar is controlled by a sequence of a different
person.

##  Appendix C Qualitative Comparison of Ablations

As an addition to Tab. 2 of the main paper, we provide qualitative comparisons
of three ablations in Fig. 6, Fig. 7 and Fig. 8.

##  Appendix D Societal Impact

DiffusionAvatars provides the means to generate realistic images of faces with
control over viewpoint, pose, and expression. As such, there is a risk of
misusing an avatar to the original person’s disadvantage. However, we focus on
3D applications such as immersive teleconferencing or character animation, not
indistinguishable 2D video synthesis. Our method does not model lighting, the
background, or the torso. Therefore, we believe a detection system can be
built to identify generations of our method by exploiting such characteristics
[56].
All subjects shown in the paper and the supplemental video are part of the
NeRSemble dataset [33] and have consented to their recordings being used for
research purposes.

FLAME [35] |  Ours |  GT
---|---|---

Figure 6: Effect of FLAME vs. NPHM. Using FLAME instead of NPHM as the
underlying morphable head model makes the generated faces less expressive.
This is due to FLAME’s limited geometry and expression space, which does not
provide enough geometric cues for our diffusion-based neural renderer.

w/o exp. cond. |  Ours |  GT
---|---|---

Figure 7: Effect of expression conditioning. Directly providing $z_{exp}$ to
the diffusion model helps synthesize areas not modeled by NPHM’s geometry,
such as eye movement.

w/o LDM prior |  Ours |  GT
---|---|---

Figure 8: Effect of LDM prior. Using a pre-trained LDM instead of training the
diffusion model from scratch steers the generation towards more realistic
faces.