# Speech2UnifiedExpressions: Synchronous Synthesis of Co-Speech Affective Face
and Body Expressions from Affordable Inputs

Uttaran Bhattacharya
Adobe Inc.
San Jose, CA, USA Work partially done while Uttaran was a Ph.D. student at UMD
Aniket Bera
Purdue University
West Lafayette, IN, USA  Dinesh Manocha
University of Maryland
College Park, MD, USA

###### Abstract

We present a multimodal learning-based method to simultaneously synthesize co-
speech facial expressions and upper-body gestures for digital characters using
RGB video data captured using commodity cameras. Our approach learns from
sparse face landmarks and upper-body joints, estimated directly from video
data, to generate plausible emotive character motions. Given a speech audio
waveform and a token sequence of the speaker’s face landmark motion and body-
joint motion computed from a video, our method synthesizes the motion
sequences for the speaker’s face landmarks and body joints to match the
content and the affect of the speech. We design a generator consisting of a
set of encoders to transform all the inputs into a multimodal embedding space
capturing their correlations, followed by a pair of decoders to synthesize the
desired face and pose motions. To enhance the plausibility of synthesis, we
use an adversarial discriminator that learns to differentiate between the face
and pose motions computed from the original videos and our synthesized motions
based on their affective expressions. To evaluate our approach, we extend the
TED Gesture Dataset to include view-normalized, co-speech face landmarks in
addition to body gestures. We demonstrate the performance of our method
through thorough quantitative and qualitative experiments on multiple
evaluation metrics and via a user study. We observe that our method results in
low reconstruction error and produces synthesized samples with diverse facial
expressions and body gestures for digital characters. The relevant source code
and dataset are available at
https://github.com/UttaranB127/speech2unified_expressions.

Figure 1: Synthesizing unified co-speech 3D face and pose expressions. Our
method uses the speech audio, the corresponding text transcripts, the
speaker’s unique IDs, and their sparse 3D face landmarks and pose sequences
computed from RGB video data. It learns a combined embedding space that
captures the correlations between all these inputs and leverages them to
generate synchronous affective expressions for faces and poses in a continuous
motion space.

##  1 Introduction

Human communications through digital platforms and virtual spaces are
prevalent in many applications, including online learning [34, 36, 52],
virtual interviewing [7], counseling [16], social robotics [60], automated
character designing [40], storyboard visualizing for consumer media [31, 58],
and creating large-scale metaverse worlds [45]. Simulating immersive
experiences in such digital applications necessitates the development of
plausible human avatars with expressive faces and body motions. This is a
challenging problem to approach at scale, given the necessity and diversity of
human expressions in human-human interactions [46, 42]. Further, humans
express simultaneously through multiple cues or modalities, such as their
speech, facial expressions, and body gestures [43], increasing the
dimensionality of the problem. The emotional expressions from these different
modalities are also synchronous, i.e., they follow the same rhythm of
communication and complement each other to convey a sense of presence [32].

In this paper, we consider the problem of synthesizing 3D digital human
motions with synchronous facial expressions and upper-body gestures aligned
with speech audio inputs. Given the speech audio, existing approaches commonly
tackle the sub-problems of “talking heads” [29] – synthesizing lip movements
and facial expressions given the speech audio, and co-speech gesture synthesis
[61] – synthesizing poses for upper-body gestures, including head motions.
Recent approaches synthesize head and body motions simultaneously [26, 59],
but consider a limited set of speakers and their expressions. More general
motion synthesis methods consider full-body motions from various modalities,
including text prompts [19, 15], object interactions [20, 55], and guidance
motions [51, 21], but do not combine modalities (such as face and pose) in the
output. The inherent difficulty in synthesizing expressions synchronized
across diverse speakers is to under the correlations between the modalities
for both the expressions and the individual styles [2]. In other words, not
only is the combined space of the multimodal expressions very high-
dimensional, but only a small fraction of that space corresponds to valid
expressions for different speakers. Moreover, existing approaches generally
require specialized data, such as dense 3D face scans [14] and motion-captured
gestures [12, 11], often employ parameter-dense and compute-heavy approaches,
such as those based on denoising diffusion [63, 4] to provide meaningful
results. By contrast, we aim to develop a lightweight method for synchronous
co-speech face and pose expressions by leveraging large-scale video datasets
[60], paving the way to synthesize fully expressive 3D digital humans for
democratized use.

Main Contributions. We present a multimodal learning method to synthesize 3D
digital characters with synchronous affective expressions on faces and upper-
body poses, given speech audio. We also consider both intra- and inter-speaker
variabilities by random sampling on a latent space for speakers. Our main
contributions include:

  * •

Synchronous co-speech face and pose expression synthesis. Our method
simultaneously synthesizes face and upper-body pose expressions given speech
audio through a generative multimodal embedding space and an affective
discriminator. Our method reduces the mean absolute errors on the face
landmarks by $30\%$, and the body poses by $21\%$, compared to the respective
baselines for faces and poses, thereby indicating measurable benefits over
asynchronously combining the synthesized outputs of the two modalities.

  * •

Using data from affordable commodity cameras. In contrast to facial expression
synthesis using dense 3D face scans or gesture synthesis from expensive
motion-captured data, our method only relies on sparse face landmarks and pose
joints obtainable from commodity hardware such as video cameras. As a result,
our method scales affordably to large datasets and is applicable in large-
scale social applications.

  * •

Plausible motions, evaluation metric for facial expressions. Through
quantitative evaluations and user studies, we verify that our synthesized
synchronous expressions are satisfactory to human observers. We also propose
the Fréchet Landmark Distance to evaluate the quality of the synthesized face
landmarks.

  * •

TED Gesture+Face Dataset. We extend the TED Gesture Dataset to include 3D face
landmarks extracted from the raw videos that we denoise and align with the
poses. We release this multimodal dataset of speech audio, 3D face landmarks,
and 3D body pose joints with our paper and the associated source code.

Figure 2: Network architecture for synchronous synthesis of co-speech face and
pose expressions. Our generator encodes all the inputs: the speech audio, the
corresponding test transcript, the speaker ID, the seed 3D face landmarks, and
the seed 3D poses into a multimodal embedding space. It decodes variables from
this space to produce the synchronized sequences of co-speech 3D face
landmarks and poses. Our discriminator classifies these synthesized sequences
and the corresponding ground truths (3D motions of the original speakers),
computed directly from the videos, into two different classes based both on
their plausibility and their synchronous expressions. To obtain our rendered
3D character motions, we combine the outputs of our generator with our phoneme
predictor network and map them to 3D meshes.

##  2 Related Work

We briefly review prior work on perceiving multimodal affective expressions,
particularly from faces, speech, and gestures, and synthesis of co-speech face
and pose motions.

##### Perceiving Multimodal Affective Expressions.

Studies in psychology and affective computing indicate that humans express
emotions simultaneously through multiple modalities, including facial
expressions, prosody and intonations of the voice, and body gestures [54, 43].
Methods for detecting facial expressions [22] generally depend on facial
action units [62]. Methods for detecting various affective vocal patterns
commonly use Mel-Frequency Cepstral Coefficients (MFCCs) [44]. Methods to
detect emotions from body gestures use physiological features, such as arm
swings, spine posture, and head motions that are either pre-defined [8, 6] or
learned automatically from the gestures [9]. The emotions themselves can be
represented either as discrete categories, such as the Ekman emotions [17] or
as combinations of continuous dimensions, such as the Valence-Arousal-
Dominance (VAD) model [41]. In our work, we leverage the current approaches
for detecting facial, vocal, and pose expressions to design our co-speech face
and gesture synthesis method. While we do not explicitly consider specific
emotions, our representation implicitly considers emotions in the continuous
VAD space, leading to appropriately expressive face and pose synthesis.

##### Synthesizing Co-Speech Motions.

We consider digital characters with faces and body gestures.

Co-Speech Facial Expressions. Wang and Soong [57] compute controllable
parameters for synthesizing talking heads with desired facial expressions
using a Hidden Markov Model and MFCCs of the speech audio. Recent techniques
automate the facial motions for large-scale synthesis using generative
paradigms, such as VAEs [25] and GANs [49]. Karras et al. [29] train a DNN to
map speech audio to 3D face vertices conditioned on learned latent features
corresponding to different facial expressions. Zhou et al. [64], learn
sequences of predefined visemes using LSTM networks from audio. Cudeiro et al.
[14] propose a dataset of 4D face scans and learn per-vertex offsets to
synthesize the face motions from audio. Richard et al. [48] learn co-speech
facial motions using dense face meshes by disentangling facial features
correlated and uncorrelated with speech. Sinha et al. [53] focus on adding
emotional expressions to the faces. Lahiri et al. [33] focus on the accuracy
of the lip movements and use an autoregressive approach to synthesize 3D
vertex sequences for the lips synced with the speech audio. In contrast to
these approaches, our facial expression synthesis method uses much sparser 3D
face landmarks detected from real-world videos with arbitrary orientations and
lighting conditions of the faces w.r.t. the cameras, and synthesizes mutually
coherent facial and pose expressions.

Co-Speech Gestures. Co-speech gesture synthesis is a special case of gesture
stylization, where the style refers to the pose expressions inferred from and
aligned with the speech. This line of work has been richly explored [13, 47,
35, 3, 27, 38, 37, 39]. Ginosar et al. [23] propose a method to synthesize
speaker-specific co-speech gestures by training a neural network given their
identities and individual gesticulation patterns. Ferstl et al. [18]
additionally propose using adversarial losses in the training process to
improve the fidelity of the synthesized gestures. Yoon et al. [61] extend the
concept of individualized gestures to a continuous space of speakers to
incorporate natural variability in the synthesized gestures even for the same
speaker. Bhattacharya et al. [10] build on top of [61] to improve the
affective expressions in the co-speech gestures. More recent methods have also
explored diffusion-based approaches for editability [4]. Our method conditions
the gesture synthesis on both the input speech and the synthesized facial
expressions.

Co-Speech Multimodal Expressions. Co-speech face and upper-body generation has
gained particular interest recently, primarily due to the availability of rich
3D datasets of popular speakers [26]. Current approaches train adversarial
encoder-decoder models on datasets of one speaker at a time [26] and use
vector quantization for tokenized generation using a transformer [59]. These
approaches consider a fixed set of speakers and lose fine-grained expressions
when using quantization. In our work, we consider the combined continuous
space of affective face and body expressions and develop a network
generalizable to multiple speakers.

##  3 Synchronous Face and Pose Synthesis

Given a speech audio waveform $a$, the corresponding text transcript $w$, the
speaker’s unique ID $k$ in a set of speakers $K$, and the associated seed face
landmark deltas $f_{1:T_{s}}$ and seed pose unit vectors $u_{1:T_{s}}$,
$T_{s}$ being the number of seed time steps, we synthesize the synchronous
sequences of face landmark deltas $f_{1:T}$ and pose unit vectors $u_{1:T}$
for the speaker for the $T$ prediction time steps ($T\gg T_{s}$), matching the
content and the affect in their speech. We describe our end-to-end pipeline,
including a detailed description of our inputs and outputs and their usage. We
also provide the details of obtaining these facial landmarks and poses from
input videos.

###  3.1 Face and Pose Preprocessing from Video

Given a video, we use Multi-Task Cascaded CNNs [62] to extract the 3D face
landmarks. Since the faces can be arbitrarily oriented w.r.t. the camera, we
rigidly transform the face landmarks per frame to a reference frame in the
normalized view, where the face looks towards the camera. For each frame in
the input video, we use the rotation and the translation given by the Umeyama
method [56] to map the face landmarks in that frame to the face landmarks in
the reference frame. We also use similarly view-normalized 3D poses. View
normalization is helpful for two key reasons. First, it eliminates relative
camera movements across the video frames and prevents a learning-based method
from confusing camera movements with face and pose expression changes. Second,
a frontal view offers maximal visibility of the faces and the poses, and
minimizes errors in detecting the 3D face landmarks and body joints.

###  3.2 Computing Face and Pose Expressions

We consider a reference neutral expression $\mathcal{F}\in\mathbb{R}^{L\times
3}$ for each user, $L$ being the number of face landmarks. To synthesize
facial expressions, we compute the relative motion of each landmark w.r.t. the
reference expression. Specifically, we obtain the configuration
$\mathcal{F}_{t}$ at time step $t$ as

$\mathcal{F}_{t}=\mathcal{F}+f_{t},$ (1)

where $f_{t}\in\mathbb{R}^{L}$ denotes the set of relative motions of the
landmarks w.r.t. $\mathcal{F}$ at time step $t$.

(a) Face encoder

(b) Pose Encoder

(c) Face Decoder

(d) Pose Decoder

Figure 3: Face and pose encoders and decoders. We show their architectures
with the layer sizes denoted (details in Sec. 3.3.2). Our architectures depend
on the hierarchical anatomical component (AC) graphs for both faces and poses
that efficiently learn their corresponding affect representations using
spatial-temporal graph convolutions (green nodes and edges), 2D convolutions
(teal blocks), 2D batch normalizations (pink blocks), and fully-connected
layers (orange planes).

On the other hand, we assume the body joints are rigidly connected by the
bones. We represent each user’s body joints as 3D point vectors
$\mathcal{P}\in\mathbb{R}^{J\times 3}$ in a global coordinate space, where $J$
is the number of joints. We consider directed line vectors connecting adjacent
joints. The direction is along the path from the root (pelvis) joint to the
end effectors (such as wrists). These 3D point and line vectors collectively
form a directed tree with $J$ nodes and $J-1$ edges. We assume that the
magnitudes of these line vectors correspond to the bone lengths and that these
magnitudes are known and fixed. To synthesize the users’ body gestures, we
compute the orientations of these line vectors at each time step $t$ in the
reference frame of the global coordinate space. Specifically, for each bone
$b$ with bone length (magnitude) $\left\|b\right\|$ and connecting the source
joint $s_{b}\left(t\right)$ to the destination joint $d_{b}\left(t\right)$ at
time step $t$, we compute a unit vector $u_{t}$ such that

$d_{b}=s_{b}+\frac{\left\|b\right\|}{\left\|u_{t}\right\|}u_{t}.$ (2)

We do not assume any locomotion, i.e., we consider the root joint is fixed at
the global origin at all the time steps.

###  3.3 Synthesizing Faces and Poses

Our network architecture (Fig. 2) consists of a phoneme predictor to predict
the lip shapes corresponding to the audio and a generator-discriminator pair
to synthesize plausible co-speech face and pose expressions. We design our
phoneme predictor following prior approaches [33] and provide its details in
Sec. 3.3.5. Our generator follows a multimodal learning strategy. It consists
of separate encoders to transform the speech audio, the text transcript, the
speaker ID, the seed face landmark deltas, and the seed pose unit vectors into
a latent embedding space representing their correlations. It subsequently
synthesizes the appropriate face and pose motions from this multimodal
embedding space. Our discriminator enforces our generator to synthesize
plausible face and pose motions in terms of their affective expressions. To
this end, we use the same encoder architecture for the faces and the poses as
in our generator, but learned separately. We describe each of the components
of our generator and discriminator.

####  3.3.1 Encoding Speech, Text, and Speaker IDs

We use the Mel-Frequency Cepstral Coefficients (MFCCs) for the speech audio to
accurately capture the affective intonations in the speech, and use an MFCC
encoder to obtain speech-based latent embeddings
$\hat{a}\in\mathbb{R}^{T\times D_{a}}$ of dimension $D_{a}$ as

$\hat{a}=\textrm{MFCCEncoder}\left(a;\theta_{\textrm{MFCC}}\right),$ (3)

where $\theta_{\textrm{MFCC}}$ represents the trainable parameters.

Similarly, we use the sentiment-aware FastText [50] embeddings of the words in
the transcript and a convolution-based text encoder to obtain the text-based
latent embeddings $\hat{w}\in\mathbb{R}^{T\times D_{w}}$ of dimensions $D_{w}$
as

$\hat{w}=\textrm{TextEncoder}\left(w;\theta_{\textrm{text}}\right),$ (4)

where $\theta_{\textrm{text}}$ represents the trainable parameters.

We also represent the speaker IDs $k\in\left\\{0,1\right\\}^{K}$ as one-hot
vectors for a total of $K$ speakers and use a speaker encoder to obtain the
parameters $\mu_{k}\in\mathbb{R}^{D_{k}}$ and
$\Sigma_{k}\in\mathbb{R}_{+}^{D_{k}\times D_{k}}$ of a latent distribution
space of dimension $D_{k}$ as

$\mu_{k},\Sigma_{k}=\textrm{SpeakerEncoder}\left(k;\theta_{\textrm{speaker}}\right),$
(5)

where $\theta_{\textrm{speaker}}$ represents the trainable parameters. The
latent distribution space enables us to sample a random vector $\hat{k}$
representing a speaker who is an arbitrary combination of the $K$ speakers in
the dataset. This allows for variations in the synthesized motions even for
the same original speaker by slightly perturbing their speaker IDs in the
latent distribution space, leading to more plausible results on multiple runs
of our network. To learn faces and poses with appropriate expressions, we
represent them as multi-scale graphs and encode them using graph convolutional
networks.

####  3.3.2 Encoding Affective Expressions

Our face landmarks are based on action units [62]. We represent the sequence
of 3D landmarks $f_{1:T_{s}}\in\mathbb{R}^{T_{s}\times L\times 3}$ as a
spatial-temporal anatomical component (AC) graph. Spatially, we consider
landmarks belonging to the same anatomical component (Sec. 3.2) and nearest
landmarks across different anatomical components to be adjacent. Temporally,
all landmarks are adjacent to their temporal counterparts (same nodes at
different time steps) within a predetermined time window. We consider the
eyes, the nose, the lips, and the lower jaw as the anatomical components. We
show the face landmarks graph in Fig. 3(a) with all the intra- and inter-
anatomical-component adjacencies marked with lines. We apply a sequence of
spatial-temporal graph convolutions on this graph to learn from the localized
motions of the landmarks and obtain embeddings
$\tilde{f}\in\mathbb{R}^{T_{s}\times L\times D_{f}}$ of feature dimension
$D_{f}$ as

$\tilde{f}=\textrm{STGCN}_{f}\left(f_{1:T_{s}};\theta_{\textrm{STGCN}_{f}}\right),$
(6)

where $\theta_{\textrm{STGCN}_{f}}$ represents the trainable parameters. We
obtain a face anatomy graph from the landmarks graph, where we consider the
nodes to represent entire anatomical components and the graph to be fully
connected. To compute such a graph, we append the features of intra-
anatomical-component nodes in the graph into collated features
$l\in\mathbb{R}^{T_{s}\times L_{l}\times n_{l}D_{f}}$, where $L_{l}$ denotes
the number of anatomical components and $n_{l}$ denotes the number of landmark
nodes within each anatomical component. We take $n_{l}$ as the number of nodes
in the anatomical component with the most landmarks and perform zero padding
as appropriate to obtain the full collated features for the other components.
This hierarchically pooled representation provides a “higher-level” view of
the face and helps our network learn from the correlations between the motions
of the different anatomical components. Specifically, we use another set of
spatial-temporal graph convolutions to obtain the embeddings
$\tilde{l}\in\mathbb{R}^{T_{s}\times L_{l}\times D_{l}}$ of feature dimension
$D_{l}$ as

$\tilde{l}=\textrm{STGCN}_{l}\left(l;\theta_{\textrm{STGCN}_{l}}\right),$ (7)

where $\theta_{\textrm{STGCN}_{l}}$ represents the trainable parameters.
Collectively, the landmarks graph and the face anatomy graph provide
complementary information to our network to encode and synthesize the required
facial expressions at both the macro (anatomy) and the micro (landmark)
levels. To complete our encoding, we flatten out the features of all the
anatomical components in $\tilde{l}$, i.e., reshaping such that
$\tilde{l}\in\mathbb{R}^{T_{s}\times L_{l}D_{l}}$, and transform them using
standard convolutional layers on the flattened feature channel and the
temporal channel separately. This gives us our latent space embeddings
$\hat{l}\in\mathbb{R}^{T\times D_{\tilde{l}}}$ as

$\hat{l}=\textrm{ConvT}_{\tilde{l}}\left(\textrm{ConvS}_{\tilde{l}}\left(\tilde{l};\theta_{\textrm{ConvS}_{\tilde{l}}}\right);\theta_{\textrm{ConvT}_{\tilde{l}}}\right),$
(8)

where $\theta_{\textrm{ConvS}_{\tilde{l}}}$ and
$\theta_{\textrm{ConvT}_{\tilde{l}}}$ represent the trainable parameters.

For the pose representation, we consider a pose graph of the upper body with
$J-1$ bones represented with line vectors $u_{1:T_{s}}$ (Fig. 3(b)). We
consider bones connected to each other or connected through a third bone to be
adjacent. We use a set of spatial-temporal graph convolutions to leverage the
localized motions of these bones and obtain embeddings
$\tilde{u}\in\mathbb{R}^{T_{s}\times D_{u}}$ of feature dimension $D_{u}$ as

$\tilde{u}=\textrm{STGCN}_{u}\left(u_{1:T_{s}};\theta_{\textrm{STGCN}_{u}}\right),$
(9)

where $\theta_{\textrm{STGCN}_{u}}$ represents the trainable parameters.
Similar to the face landmarks, we also consider a hierarchically pooled
representation of the bones $v\in\mathbb{R}^{T_{s}\times L_{j}\times
n_{j}D_{u}}$, where $L_{j}=3$ are the three anatomical components, the torso
and the two arms, represented as single nodes each consisting of $n_{j}$ nodes
from the pose graph. In the pose anatomy graph, we consider the two arms to be
adjacent to the torso but not to each other, as they can move independently.
We apply a second set of spatial-temporal graph convolutions on the collated
features $v$ to obtain the embeddings $\tilde{v}\in\mathbb{R}^{T_{s}\times
L_{j}\times D_{v}}$ as

$\tilde{v}=\textrm{STGCN}_{v}\left(v;\theta_{\textrm{STGCN}_{v}}\right)$ (10)

where $\theta_{\textrm{STGCN}_{v}}$ represents the trainable parameters. To
subsequently obtain the latent space embeddings $\hat{v}\in\mathbb{R}^{T\times
D_{\tilde{v}}}$, we apply separate spatial and temporal convolutions on the
flattened graph-convolved features $\tilde{v}\in\mathbb{R}^{T_{s}\times
L_{j}D_{v}}$, as

$\hat{v}=\textrm{ConvT}_{\tilde{v}}\left(\textrm{ConvS}_{\tilde{v}}\left(\tilde{v};\theta_{\textrm{ConvS}_{\tilde{v}}}\right);\theta_{\textrm{ConvT}_{\tilde{v}}}\right),$
(11)

where $\theta_{\textrm{ConvS}_{\tilde{v}}}$ and
$\theta_{\textrm{ConvT}_{\tilde{v}}}$ represent the trainable params.

Figure 4: Qualitative results. Snapshots from two of our synthesized samples
showing the text transcript of the speech and the corresponding face and pose
expressions (row 1). We also zoom in on the eyebrow (row 2) and lip (row 3)
expressions for better visualization. We observe a smile, raised eyebrows, and
stretched arms (left) for the word ‘excited’, and frowns on the eyebrows and
lips (right) for the words ‘very sorry’.

####  3.3.3 Synthesizing Synchronous Motions

Our synchronous synthesis relies on learning the multimodal distributions of
the individual modalities of audio, text, speaker ID, face expressions, and
pose expressions, given their individual distributions. To this end, we append
all the latent space embeddings — $\hat{a}$ for the audio, $\hat{w}$ for the
text, $\hat{k}$ for the random speaker representation, repeated over all the
$T$ time steps, $\hat{l}$ for the seed landmarks and $\hat{v}$ for the seed
poses — into a vector $\hat{e}\in\mathbb{R}^{T\times H}$ representing a
multimodal embedding space of all the inputs. Here,
$H=D_{a}+D_{w}+D_{k}+D_{\tilde{l}}+D_{\tilde{v}}$ denotes the latent space
dimension. On training, our network learns the correlations between the
different inputs in this multimodal embedding space. To synthesize our face
landmark motions $f_{1:T}\in\mathbb{R}^{T\times L\times 3}$, we apply separate
spatial and temporal convolutions on the multimodal embeddings $\hat{e}$ to
capture localized dependencies between the feature values followed by fully-
connected layers capturing all the dependencies between the feature values
(Fig. 3(c)), as

$f_{1:T}=\textrm{FC}_{f\hat{e}}\left(\textrm{ConvS}_{f\hat{e}}\left(\textrm{ConvT}_{f\hat{e}}\left(\hat{e};\theta_{\textrm{ConvT}_{f\hat{e}}}\right);\theta_{\textrm{ConvS}_{f\hat{e}}}\right);\theta_{\textrm{FC}_{f\hat{e}}}\right)$,

(12)

where $\theta_{\textrm{ConvT}_{f\hat{e}}}$,
$\theta_{\textrm{ConvS}_{f\hat{e}}}$, and $\theta_{\textrm{FC}_{f\hat{e}}}$
represent the trainable parameters. The output $f_{1:T}$ from the fully
connected layers has shape $T\times 3L$, which we reshape into $T\times
L\times 3$ to get our desired 3D face landmark sequences.

We similarly synthesize the line vectors
$u_{1:T}\in\mathbb{R}^{T\times\left(J-1\right)\times 3}$ using separate
spatial and temporal convolutions on the multimodal embeddings $\hat{e}$,
followed by fully-connected layers (Fig. 3(d)), as

$u_{1:T}=\textrm{FC}_{u\hat{e}}\left(\textrm{ConvS}_{u\hat{e}}\left(\textrm{ConvT}_{u\hat{e}}\left(\hat{e};\theta_{\textrm{ConvT}_{u\hat{e}}}\right);\theta_{\textrm{ConvS}_{u\hat{e}}}\right);\theta_{\textrm{FC}_{u\hat{e}}}\right)$,

(13)

where $\theta_{\textrm{ConvT}_{u\hat{e}}}$,
$\theta_{\textrm{ConvS}_{u\hat{e}}}$, and $\theta_{\textrm{FC}_{u\hat{e}}}$
represent the trainable parameters. Given the synthesized face and pose
motions, we use our discriminator to determine how well their affective
expressions match the corresponding ground truths in the training data. We
obtain our ground truths as the 3D face landmarks and the 3D pose sequences
computed from the full training video data.

####  3.3.4 Determining Plausibility Using Discriminator

Our discriminator takes in the synchronously synthesized face motions
$f_{1:T}$ and pose motions $u_{1:T}$, and encodes them using encoders with the
same architecture as our generator (Sec. 3.3.2), with only the number of input
time steps being $T$ instead of $T_{s}$. This gives us the corresponding
latent space embeddings $\hat{l}$ and $\hat{v}$. Similar to our generator, we
concatenate these embeddings into a multimodal embedding vector
$\hat{e}\in\mathbb{R}^{T\times\left(D_{\tilde{l}}+D_{\tilde{v}}\right)}$. But
different from our generator, we pass these multimodal embeddings through a
fully-connected classifier network $\textrm{FC}_{\textrm{disc}}$ to obtain
class probabilities $c_{\textrm{disc}}\in\left[0,1\right]$ per sample, as

$c_{\textrm{disc}}=\textrm{FC}_{\textrm{disc}}\left(\hat{e};\theta_{\textrm{FC}_{\textrm{disc}}}\right),$
(14)

where $\theta_{\textrm{FC}_{\textrm{disc}}}$ represents the trainable
parameters. Our discriminator learns to perform unweighted binary
classification between the synthesized face and pose motions and the ground
truths in terms of their synchronous affective expressions. Our generator, on
the other hand, learns to synthesize samples that our discriminator cannot
distinguish from the ground truth based on those affective expressions.

Figure 5: Qualitative comparisons. For the same input speech, represented by
the text transcript at the top, we compare the visual quality of our
synthesized character motions with the original speaker motions and three of
our ablated versions: one without synchronous face and pose synthesis, one
without our anatomical component (AC) graphs for faces and poses, and one
without our discriminator. We observe that our synthesized motions are
visually the closest to the original speaker motions compared to the ablated
versions. We elaborate on their visual qualities in Sec. 6.4.

####  3.3.5 Phoneme Predictor

We train a separate network to learn the positions of the lip landmarks for
the different phonemes in the audio. Our synthesis network separately learns
the motions of the lip corners denoting the different facial expressions, and
we superpose them to the phoneme-based lip shapes to complete the lip motions.
Our phoneme predictor predicts the 3D positions of all the landmarks on the
inner and the boundaries of the lips over all the $T$ prediction time steps,
which we denote as $p_{1:T}\ in\mathbb{R}^{T\times L_{\textrm{lip}}\times 3}$.
Following prior approaches [33], we design a CNN backbone connected to fully
connected blocks to predict the lip landmarks from the spectrograms of the
speech inputs. Specifically, given the speech audio waveform $a$, we compute

$p_{1:T}=\textrm{PhonemePred}\left(a;\theta_{\textrm{PhonemePred}}\right),$
(15)

where $\theta_{\textrm{PhonemePred}}$ represents the trainable parameters.

##  4 TED Gesture+Face Dataset

We present our TED Gesture+Face Dataset, which we use to train and test our
network. We elaborate on collecting and processing our dataset for training
and testing.

##### Dataset Collection.

The TED Gesture Dataset [60] consists of videos of TED talk speakers together
with text transcripts of their speeches and their 3D body poses extracted in a
global frame of reference. The topics range from personal and professional
experiences to discourses on educational topics and instructional and
motivational storytelling. The speakers come from a wide variety of social,
cultural, and economic backgrounds, and are diverse in age, gender, and
physical abilities.

##### Dataset Processing.

The 3D poses in the original TED Gesture Dataset [60] are view-normalized to
face front and center at all time steps. We compute similarly view-normalized
3D face landmarks of the speakers (Sec. A.1). Similar to the original TED
dataset, we divide the 3D pose and face landmark sequences into equally-sized
chunks of size $T=34$ time steps at a rate of 15 fps. Additionally, to reduce
the jitter in the predicted 3D face landmarks and pose joints from each video,
we sample a set of “anchor” frames at a rate of 5 fps and perform bicubic
interpolation to compute the face landmark and pose joint values in the
remaining frames. We use the first $4$ time steps of pose and face landmarks
as our seed values (Sec. 3.3), and predict the next $30$ time steps. The
processed dataset consists of 200,038 training samples, 26,903 validation
samples, and 26,245 test samples, following a split of 80%-10%-10%.

##  5 Training Details

We train our phoneme predictor network using reconstruction losses for the lip
shapes. We train our synthesis network using a combination of reconstruction
losses for the face and the pose motions, the cross-speaker diversity loss to
enforce visual differences in expressions across speakers, and the generative
adversarial loss for added regularization. We describe these loss functions
and our training and testing procedures.

###  5.1 Phoneme Predictor Losses

We represent our phoneme predictor loss as the robust $\ell_{1}$-norm
reconstruction loss between the ground truth and the synthesized lip landmark
positions and velocities over the prediction time steps $T$ as

$\mathcal{L}_{\textrm{ph}}=\sum_{t=1}^{T}{\left\|p^{\left(\textrm{GT}\right)}_{t}-p^{\left(\textrm{sn}\right)}_{t}\right\|_{1}+\left\|\Delta_{t}p^{\left(\textrm{GT}\right)}_{t}-\Delta_{t}p^{\left(\textrm{sn}\right)}_{t}\right\|_{1}},$
(16)

where the superscripts $\left(\textrm{GT}\right)$ and
$\left(\textrm{sn}\right)$, respectively, denote the ground-truth and the
synthesized data. $\Delta_{t}$ denotes the discrete forward difference between
adjacent time steps $t$ and $t-1$.

###  5.2 Synchronous Synthesis Network Losses

We use reconstruction losses to robustly align the outputs of our generator
with the corresponding ground-truth face and pose motions. We use the
generative adversarial loss to ensure that the synthesized motions are
plausible, the affective expressions match the corresponding ground truths,
and prevent the mode collapse of only synthesizing singular expressions.

####  5.2.1 Reconstruction Losses

We write our reconstruction losses as the $\ell_{1}$-norm difference between
the ground truth and the synthesized face and pose positions and motions over
the $T$ prediction time steps as

$\displaystyle\mathcal{L}_{\textrm{Rec}}$
$\displaystyle=\sum_{t=1}^{T}\left\|\mathcal{F}^{\left(\textrm{GT}\right)}_{t}-\mathcal{F}^{\left(\textrm{sn}\right)}_{t}\right\|_{1}+\left\|\mathcal{P}^{\left(\textrm{GT}\right)}_{t}-\mathcal{P}^{\left(\textrm{sn}\right)}_{t}\right\|_{1}$
$\displaystyle+\lambda_{\textrm{vel}}\left(\left\|f^{\left(\textrm{GT}\right)}_{t}-f^{\left(\textrm{sn}\right)}_{t}\right\|_{1}+\left\|u^{\left(\textrm{GT}\right)}_{t}-u^{\left(\textrm{sn}\right)}_{t}\right\|_{1}\right)$
$\displaystyle+\lambda_{\textrm{acc}}\left(\left\|\Delta_{t}f^{\left(\textrm{GT}\right)}_{t}-\Delta_{t}f^{\left(\textrm{sn}\right)}_{t}\right\|_{1}+\left\|\Delta_{t}u^{\left(\textrm{GT}\right)}_{t}-\Delta_{t}u^{\left(\textrm{sn}\right)}_{t}\right\|_{1}\right),$
(17)

where $\lambda_{\textrm{vel}}$ and $\lambda_{\textrm{acc}}$ are the relative
weighting factors. We use the velocity and acceleration losses to enforce
smoothness in the synthesized motions by reducing jitters.

####  5.2.2 Cross-Speaker Diversity Loss

Our cross-speaker diversity loss $\mathcal{L}_{\textrm{CSD}}$ follows that of
Yoon et al. [61], consisting of a ranking loss between the ground-truth face
and pose motions, and the synthesized face and pose motions using the same
speaker as the ground-truth (positive example) and a randomly chosen different
speaker (negative example).

####  5.2.3 Generative Adversarial Loss

The generative adversarial loss consists of opposing losses
$\mathcal{L}_{\textrm{Gen}}$ for the generator and
$\mathcal{L}_{\textrm{Dis}}$ for the discriminator, following a min-max
optimization strategy [24]. We write these losses as

$\displaystyle\mathcal{L}_{Gen}$
$\displaystyle=-\mathbb{E}\left[\log\left(c_{\textrm{disc}}^{\textrm{GT}}\right)\right],$
(18) $\displaystyle\mathcal{L}_{Dis}$
$\displaystyle=-\mathbb{E}\left[\log\left(c_{\textrm{disc}}^{\textrm{GT}}\right)\right]-\mathbb{E}\left[\log\left(1-c_{\textrm{disc}}^{\textrm{sn}}\right)\right],$
(19)

where $c_{\textrm{disc}}$ denotes the output of our discriminator network (Eq.
15). This loss adds plausibility to our synthesized samples by forcing them to
have affective expressions similar to those of the corresponding ground-truth
samples.

###  5.3 Training Procedure

We train our phoneme predictor network using the Adam optimizer [30] with
$\beta_{1}=0.5$, $\beta_{2}=0.999$, a batch size of $1024$, and a learning
rate of $10^{-3}$ for 500 epochs. We train our synthesis network using the
Adam optimizer [30] with $\beta_{1}=0.5$, $\beta_{2}=0.999$, a batch size of
$256$, and learning rates of $10^{-4}$ for our generator and $5\times 10^{-5}$
for our discriminator, both decayed by a factor of $0.999$ per epoch, for 1000
epochs. We train both our phoneme detector network and our synthesis network
on an NVIDIA GeForce RTX 2080 Ti GPU, which takes $3$ seconds and $7$ seconds
per epoch, respectively.

###  5.4 Testing and Rendering

We provide the details of the testing procedure of our network and the
rendering of our synthesized outputs in a 3D environment.

###  5.5 Testing Procedure and Mapping to Digital Characters

Each test sample for our network consists of a speech audio waveform, the
corresponding text transcript, a speaker ID, and the speaker’s seed face and
pose motions. Our phoneme predictor network provides the lip sync for the
given speech audio, and the generator of our synthesis network provides the
required face and pose motions. We superpose the lip landmarks given by our
phoneme predictor network with the lip corner landmarks given by our generator
at each prediction time step to obtain the complete lip motions of the
speaker. We map these motions to a rigged 3D human upper-body mesh in Blender.
For mapping the face motions, we set a one-to-one mapping between our face
landmarks and the landmarks on the face of the human mesh, and use them as
control points for the facial motions of the mesh. For mapping the pose
motions, we use FABRIK [5] to obtain the joint rotations given our predicted
joint positions and use those rotations to animate the rigged human mesh.

###  5.6 Rendering and Visualization

Given an input speech audio, we can synthesize the motions for our pre-rigged
digital characters at an interactive rate of about 250 frames per second on an
NVIDIA GeForce RTX 2080 Ti GPU. We design our digital environment using
Blender. For each of our digital characters, we place them on a stage and
position the camera such that it looks front and center at the agent. As the
character narrates the input speech audio using our synthesized face and
upper-body expressions, we slowly pan the camera in to get a more focused view
of those expressions. Since we do not synthesize any lower-body motions, our
digital characters stay standing at their initial positions during the entire
narration. The full video demos are available with our supplementary material.

##  6 Experiments and Results

We run quantitative experiments using ablated versions of our method as
baselines. We note that Habibie et al. [26] retrain their network separately
for individual speakers belonging to the same profession (talk show hosts),
making it unsuitable for our generalized paradigm consisting of less than 50
samples each of multiple, diverse speakers. Yi et al. [59] use VQ with
transformers to synthesize faces and gestures, but are limited to the same set
of fixed speakers. We also conducted a web-based user study to evaluate the
qualitative performance of our method.

Table 1: Quantitative evaluations. Comparison with existing co-speech gesture
synthesis methods and our ablated versions (Sec. 6.1) on the metrics MALE (in
mm), MAJE (in mm), MAcE for landmarks (MAcE-LM) (in mm/s2), MAcE for poses
(MAcE-P) (in mm/s2), FLD, and FGD (Sec. 6.2). Lower values are better, bold
indicates best, and underline indicates second-best.

Method | MALE | MAJE | MAcE-LM | MAcE-P | FLD | FGD
---|---|---|---|---|---|---
Seq2Seq [60] | – | 45.62 | – | 6.33 | – | 6.62
S2G-IS [23] | – | 45.11 | – | 7.22 | – | 6.73
JEM [1] | – | 48.56 | – | 4.31 | – | 5.88
GTC [61] | – | 27.30 | – | 3.20 | – | 4.49
Speech2AffectiveGestures [10] | – | 24.49 | – | 2.93 | – | 3.54
SpeechGestureMatching [27] | – | 21.10 | – | 2.75 | – | 2.64
Ours w/o Face Synthesis | – | 28.32 | – | 3.89 | – | 4.01
Ours w/o Pose Synthesis | 11.76 | – | 9.38 | – | 22.65 | –
Ours w/o Vel.+Acc. Losses | 26.33 | 24.41 | 21.69 | 7.58 | 27.54 | 7.72
Ours w/o Discriminator | 14.62 | 27.40 | 13.44 | 11.60 | 31.93 | 8.79
Ours w/o Face AC Graph | 13.05 | 25.97 | 14.24 | 2.74 | 25.61 | 2.25
Ours w/o Pose AC Graph | 11.84 | 25.46 | 8.12 | 13.88 | 19.23 | 6.94
Ours w/o Synchronous Synthesis | 10.72 | 25.03 | 7.83 | 3.22 | 18.03 | 3.92
Ours | 9.00 | 18.36 | 6.34 | 2.52 | 15.02 | 1.79

###  6.1 Baselines

We use seven ablated versions of our method as baselines. The first two
ablations correspondingly remove the entire face (Figs. 3(a), 3(c)) and pose
components (Figs. 3(b), 3(d)) from our network, making our network learn only
talking head and only co-speech gesture syntheses. The third ablation removes
the velocity and acceleration losses from our reconstruction loss (Eqn. C.2) ,
leading to jittery motions. The fourth ablation removes the discriminator and
its associated losses (Eqn. C.4) from our training pipeline, leading to
unstable motions without appreciable expressions. The fifth and the sixth
ablations correspondingly remove the “higher-level” anatomical component (AC)
graphs of the faces (Eqn. 7) and the poses (Eqn. 10), leading to reduced
movements. The final ablation trains the face and the pose expressions
separately, learning marginal embeddings for the two modalities based on the
speech but not attending to their mutual synchronization. This ablation
directly evaluates the co-speech motions when combining separately synthesized
face and pose expressions. For completeness, we also compare with co-speech
gesture synthesis methods that only synthesize body poses. We evaluate all the
methods on our TED Gesture+Face Dataset.

###  6.2 Evaluation Metrics

Inspired by prior work [61], we evaluate using four reconstruction errors and
two plausibility errors (PEs). Our reconstruction errors include the mean
absolute landmark error (MALE) for the faces, the mean absolute joint error
(MAJE) for the poses, and their respective mean acceleration errors (MAcEs).
MALE and MAJE indicate the overall fidelity of the synthesized samples w.r.t.
the corresponding ground truths, and the MAcEs indicate whether or not the
synthesized landmarks and poses have regressed to their mean absolute
positions. To report these metrics, we multiply our ground truth and
synthesized samples by a constant scaling factor such that they all lie inside
a bounding box of diagonal length 1 m. For our PE, we use the Fréchet Gesture
Distance (FGD) designed by [61] to indicate the perceived plausibility of the
synthesized poses. To similarly indicate the perceived plausibility of the
synthesized face landmarks, we also design the Fréchet Landmark Distance
(FLD). We train an autoencoder network to reconstruct the set of face
landmarks at all time steps for all the samples in the training set of our TED
Gesture+Face Dataset. To compute FLD, we then obtain the Fréchet Inception
Distance [28] between the encoded features of the ground truth and the
synthesized samples.

###  6.3 Quantitative Evaluations

We show our quantitative evaluations in Table 1.

##### Comparison with Co-Speech Gesture Synthesis.

Since co-speech gesture synthesis methods do not synthesize face expressions,
we leave those numbers blank. For these methods, we have taken the numbers
reported by Bhattacharya et al. [10]. For the method of SpeechGestureMatching
[27], we retrain their method on the TED Gesture Dataset to report the
numbers. However, we were unable to perform similar comparative evaluations
with co-speech face synthesis methods as existing methods synthesize dense
landmarks [29] or blendshape-like features [14], which cannot be mapped one-
to-one with our sparser face landmarks.

##### Comparison with Ablated Versions

Removing either the face or the gesture components of our network leads to
poorer values across the board than using both. Without the velocity and
acceleration losses, the motions are jittery, and the MAcE losses are higher,
especially MAcE for the face landmarks. Without the discriminator, the
synthesized samples suffer from mode collapse and often produce implausible
motions, leading to higher values across the board. Without the AC graphs,
there are fewer movements in the synthesized motions and the reconstruction
errors are higher. When synthesizing face and pose expressions separately and
not synchronizing them, we observe some mismatches in when the expressions
from either modality appear and how intense they are. This indicates that
synchronous synthesis of facial expressions and body gestures leads to more
accurate and plausible movements for both the modalities, including a $30\%$
improvement on MALE and a $21\%$ improvement on MAJE, compared to trivially
combining synthesized outputs of the individual modalities.

###  6.4 Qualitative Comparisons

We visualize some of our synthesized samples in Fig. 4 and provide more
results in our supplementary video. We observe the synchronization between the
face and the pose expressions for two contrasting emotions. We also visually
compare with the original speaker motions rendered using their face landmarks
and the poses extracted from the videos and three of our ablated versions in
Fig. 5. The original speaker motions provide an “upper bound” of our
performance. The three ablated versions we compare with are: one without the
synchronous synthesis, one without our face and pose AC graphs, and one
without our discriminator. The ablated versions without either the face or the
pose synthesis, without the velocity and acceleration losses, and without our
discriminator are visually inferior in obvious ways. Therefore, we leave them
out. Without either face or pose synthesis, that modality remains static while
there is movement in the other. Without the velocity and the acceleration
losses, the overall motions regress to the mean pose. Without our
discriminator, our generator often fails to understand plausible movement
patterns, leading to unnatural limb and body shapes. Of these, we only keep
the ablations without our discriminator as our “lower bound” baseline because,
unlike the other two, this ablation has visible movements in both the face and
the pose modalities.

##  7 User Study

We conducted a web-based user study to evaluate the visual quality of our
synthesized motions in terms of their plausibility and synchronization.

(a) Set 1: Motion plausibility. Compared to the ablated versions, we observe a
higher distribution of “OK” or better for the motions of the original speakers
and our synthesized agents. The modes of all the distributions are on “OK”,
implying that the corresponding participants found the visual qualities of all
the motions to be reasonable.

(b) Set 1: Synchronization between the face and the pose expressions given the
speech. Compared to the ablated versions, we obverse clear preferences for the
motions of the original speakers and our synthesized agents. The modes of the
distributions for these two types of motions are on “somewhat expressive”
while the modes of the two ablated versions are on “no/arbitrary movements”.

(c) Set 2: Motion plausibility. Compared to the ablated version without
synchronous synthesis, we observe a higher distribution of “OK” or better for
the motions of the original speakers and our synthesized agents. Similar to
the motion plausibility in set 1, we observe modes of all the distributions on
“OK”.

(d) Set 2: Synchronization between the face and the pose expressions given the
speech. We again obverse clear preferences for the motions of the original
speakers and our synthesized agents compared to the ablated version without
synchronous synthesis. However, in contrast to the same study in set 1, we
notice the modes of the distributions for the first two types of motions are
one point lower on the Likert scale, whereas the mode for the ablated version
remains on “no/arbitrary movements”. We hypothesize this to be the consequence
of removing the other ablated versions from the participants’ cognitive
window: in the absence of other variants, the participants focused more
closely on the relative qualities of asynchronous vs. synchronous motions and
assessed them more critically.

Figure 6: Distributions of the user study responses. Likert-scale response
distributions to the two sets of motions rendered using the five different
types of face landmark and pose data (Sec. 7). We show the distributions of
each of the five Likert-scale points for each type of motion as a percentage
of the total responses across all the groups in each set.

##### Setup.

A total of 90 participants participated in our user study. All participants
were aged 18 years or older, and had normal or corrected to normal vision and
hearing. Each participant observed two sets of character motions. There were
eight groups of motions in each set, each group having a unique input speech.
In the first set, there were four types of motions in each group corresponding
to the same speech: the original speaker motions rendered using their face
landmarks and the poses extracted from the video, and motions rendered using
the face landmarks and poses synthesized by our network and two of its ablated
versions. One ablated version was without using the face and pose anatomical
component (AC) graphs for training, and one without our discriminator. In the
second set, there were three types of motions in each group corresponding to
the same speech: the original speaker motions, motions rendered using the face
landmarks and poses synthesized by our network, and the ablated version using
asynchronously synthesized faces and poses. Our motivation to separately
compare with the asynchronously synthesized motions was to eliminate
distractors from other motions and enable our participants to focus more
closely on the synchronization between the face and the pose expressions. We
randomized the order of these motions in each group in each set and kept the
order unknown to the participants. We did not present our other ablated
versions to the participants as they did not have sufficient motion and were
visually inferior in obvious ways.

Table 2: Likert-scale score statistics. We compute the mean and the standard
deviation of the Likert-scale scores across all the motions. For the mean
scores, higher values are better, bold indicates best, and underline indicates
second-best.

| Synthesis type | Plausibility | Synchronization
---|---|---|---
|  | Mean | St. Dev. | Mean | St. Dev.
Set 1 | Original Speaker | 3.25 | 0.90 | 3.10 | 1.34
Ours | 3.27 | 0.86 | 3.15 | 1.32
w/o AC Graphs | 2.61 | 1.14 | 2.48 | 1.38
w/o Disc. | 2.79 | 1.02 | 2.02 | 1.30
Set 2 | Original Speaker | 2.99 | 0.80 | 2.79 | 1.08
Ours | 3.01 | 0.82 | 2.79 | 1.07
w/o Synchronous Synthesis | 2.41 | 0.78 | 1.79 | 0.88

##### Evaluation Process.

Our aim in the user study is to evaluate our synthesized motions on two key
aspects: (i) how plausible they appear to human observers compared to the
motions of the original speakers and the ablated versions, and (ii) whether
synchronous synthesis of face and pose expressions produces perceptible
improvements over asynchronous synthesis. To evaluate plausibility, we ask the
participants to rate each motion in each group in each set on “how natural the
motion looks” on a five-point Likert scale, with the options “very unnatural”
(worst), “not realistic”, “looks OK” (average), “looks good”, and “looks
great” (best). To evaluate the effect of synchronous synthesis, we ask the
participants to observe the face and the pose movements in each motion in each
group in each set and rate them on “how the face and the pose sync with the
speech” on a five-point Likert scale, with the options “no/arbitrary
movements” (worst), “slight movements”, “has movements, but are not
expressive” (average), “somewhat expressive movements”, and “have movements
with appropriate expressions” (best).

##### Results.

Since we randomly select the speech for each of the eight groups of motions
each participant watched, and we also randomized the order of the motions in
each group in each set, we can consider the participants’ responses in each
group to be independent of all the other groups. Thus, we aggregate their
responses to each type of motion across all the groups within a set to obtain
the overall distributions of the Likert-scale scores of the motions for that
set. We show these distributions for each of the two questions on plausibility
and synchronization in each set in Fig. 6. We also report the Likert-scale
score statistics for each type of motion on the two questions in each set in
Table 2. For the purpose of scoring, we assign scores 1 through 5, with 1 for
“worst” and 5 for “best”. We observe that the scores for our synthesized
samples are comparable to the corresponding original speaker motions and
significantly better than the ablated versions. To further affirm this, we
plot the cumulative lower bound of participant responses for each Likert-scale
score for each type of motion in each set in Fig. 7. We note that the scores
for our synchronously synthesized samples remain close to the original speaker
scores and consistently above the other ablated versions, indicating a clear
preference. Overall, in the two sets, $88.89\%$ and $80.00\%$ participants
respectively marked our synchronously synthesized motions 3 or above on the
first question, and $65.46\%$ and $62.87\%$ participants respectively marked 3
or above on the second question. This indicates that the majority of
participants found the motions satisfactory.

Figure 7: Cumulative lower-bound of participant responses. We plot the
cumulative lower-bound (LB) percentage of responses across the Likert-scale
scores for each type of character motion in each set. A cumulative LB
percentage $X$ for a Likert-scale score $s$ denotes $X\%$ of responses had a
score of $s$ or higher. We observe that the curve for our synchronously
synthesized motions stays at the top, indicating that the participants
preferred it over the other motions.

##  8 Conclusion, Limitations and Future Work

We have presented a method to synthesize synchronous co-speech face and pose
expressions for 3D digital characters. Our method learns to synthesize these
expressions from 3D face landmarks and 3D upper-body pose joints computed
directly from videos. Our work also has some limitations. We use sparse face
landmarks and pose joints to synthesize co-speech face and pose expressions.
To synthesize more fine-grained expressions, we plan to extract more detailed
face meshes and additional pose joints from videos. Further, given the
sparsity of our face and pose representations and the noise associated with
extracting them from videos, the quality of our synthesized motions does not
match those synthesized from high-end facial scans and motion-capture data,
and using parameter-dense, compute-heavy methods, such as denoising diffusion.
We aim to bridge this gap by building techniques to develop more robust face
and pose representations from videos. We also plan to combine our work with
lower-body actions such as sitting, standing, and walking to synthesize
3D-animated digital humans in a wider variety of scenarios. In terms of its
running-time cost, our method uses commercial GPUs to obtain real-time
performance. We plan to explore knowledge distillation techniques to reduce
our running-time cost and implement our method in real-time on commodity
devices such as digital personal assistants.

## Acknowledgments

This work was partly supported by ARO Grants W911NF2110026, W911NF2310046,
W911NF2310352, and Army Cooperative Agreement W911NF2120076.

## References

  * Ahuja and Morency [2019] C. Ahuja and L. Morency.  Language2pose: Natural language grounded pose forecasting.  In _2019 International Conference on 3D Vision (3DV)_ , pages 719–728, 2019.
  * Ambady and Rosenthal [1992] Nalini Ambady and Robert Rosenthal.  Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.  _Psychological bulletin_ , 111(2):256, 1992.
  * Ao et al. [2022] Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu.  Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings.  _ACM Trans. Graph._ , 41(6), 2022.
  * Ao et al. [2023] Tenglong Ao, Zeyi Zhang, and Libin Liu.  Gesturediffuclip: Gesture diffusion model with clip latents.  _ACM Trans. Graph._ , 2023.
  * Aristidou and Lasenby [2011] Andreas Aristidou and Joan Lasenby.  Fabrik: A fast, iterative solver for the inverse kinematics problem.  _Graphical Models_ , 73(5):243–260, 2011.
  * Banerjee et al. [2022] Abhishek Banerjee, Uttaran Bhattacharya, and Aniket Bera.  Learning unseen emotions from gestures via semantically-conditioned zero-shot perception with adversarial autoencoders.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(1):3–10, 2022.
  * Baur et al. [2013] T. Baur, I. Damian, P. Gebhard, K. Porayska-Pomsta, and E. André.  A job interview simulation: Social cue-based interaction with a virtual character.  In _2013 International Conference on Social Computing_ , pages 220–227, 2013.
  * Bhattacharya et al. [2020a] Uttaran Bhattacharya, Trisha Mittal, Rohan Chandra, Tanmay Randhavane, Aniket Bera, and Dinesh Manocha.  Step: Spatial temporal graph convolutional networks for emotion perception from gaits.  In _Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence_ , page 1342–1350. AAAI Press, 2020a.
  * Bhattacharya et al. [2020b] Uttaran Bhattacharya, Christian Roncal, Trisha Mittal, Rohan Chandra, Kyra Kapsaskis, Kurt Gray, Aniket Bera, and Dinesh Manocha.  Take an emotion walk: Perceiving emotions from gaits using hierarchical attention pooling and affective mapping.  In _Computer Vision – ECCV 2020_ , pages 145–163, Cham, 2020b. Springer International Publishing.
  * Bhattacharya et al. [2021a] Uttaran Bhattacharya, Elizabeth Childs, Nicholas Rewkowski, and Dinesh Manocha.  _Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning_ , page 2027–2036.  Association for Computing Machinery, New York, NY, USA, 2021a.
  * Bhattacharya et al. [2021b] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In _2021 IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR)_. IEEE, 2021b.
  * Busso et al. [2008] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan.  Iemocap: Interactive emotional dyadic motion capture database.  _Language resources and evaluation_ , 42(4):335–359, 2008.
  * Cai et al. [2019] Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Magnenat Thalmann.  Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Cudeiro et al. [2019] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J. Black.  Capture, learning, and synthesis of 3d speaking styles.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Dabral et al. [2023] Rishabh Dabral, Muhammad Hamza Mughal, Vladislav Golyanik, and Christian Theobalt.  Mofusion: A framework for denoising-diffusion-based motion synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9760–9770, 2023.
  * DeVault et al. [2014] David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al.  Simsensei kiosk: A virtual human interviewer for healthcare decision support.  In _Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems_ , pages 1061–1068, 2014.
  * Ekman [1992] Paul Ekman.  Are there basic emotions?  1992\.
  * Ferstl et al. [2019] Ylva Ferstl, Michael Neff, and Rachel McDonnell.  Multi-objective adversarial gesture generation.  In _Motion, Interaction and Games_ , New York, NY, USA, 2019. Association for Computing Machinery.
  * Ghosh et al. [2021] Anindita Ghosh, Noshaba Cheema, Cennet Oguz, Christian Theobalt, and Philipp Slusallek.  Synthesis of compositional animations from textual descriptions.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1396–1406, 2021.
  * Ghosh et al. [2023a] Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek.  Imos: Intent-driven full-body motion synthesis for human-object interactions.  _Computer Graphics Forum_ , 42(2):1–12, 2023a.
  * Ghosh et al. [2023b] Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek.  Remos: Reactive 3d motion synthesis for two-person interactions.  _arXiv preprint arXiv:2311.17057_ , 2023b.
  * Giannopoulos et al. [2018] Panagiotis Giannopoulos, Isidoros Perikos, and Ioannis Hatzilygeroudis.  _Deep Learning Approaches for Facial Emotion Recognition: A Case Study on FER-2013_ , pages 1–16.  Springer International Publishing, Cham, 2018.
  * Ginosar et al. [2019] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik.  Learning individual styles of conversational gesture.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2014.
  * Greenwood et al. [2017] David Greenwood, Stephen Laycock, and Iain Matthews.  Predicting head pose from speech with a conditional variational autoencoder.  ISCA, 2017.
  * Habibie et al. [2021] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt.  Learning speech-driven 3d conversational gestures from video.  In _Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents_ , page 101–108, New York, NY, USA, 2021. Association for Computing Machinery.
  * Habibie et al. [2022] Ikhsanul Habibie, Mohamed Elgharib, Kripasindhu Sarkar, Ahsan Abdullah, Simbarashe Nyatsanga, Michael Neff, and Christian Theobalt.  A motion matching-based framework for controllable gesture synthesis from speech.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , New York, NY, USA, 2022. Association for Computing Machinery.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen.  Audio-driven facial animation by joint end-to-end learning of pose and emotion.  _ACM Trans. Graph._ , 36(4), 2017.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kucherenko et al. [2020] Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellström.  Gesticulator: A framework for semantically-aware speech-driven gesture generation.  page 242–250, New York, NY, USA, 2020. Association for Computing Machinery.
  * Laban and Ullmann [1971] Rudolf Laban and Lisa Ullmann.  The mastery of movement.  1971\.
  * Lahiri et al. [2021] Avisek Lahiri, Vivek Kwatra, Christian Frueh, John Lewis, and Chris Bregler.  Lipsync3d: Data-efficient learning of personalized 3d talking faces from video using pose and lighting normalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2755–2764, 2021.
  * Li et al. [2016] Jamy Li, René Kizilcec, Jeremy Bailenson, and Wendy Ju.  Social robots and virtual agents as lecturers for video instruction.  _Computers in Human Behavior_ , 55:1222 – 1230, 2016.
  * Li et al. [2021] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao.  Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11293–11302, 2021.
  * Liao et al. [2019] M. Liao, C. Sung, H. Wang, and W. Lin.  Virtual classmates: Embodying historical learners’ messages as learning companions in a vr classroom through comment mapping.  In _2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)_ , pages 163–171, 2019.
  * Liu et al. [2022a] Haiyang Liu, Naoya Iwamoto, Zihao Zhu, Zhengqing Li, You Zhou, Elif Bozkurt, and Bo Zheng.  Disco: Disentangled implicit content and rhythm learning for diverse co-speech gestures synthesis.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , page 3764–3773, New York, NY, USA, 2022a. Association for Computing Machinery.
  * Liu et al. [2022b] Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu.  Audio-driven co-speech gesture video generation.  In _Advances in Neural Information Processing Systems_ , pages 21386–21399. Curran Associates, Inc., 2022b.
  * Liu et al. [2022c] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou.  Learning hierarchical cross-modal association for co-speech gesture generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10462–10472, 2022c.
  * Mascarenhas et al. [2018] S. Mascarenhas, M. Guimarães, R. Prada, J. Dias, P. A. Santos, K. Star, B. Hirsh, E. Spice, and R. Kommeren.  A virtual agent toolkit for serious games developers.  In _2018 IEEE Conference on Computational Intelligence and Games (CIG)_ , pages 1–7, 2018.
  * Mehrabian and Russell [1974] Albert Mehrabian and James A Russell.  _An approach to environmental psychology._ the MIT Press, 1974.
  * Mesquita and Boiger [2014] Batja Mesquita and Michael Boiger.  Emotions in context: A sociodynamic model of emotions.  _Emotion Review_ , 6(4):298–302, 2014.
  * Mittal et al. [2020] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.  In _Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence_ , pages 1359–1367. AAAI Press, 2020.
  * Neiberg et al. [2006] Daniel Neiberg, Kjell Elenius, and Kornel Laskowski.  Emotion recognition in spontaneous speech using gmms.  In _Ninth international conference on spoken language processing_ , 2006.
  * Omniverse [2021] NVIDIA Omniverse.  _NVIDIA Omniverse, https://www.nvidia.com/en-us/omniverse/_ , 2021.
  * Parkinson et al. [2005] Brian Parkinson, Agneta H Fischer, and Antony SR Manstead.  _Emotion in social relations: Cultural, group, and interpersonal processes_.  Psychology press, 2005.
  * Qian et al. [2021] Shenhan Qian, Zhi Tu, Yihao Zhi, Wen Liu, and Shenghua Gao.  Speech drives templates: Co-speech gesture synthesis with learned templates.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11077–11086, 2021.
  * Richard et al. [2021] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando de la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1173–1182, 2021.
  * Sadoughi and Busso [2018] N. Sadoughi and C. Busso.  Novel realizations of speech-driven head movements with generative adversarial networks.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6169–6173, 2018.
  * Santos et al. [2017] I. Santos, N. Nedjah, and L. de Macedo Mourelle.  Sentiment analysis using convolutional neural network with fasttext embeddings.  In _2017 IEEE Latin American Conference on Computational Intelligence (LA-CCI)_ , pages 1–5, 2017.
  * Shafir et al. [2024] Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano.  Human motion diffusion as a generative prior.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Simeone et al. [2019] Adalberto L Simeone, Marco Speicher, Andreea Molnar, Adriana Wilde, and Florian Daiber.  Live: The human role in learning in immersive virtual environments.  In _Symposium on Spatial User Interaction_ , New York, NY, USA, 2019. Association for Computing Machinery.
  * Sinha et al. [2022] Sanjana Sinha, Sandika Biswas, Ravindra Yadav, and Brojeshwar Bhowmick.  Emotion-controllable generalized talking face generation.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 1320–1327. International Joint Conferences on Artificial Intelligence Organization, 2022.  Main Track.
  * Soleymani et al. [2012] Mohammad Soleymani, Maja Pantic, and Thierry Pun.  Multimodal emotion recognition in response to videos.  _IEEE Transactions on Affective Computing_ , 3(2):211–223, 2012.
  * Taheri et al. [2024] Omid Taheri, Yi Zhou, Dimitrios Tzionas, Yang Zhou, Duygu Ceylan, Soren Pirk, and Michael J Black.  Grip: Generating interaction poses using spatial cues and latent consistency.  In _International conference on 3D vision (3DV)_ , 2024.
  * Umeyama [1991] Shinji Umeyama.  Least-squares estimation of transformation parameters between two point patterns.  _TPAMI_ , pages 376–380, 1991.
  * Wang and Soong [2015] Lijuan Wang and Frank K Soong.  Hmm trajectory-guided sample selection for photo-realistic talking head.  _Multimedia Tools and Applications_ , 74(22):9849–9869, 2015.
  * Watson et al. [2019] Katie Watson, Samuel S. Sohn, Sasha Schriber, Markus Gross, Carlos Manuel Muniz, and Mubbasir Kapadia.  Storyprint: An interactive visualization of stories.  In _Proceedings of the 24th International Conference on Intelligent User Interfaces_ , page 303–311, New York, NY, USA, 2019. Association for Computing Machinery.
  * Yi et al. [2023] Hongwei Yi, Hualin Liang, Yifei Liu, Qiong Cao, Yandong Wen, Timo Bolkart, Dacheng Tao, and Michael J. Black.  Generating holistic 3d human motion from speech.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 469–480, 2023.
  * Yoon et al. [2019] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In _Proc. of The International Conference in Robotics and Automation (ICRA)_ , 2019.
  * Yoon et al. [2020] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  _ACM Transactions on Graphics_ , 39(6), 2020.
  * Zhang et al. [2016] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE Signal Processing Letters_ , 23(10):1499–1503, 2016.
  * Zhang et al. [2024] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu.  Motiondiffuse: Text-driven human motion generation with diffusion model.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–15, 2024.
  * Zhou et al. [2018] Yang Zhou, Zhan Xu, Chris Landreth, Evangelos Kalogerakis, Subhransu Maji, and Karan Singh.  Visemenet: Audio-driven animator-centric speech animation.  _ACM Trans. Graph._ , 37(4), 2018.