# Audio-Driven Co-Speech Gesture Video Generation

Xian Liu1, Qianyi Wu2, Hang Zhou1, Yuanqi Du3, Wayne Wu4, Dahua Lin1,4, Ziwei
Liu5
1Multimedia Laboratory, The Chinese University of Hong Kong 2Monash University
3Cornell University 4Shanghai AI Laboratory 5S-Lab, Nanyang Technological
University

###### Abstract

Co-speech gesture is crucial for human-machine interaction and digital
entertainment. While previous works mostly map speech audio to human skeletons
(e.g., 2D keypoints), directly generating speakers’ gestures in the image
domain remains unsolved. In this work, we formally define and study this
challenging problem of _audio-driven co-speech gesture video generation_ ,
i.e., using a _unified_ framework to generate speaker _image sequence_ driven
by speech audio. Our key insight is that the co-speech gestures can be
decomposed into common motion patterns and subtle rhythmic dynamics. To this
end, we propose a novel framework, Audio-driveN Gesture vIdeo gEneration
(ANGIE), to effectively capture the reusable co-speech gesture patterns as
well as fine-grained rhythmic movements. To achieve high-fidelity image
sequence generation, we leverage an unsupervised motion representation instead
of a structural human body prior (e.g., 2D skeletons). Specifically, 1) we
propose a vector quantized motion extractor (VQ-Motion Extractor) to summarize
common co-speech gesture patterns from implicit motion representation to
codebooks. 2) Moreover, a co-speech gesture GPT with motion refinement (Co-
Speech GPT) is devised to complement the subtle prosodic motion details.
Extensive experiments demonstrate that our framework renders realistic and
vivid co-speech gesture video. Demo video and more resources can be found in:
https://alvinliu0.github.io/projects/ANGIE

##  1 Introduction

During daily conversation among humans, speakers naturally emit co-speech
gestures to complement the verbal channels and express their thoughts [18, 37,
58]. Such non-verbal behaviors ease speech comprehension [11, 60] and bridge
the communicator’s gap for better credibility [8, 56]. Therefore, equipping
the social robot with conversation skills constitutes a crucial step to human-
machine interaction. To achieve it, researchers delve into the task of co-
speech gesture generation [22, 41, 65], where audio-coherent human gesture
sequences are synthesized in the form of structural human representation
(e.g., skeletons). However, such representation contains no appearance
information of the target speaker, which is crucial for human perception. As
demonstrated in audio-driven talking head synthesis [35, 69], generating real-
world subjects in the image domain is highly desirable. To this end, we
explore the problem of _audio-driven co-speech gesture video generation_ ,
i.e., using a _unified_ framework to generate speaker _image sequence_ driven
by speech audio (illustrated in Fig. 1).

Conventional methods require exhaustive human efforts to pre-define the
speech-gesture pairs and connection rules for coherent result [12, 13]. With
the development of deep learning, neural networks are leveraged to learn the
mapping from encoded audio feature to human skeletons in a data-driven manner
[22, 41, 65]. Notably, one category of approaches relies on small-scale MoCap
datasets in co-speech setting [17, 19, 50], which contributes to specific
models with limited capacity and robustness. To capture more general speech-
gesture correlations, another category of methods builds large training corpus
by exploiting off-the-shelf pose estimators [10, 16] to label enormous online
videos as pseudo ground truth [22, 66]. However, the inaccurate pose
annotations induce error accumulation in the training phase, which makes the
generated results unnatural. Besides, most previous works ignore the problem
of co-speech gesture video generation. Only few works [22, 41] animate in the
image domain as an _independent post-processing step_ , which borrows from the
existing pose-to-image generators [6, 14] to train on the target person’s
images. How to design a _unified_ framework to generate speaker _image
sequence_ driven by speech audio remains unsolved.

To effectively learn the mapping from audio to co-speech gesture video, we
pinpoint two important observations from current studies: 1) hand-crafted
structural human priors like 2D/3D skeletons would eliminate articulated human
body region information. Such a zeroth-order motion representation fails to
formulate first-order motion like local affine transformation in image
animation [46]. Besides, the error in structural prior labeling impairs cross-
modal audio-to-gesture learning [34]. 2) Motivated by previous linguistic
studies [28, 49], the co-speech gestures could be decomposed into common
motion patterns and rhythmic dynamics, where the former ones refer to large-
scale motion templates (e.g., periodically put hands up and down), while the
latter ones play a refinement role to complement subtle prosodic movements and
synchronize with speech audio (e.g., finger flickers).

We take inspiration from the above observations and propose a novel framework
Audio-driveN Gesture vIdeo gEneration (ANGIE) to generate co-speech gesture
video. The key insight is to _summarize common co-speech gesture patterns from
motion representation to quantized codebooks_ and further _refine subtle
rhythmic details by motion residuals for fine-grained results_. In particular,
two modules are designed, namely VQ-Motion Extractor and Co-Speech GPT. In VQ-
Motion Extractor, we utilize an unsupervised motion representation to depict
the articulated human body and first-order gestures [47]. The codebooks are
established to quantize the reusable common co-speech gesture patterns from
unsupervised motion representation. To guarantee the validity of gesture
patterns, we propose a cholesky decomposition based quantization scheme to
relax the motion component constraint. The position-irrelevant motion pattern
is extracted as final quantization target to represent the relative motion. In
this way, the quantized codebooks naturally contain rich common gesture
pattern information. With the quantized motion code sequence, in Co-Speech GPT
we use a GPT-like [42] structure to predict discrete motion patterns from
speech audio. Finally, a motion refinement network is used to complement
subtle rhythmic details for fine-grained results.

To summarize, our main contributions are three-fold: 1) We explore a
challenging problem of audio-driven co-speech gesture _video_ generation. To
the best of our knowledge, we are the first to generate co-speech gesture in
image domain with a _unified_ framework _without any structural human body
prior_. 2) We propose the VQ-Motion Extractor to quantize the motion
representation into common gesture patterns and the Co-Speech GPT to refine
subtle rhythmic movement details. The codebooks naturally contain reusable
motion pattern information. 3) Extensive experiments demonstrate that the
proposed framework ANGIE renders realistic and vivid co-speech gesture video
generation results.

##  2 Related Work

Figure 1: Illustration of Problem Setting. In this paper, we focus on audio-
driven co-speech gesture video generation. Given an image with speech audio,
we generate aligned speaker _image sequence_.

Co-Speech Gesture Generation. Synthesizing co-speech gesture has gained
research interest in vision [3, 22, 30], graphics [4, 63, 65] and robotics
[24, 26, 66] domains. Recent researches resort to deep neural networks to
learn the speech-gesture mapping in a data-driven manner, with major focuses
on below perspectives: 1) Dataset. One strand of methods use small-scale MoCap
datasets to learn specific models [17, 19, 31, 44, 50, 53, 57], while another
strand of works exploit off-the-shelf estimator to label enormous videos as
structural prior [1, 2, 3, 22, 34, 41, 64, 65, 66]. The dataset scale _v.s._
pose annotation accuracy often acts as a trade-off in this task: A large
amount of speech-gesture pairs facilitate the training of more general models
with better capacity and robustness, yet error accumulation in annotations
induces unnatural results. 2) Framework architecture. CNN-based [22], RNN-
based [65] and Transformer-based [7] frameworks show promising results. To
further improve the diversity of generated gestures and grasp the fine-grained
cross-modal associations, components like adversarial loss [22], VAE sampling
[31, 62] and hierarchical encoder-decoder design [34] are proposed. 3) Input
modality. Some approaches treat single modality of speech audio [2, 20, 22,
23, 24, 31, 41] or text transcription [3, 7, 26, 66] as input to drive the co-
speech gesture, while some others use both modalities as stimuli for
generation [1, 34, 65]. To ease the learning of implicit cross-modal mapping
from speech to gesture and create more stable results, recent works involve
auxiliary input modality such as speaker style [2], pose mode [62] and motion
template [41].

In this work, we take a step further in the above three aspects: 1) For the
dataset, we collect a new co-speech gesture dataset in _image_ domain, where
an unsupervised motion representation is used to model articulated human body
and bypass the inaccuracy from structural prior annotations. 2) For the
architecture, a vector quantized (VQ) network with novel discretization scheme
is proposed to extract valid relative motion patterns. We further devise a
motion refinement network to complement subtle rhythmic dynamics. 3) For the
input modality, we explicitly decouple the common motion pattern from co-
speech gestures, which serves a similar role as motion template [41] to
provide auxiliary input. However, our discrete codebook design is more
suitable for finite gesture patterns than continuous representation, which is
also proven in recent cross-modal generation tasks [38, 43, 48]. Furthermore,
we propose a novel vector quantization network with cholesky decomposition
scheme to extract the valid motion patterns. We improve the quantization
scheme to encode the relative motion representation that is position (absolute
location) irrelevant. A motion refinement network is further devised to
complement subtle rhythmic dynamics. Notably, our approach gives an idea on
how to deal with the constraints in vector quantization and how to complement
sequential results with missing details. Such design could prospectively
provide insights for relevant domains like constrained vector quantization
problem, cross-modal learning [33] and video generation tasks [48].

Figure 2: Overview of the Audio-driveN Gesture vIdeo gEneration (ANGIE)
framework. In VQ-Motion Extractor, the cholesky decomposition with position-
irrelevant design transforms the shift-translation $\mu$ and covariance $C$ to
relative motion pattern representation of $\Delta\mu$ and $\Delta L$, which
are further quantized by codebooks to extract the common motion patterns.
Given the driving audio and starting gesture codes, the Co-Speech GPT predicts
the future motion fields. A Motion Refinement network further learns motion
residuals to complement the subtle rhythmic dynamics.

Video/Audio-Driven Video Generation. Traditional video-driven approaches for
image animation can be categorized into supervised and unsupervised, where the
supervised methods typically involve structural human body prior such as
landmarks [9, 68] and 3D parametric models [21, 52], while the unsupervised
approaches design self-supervised tasks to animate unlabeled images [45, 46,
47, 59]. To facilitate broader applications, researchers explore audio-driven
video generation, where one of the most relevant tasks is talking face
generation [15, 40]. Different from the strong correlations between audio and
mouth shape, the mapping from audio to complicated co-speech motion is multi-
modal and harder to learn. Most co-speech gesture studies synthesize human
skeletons as final results (e.g., 2D keypoints), while only few works [22, 41]
generate co-speech images in a _post-processing_ manner.

##  3 Our Approach

We present ANGIE that generates audio-driven co-speech gesture in image
domain, where the speakers’ image sequence is driven by speech audio as shown
in Fig. 1. The whole pipeline is illustrated in Fig. 2. To make the content
self-contained and narration clearer, we first introduce the preliminaries and
problem setting in Sec. 3.1. Then, we present the VQ-Motion Extractor which
extracts common co-speech gesture patterns as quantized codebooks in Sec. 3.2.
Finally, we elaborate the Co-Speech GPT to complement subtle rhythmic dynamics
for fine-grained results in Sec. 3.3.

###  3.1 Preliminaries and Problem Setting

Unsupervised Motion Representation. To achieve high-fidelity image animation,
we take inspiration from MRAA [47] that uses an unsupervised motion
representation to drive articulated objects. MRAA first estimates a coarse
motion representation from the source and driving frames, then predicts dense
pixel-wise flow for image generation. Specifically, an encoder-decoder
keypoint predictor produces $K$ different heatmaps
$\mathbf{H}^{1},\mathbf{H}^{2},\dots,\mathbf{H}^{K}$, where $K$ is region
number and $\mathbf{H}^{k}$ denotes the $k$-th image region. Afterwards, each
heatmap is normalized by softmax operation, _i.e_ ,
$\sum_{z\in\mathcal{Z}}\mathbf{H}^{k}(z)=1$, where $z$ is the image pixel
location and $\mathcal{Z}$ is the set of all pixels. The key insight behind
MRAA is to represent each region’s motion by affine transformation with a
shift-translation component. The shift-translation component
$\mu^{k}\in\mathbb{R}^{2}$ and the distribution $C^{k}$ of the $k$-th part can
be calculated as:

$\displaystyle\mu^{k}=\sum_{z\in\mathcal{Z}}\mathbf{H}^{k}(z)z,\quad
C^{k}=\sum_{z\in\mathcal{Z}}\mathbf{H}^{k}(z)(z-\mu^{k})(z-\mu^{k})^{\mathsf{T}},$
(1)

where $\mathsf{T}$ is matrix transpose, $C^{k}\in\mathbb{R}^{2\times 2}$
measures the covariance of heatmap value. It naturally depicts the size and
shape of an articulated region. To represent the affine transformation
$A^{k}\in\mathbb{R}^{2\times 2}$ of the $k$-th region, we apply singular value
decomposition (SVD) to $C^{k}$ and derive $A^{k}$ as:

$\displaystyle C^{k}=U^{k}\Sigma^{k}(V^{k})^{\mathsf{T}},\quad
A^{k}=U^{k}{\Sigma^{k}}^{\frac{1}{2}},$ (2)

where unitary matrices $U^{k}$, $V^{k}$ and diagonal matrix $\Sigma^{k}$ are
the SVD result of covariance matrix $C^{k}$. The representation $\mathcal{M}$
extracted by motion estimator is the concatenation of
$\left[\mu;C;A\right]\in\mathbb{R}^{K\times(2+4+4)}$ for $K$ distinct regions,
an image generation module with dense pixel-wise flow predictor synthesizes
the final generation results. In this work, the motion representation
$\mathcal{M}$ and image generation module $G$ generally follow MRAA. We
suggest the readers referring to [47] for more details.

Problem Setting for Co-Speech Gesture Image Generation. We collect training
data of massively-available speaking videos with clear co-speech gestures for
natural self-reconstruction supervision. Specifically, given an $(N+1)$-frame
video clip $\mathbf{V}=\\{I_{(0)},\dots,I_{(N)}\\}$, the goal of our framework
at the training stage is to predict the motion representation
$\widehat{\mathcal{M}}_{(1:N)}$ based on the first image frame $I_{(0)}$ and
video’s accompanying audio sequence $\mathbf{a}=\\{a_{(1)},\dots,a_{(N)}\\}$.
Further, the image generation module $G$ reconstructs the video frames
$\widehat{I}_{(1:N)}$. At the inference stage, an arbitrary reference image
with speech audio clip is provided to generate subsequent image frames.
According to the observation in Sec. 1, we decompose the co-speech gestures
into common motion patterns and subtle rhythmic dynamics. The overall training
setting can be formulated as:

$\widehat{I}_{(1:N)}=G(I_{(0)},\widehat{\mathcal{M}}_{(1:N)}(\mathbf{a})),\
\quad\ \
\widehat{\mathcal{M}}_{(1:N)}=\widehat{\mathcal{M}}^{\textit{pattern}}_{(1:N)}+\widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:N)},$
(3)

where $\mathcal{M}^{\textit{pattern}}$ denotes the gesture pattern (Sec. 3.2)
and $\mathcal{M}^{\textit{rhythmic}}$ is rhythmic movement (Sec. 3.3).

###  3.2 Vector Quantized Motion Pattern Extractor

To decompose the co-speech gestures, we propose to firstly extract the common
motion patterns. However, three problems remain: 1) The gesture sequences are
different from each other. While some motion sequences share the same action
pattern, the dynamic details may vary a lot. How to extract the major motion
pattern despite the influence from minor prosodic movements? 2) The covariance
matrix $C$ is symmetric positive definite (Eq. 1), which further constrains
the range of affine matrix $A$. How to preserve such characteristic for valid
gesture patterns? 3) Since the unsupervised motion representation is extracted
in the image pixel space, it is affected by the absolute location of each
articulated region. How to represent the position-irrelevant motion pattern
information?

Vector Quantized Motion Pattern Learning. Our solution to the first problem is
to quantize the common motion pattern into a codebook. Since the gesture
pattern is finite, it could be summarized to discrete codebook entries.
Besides, each codebook entry refers to a certain type of gesture pattern,
which matches our goal to extract the common and reusable co-speech gesture
patterns.

A _naive_ way is to quantize the motion representation $\mathcal{M}$
separately as shift-translation $\mu$, covariance matrix $C$ and affine
transformation $A$. Specifically, for a $T$-frame co-speech gesture sequence
$I_{(1:T)}$, we transform it into
$\left[\mu;C;A\right]_{(1:T)}\in\mathbb{R}^{T\times K\times(2+4+4)}$, where
$\left[\mu;C;A\right]=\mathcal{M}$ denotes motion representation of $K$
regions as in Sec. 3.1. We first build three codebooks
$\mathcal{D}_{\mu}=\\{\mathbf{d}_{\mu,m}\\}_{m=1}^{M}$,
$\mathcal{D}_{C}=\\{\mathbf{d}_{C,m}\\}_{m=1}^{M}$ and
$\mathcal{D}_{A}=\\{\mathbf{d}_{A,m}\\}_{m=1}^{M}$ for each motion component
respectively, where $M$ is codebook size. $\mathbf{d}_{\mu,m}$,
$\mathbf{d}_{C,m}$ and $\mathbf{d}_{A,m}\in\mathbb{R}^{\ell}$ are the $m$-th
entry of $\ell$-channel for each codebook. Then, three separate encoders
$E_{\mu}$, $E_{C}$ and $E_{A}$ are utilized to encode the corresponding
context information into latent features of
$\mathbf{e}_{\mu}=\\{\mathbf{e}_{\mu,i}\\}_{i=1}^{T^{\prime}}$,
$\mathbf{e}_{C}=\\{\mathbf{e}_{C,i}\\}_{i=1}^{T^{\prime}}$ and
$\mathbf{e}_{A}=\\{\mathbf{e}_{A,i}\\}_{i=1}^{T^{\prime}}\in\mathbb{R}^{T^{\prime}\times\ell}$,
where $T^{\prime}$ is the temporal dimension and $\ell$ is the channel
dimension. Notably, we denote the $i$-th temporal feature of each motion
component as $\mathbf{e}_{\mu,i}$, $\mathbf{e}_{C,i}$ and $\mathbf{e}_{A,i}$.
The feature encoding process can be formulated as:

$\displaystyle E_{\mu}(\mu_{(1:T)})=\mathbf{e}_{\mu},\quad
E_{C}(C_{(1:T)})=\mathbf{e}_{C},\quad E_{A}(A_{(1:T)})=\mathbf{e}_{A}.$ (4)

Following the pipeline of VQ-VAE [55], we individually quantize
$\mathbf{e}_{\mu}$, $\mathbf{e}_{C}$ and $\mathbf{e}_{A}$ by substituting each
temporal feature $\mathbf{e}_{\mu,i}$, $\mathbf{e}_{C,i}$ and
$\mathbf{e}_{A,i}$ to the nearest codebook entry $\mathbf{d}_{\mu,m}$,
$\mathbf{d}_{C,m}$ and $\mathbf{d}_{A,m}$ as:

$\displaystyle\underbrace{\mathbf{e}^{\mathbf{q}}_{\mu}=\mathop{\arg\min}_{\mathbf{d}_{\mu}\in\mathcal{D}_{\mu}}||\mathbf{e}_{\mu}-\mathbf{d}_{\mu}||}_{\textit{quantize
shift-translation }\mu},\ \
\underbrace{\mathbf{e}^{\mathbf{q}}_{C}=\mathop{\arg\min}_{\mathbf{d}_{C}\in\mathcal{D}_{C}}||\mathbf{e}_{C}-\mathbf{d}_{C}||}_{\textit{quantize
covariance matrix }C},\ \
\underbrace{\mathbf{e}^{\mathbf{q}}_{A}=\mathop{\arg\min}_{\mathbf{d}_{A}\in\mathcal{D}_{A}}||\mathbf{e}_{A}-\mathbf{d}_{A}||}_{\textit{quantize
affine transformation }A},$ (5)

where
$\mathbf{e}^{\mathbf{q}}_{\mu}=\\{\mathbf{e}^{\mathbf{q}}_{\mu,i}\\}_{i=1}^{T^{\prime}}$,
$\mathbf{e}^{\mathbf{q}}_{C}=\\{\mathbf{e}^{\mathbf{q}}_{C,i}\\}_{i=1}^{T^{\prime}}$
and
$\mathbf{e}^{\mathbf{q}}_{A}=\\{\mathbf{e}^{\mathbf{q}}_{A,i}\\}_{i=1}^{T^{\prime}}\in\mathbb{R}^{T^{\prime}\times\ell}$
are the quantized code sequence of length $T^{\prime}$ for each motion
component. The $i$-th quantized code of each motion component is denoted as
$\mathbf{e}^{\mathbf{q}}_{\mu,i}$, $\mathbf{e}^{\mathbf{q}}_{C,i}$ and
$\mathbf{e}^{\mathbf{q}}_{A,i}$, respectively. Finally, three separate
decoders $D_{\mu}$, $D_{C}$ and $D_{A}$ are leveraged to reconstruct the
motion representations of each component as:

$\displaystyle\widehat{\mu}_{(1:T)}=D_{\mu}(\mathbf{e}^{\mathbf{q}}_{\mu}),\quad\widehat{C}_{(1:T)}=D_{C}(\mathbf{e}^{\mathbf{q}}_{C}),\quad\widehat{A}_{(1:T)}=D_{A}(\mathbf{e}^{\mathbf{q}}_{A}).$
(6)

Such discrete representation also ease the audio-to-gesture learning (Sec.
3.3): Previous methods predict continuous output as a _harder regression_
problem. While we only need to predict features nearer to the correct codebook
entry, which in essence resembles an _easier classification_ problem.

Quantization Design for Valid Motion Representation. To extract valid gesture
patterns, we have to preserve certain characteristics of motion
representation. Especially, the covariance matrix $C$ should be symmetric
positive definite (Eq. 1), and the affine transformation $A$ is determined by
$C$ through SVD (Eq. 2). Therefore, instead of naively quantize each component
in Eq. 5, we propose to only quantize the shift-translation $\mu$ and
covariance matrix $C$, while derive the affine transformation $A$ with SVD.
The only constraint is to guarantee that the covariance matrix $C$ is
symmetric positive definite. To satisfy such requirement, we use the _unique
cholesky decomposition theorem_ [54]:

######  Theorem 1.

For any real symmetric positive definite matrix $C\in\mathbb{S}^{n}_{++}$,
there exists a unique lower triangular matrix $L$ with positive diagonal
entries, such that $C=LL^{\mathsf{T}}$. ∎

In this way, we turn to quantize the lower triangular matrix
$L=\begin{pmatrix}l_{1}&0\\\ l_{2}&l_{3}\end{pmatrix}$, where the constraint
is much simpler as $l_{1},l_{3}>0$. The updated quantization scheme with
cholesky decomposition is:

$\displaystyle\underbrace{\mathbf{e}^{\mathbf{q}}_{\mu}=\mathop{\arg\min}_{\mathbf{d}_{\mu}\in\mathcal{D}_{\mu}}||\mathbf{e}_{\mu}-\mathbf{d}_{\mu}||}_{\textit{quantize
shift-translation }\mu},\ \
\underbrace{\mathbf{e}^{\mathbf{q}}_{L}=\mathop{\arg\min}_{\mathbf{d}_{L}\in\mathcal{D}_{L}}||\mathbf{e}_{L}-\mathbf{d}_{L}||}_{\textit{quantize
the lower triangular matrix }L},$ (7)

where $\mathbf{e}_{L}$, $\mathbf{e}^{\mathbf{q}}_{L}$, $\mathcal{D}_{L}$ and
$\mathbf{d}_{L}$ denote the encoded feature, quantized feature, codebook and
codebook entry for factorial covariance $L$, respectively. A simple
transformation of $l_{1,3}=\texttt{ReLU}(l_{1,3})+\epsilon$ guarantees the
diagonal entries to be positive, where $\epsilon$ is a small positive number.
The motion component $C$ and $A$ can be further obtained by $LL^{\mathsf{T}}$
and SVD calculation, respectively.

Position-Irrelevant Motion Pattern. Another problem arises when we inspect the
value of the motion representation: Since $\mathcal{M}$ is extracted in the
image pixel space, the object location will affect the element in
$\mathcal{M}$. For example, if a person poses the same gesture at different
image regions, the motion component differs yet the underlying motion pattern
remains the same. Thus we focus on a image location invariant motion pattern
representation. In particular, due to the linear additiveness, the relative
shift-translation $\mu$ between adjacent frames can be represented as
$\Delta\mu_{j}=\mu_{j}-\mu_{j-1}$, and the relative change of the lower
triangular matrix is $\Delta L_{j}=L_{j}-L_{j-1}$ for any $j=2,\dots,N$. Note
that with the uniqueness of cholesky decomposition, $(L+\Delta L)$ corresponds
to the sole covariance matrix $C=(L+\Delta L)(L+\Delta L)^{\mathsf{T}}$, which
further determines affine matrix $A$ by SVD. In this way, the term $\Delta L$
is sufficient to represent any relative affine transformation between two
frames. We accordingly update the quantization scheme with position-irrelevant
motion pattern representation as:

$\displaystyle\underbrace{\mathbf{e}^{\mathbf{q}}_{\Delta\mu}=\mathop{\arg\min}_{\mathbf{d}_{\Delta\mu}\in\mathcal{D}_{\Delta\mu}}||\mathbf{e}_{\Delta\mu}-\mathbf{d}_{\Delta\mu}||}_{\textit{quantize
relative shift-translation }\Delta\mu},\ \
\underbrace{\mathbf{e}^{\mathbf{q}}_{\Delta
L}=\mathop{\arg\min}_{\mathbf{d}_{\Delta L}\in\mathcal{D}_{\Delta
L}}||\mathbf{e}_{\Delta L}-\mathbf{d}_{\Delta L}||}_{\textit{quantize relative
lower triangular matrix change }\Delta L},$ (8)

where $\mathbf{e}_{\\{\Delta\mu,\Delta L\\}}$,
$\mathbf{e}^{\mathbf{q}}_{\\{\Delta\mu,\Delta L\\}}$,
$\mathcal{D}_{\\{\Delta\mu,\Delta L\\}}$ and $\mathbf{d}_{\\{\Delta\mu,\Delta
L\\}}$ are the encoded feature, quantized feature, codebook and entry for the
relative shift-translation $\Delta\mu$ and factorial covariance $\Delta L$,
respectively.

Overall Quantized Motion Pattern Learning. With the position-irrelevant motion
pattern, the codebook naturally contains reusable common co-speech gesture
patterns $\mathcal{M}^{\textit{pattern}}$. The encoders $E_{\Delta\mu}$,
$E_{\Delta L}$ and the decoders $D_{\Delta\mu}$, $D_{\Delta\mu}$ are jointly
learned with the codebooks $\mathcal{D}_{\Delta\mu}$ and $\mathcal{D}_{\Delta
L}$ via:

$\displaystyle\mathcal{L}_{\textit{VQ}}=||\widehat{\Delta\mu}-\Delta\mu||$
$\displaystyle+||\texttt{sg}[\mathbf{e}_{\Delta\mu}]-\mathbf{e}^{\mathbf{q}}_{\Delta\mu}||+\beta_{1}||\mathbf{e}_{\Delta\mu}-\texttt{sg}[\mathbf{e}^{\mathbf{q}}_{\Delta\mu}]||+$
$\displaystyle||\widehat{\Delta L}-\Delta L||$
$\displaystyle+||\texttt{sg}[\mathbf{e}_{\Delta
L}]-\mathbf{e}^{\mathbf{q}}_{\Delta L}||+\beta_{2}||\mathbf{e}_{\Delta
L}-\texttt{sg}[\mathbf{e}^{\mathbf{q}}_{\Delta L}]||,$ (9)

where sg denotes the stop gradient operation, $\beta_{1}$ and $\beta_{2}$ are
two weight balancing coefficients.

###  3.3 Co-Speech Gesture GPT with Motion Refinement

Co-Speech Gesture GPT Network. With the position-irrelevant motion pattern of
valid quantization design, each co-speech gesture clip can be transformed into
discrete representation. We then learn a co-speech gesture GPT network to map
from speech audio $\mathbf{a}_{(1:T)}$ to quantized code sequences
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(1:T^{\prime})}$ and
$\mathbf{e}^{\mathbf{q}}_{\Delta L,(1:T^{\prime})}$. Specifically, we extract
audio features $\mathbf{a}^{\textit{onset}}_{(1:T^{\prime})}$ with onset
strength information, which is more suitable for cross-modal pattern learning
[48, 51]. Then, a feature embedding layer with positional embedding is
leveraged to obtain the tokens for audio onset features, quantized relative
shift-translation and qunatized relative factorial covariance. Further, we
encode cross-attention information with a series of transformer layers.
Finally, followed by a linear transformation with softmax activation, the
$M$-dimensional output denotes the probability of each quantization code at
that time step. The whole co-speech gesture GPT is trained with cross-entropy
loss $\mathcal{L}_{\textit{CE}}$. Such design enables us to predict and sample
future quantization code $\mathbf{e}^{\mathbf{q}}_{\Delta\mu}$ and
$\mathbf{e}^{\mathbf{q}}_{\Delta L}$ with speech audio.

Motion Refinement by Learning Residuals. Now that we can reconstruct the
relative shift-translation $\widehat{\Delta\mu}$ and factorial covariance
change $\widehat{\Delta L}$ by VQ-VAE decoding. Given $\mu_{1}$ and $L_{1}$
extracted from the initial image frame $I_{(1)}$, the absolute shift-
translation and factorial covariance for the $j$-th frame can be calculated as
$\widehat{\mu_{j}}=\mu_{1}+\sum_{i=2}^{j}\widehat{\Delta\mu_{i}}$ and
$\widehat{L_{j}}=L_{1}+\sum_{i=2}^{j}\widehat{\Delta L_{i}}$, respectively.
However, since the quantized codebook is designed to only represent the large-
scale common motion pattern information, while fine-grained rhythmic details
are omitted. Therefore, we propose to refine the co-speech movements by
learning residual terms. Concretely, we extract the audio mfcc features
$\mathbf{a}^{\textit{mfcc}}_{(1:T)}$ to encode more contextual audio cues for
prosodic dynamics learning. Then a bi-directional LSTM is used to predict the
per-frame motion representation residuals $R$ to the main motion pattern
result $\widehat{\mu}_{(1:T)}$ and $\widehat{L}_{(1:T)}$, i.e.,
$\widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:T)}=\left[R(\widehat{\mu}_{(1:T)};\mathbf{a}^{\textit{mfcc}}_{(1:T)});R(\widehat{L}_{(1:T)};\mathbf{a}^{\textit{mfcc}}_{(1:T)})\right]$.
By adding residual terms, the overall co-speech gesture GPT with motion
refinement learning can be formulated as:

$\mathcal{L}_{\textit{Residual}}=||\mathcal{M}_{(1:T)}-\widehat{\mathcal{M}}_{(1:T)}(\mathbf{a})||,\textit{where
}\widehat{\mathcal{M}}_{(1:T)}(\mathbf{a})=\widehat{\mathcal{M}}^{\textit{pattern}}_{(1:T)}(\mathbf{a}^{\textit{onset}})+\widehat{\mathcal{M}}^{\textit{rhythmic}}_{(1:T)}(\mathbf{a}^{\textit{mfcc}}).$
(10)

In this way, we capture both major gesture patterns and subtle rhythmic
dynamics for vivid results.

##  4 Experiments

Table 1: The quantitative results on PATS Image Dataset. We compare Audio-driveN Gesture vIdeo gEneration (ANGIE) against recent SOTA methods [22, 34, 41, 65] and ground truth on four speakers’ subsets. For FGD the lower the better, and the higher the better for other metrics. | Oliver | Seth | Kubinec | Jon
---|---|---|---|---
Methods | FGD | BC | Div. | FGD | BC | Div. | FGD | BC | Div. | FGD | BC | Div.
GT | 0.00 | 0.76 | 54.6 | 0.00 | 0.71 | 49.3 | 0.00 | 0.84 | 38.9 | 0.00 | 0.73 | 62.8
S2G [22] | 8.57 | 0.59 | 46.1 | 5.75 | 0.62 | 38.2 | 4.76 | 0.67 | 31.6 | 6.07 | 0.51 | 47.3
HA2G [34] | 3.28 | 0.75 | 49.2 | 4.06 | 0.72 | 40.1 | 2.98 | 0.79 | 32.3 | 3.74 | 0.64 | 50.2
SDT [41] | 1.04 | 0.61 | 52.9 | 1.97 | 0.58 | 46.7 | 1.15 | 0.77 | 36.1 | 1.63 | 0.60 | 57.4
TriCon [65] | 3.63 | 0.53 | 48.3 | 3.79 | 0.52 | 40.3 | 3.27 | 0.77 | 35.7 | 3.98 | 0.61 | 49.7
ANGIE | 0.88 | 0.72 | 53.5 | 1.83 | 0.69 | 46.7 | 1.10 | 0.81 | 36.5 | 1.57 | 0.65 | 60.9

###  4.1 Experimental Settings

Dataset and Preprocessing. Pose, Audio, Transcript, Style (PATS) is a large-
scale dataset of 25 speakers with aligned pose, audio and transcripts [1, 2,
22]. The training corpus contains 251 hours of data with around 84,000
intervals of mean length 10.7s. Notably, the PATS dataset contains three
modalities of audio log-mel spectrograms, speech transcripts and per-frame
skeletons labeled with OpenPose [10]. To bypass the error accumulation in pose
annotation and facilitate co-speech gesture image generation task, we extend
PATS with more features: 1) preprocessed image frames and 2) onset strength
audio features which are more suitable for co-speech gesture pattern learning.

We conduct the experiments on four speakers’ co-speech video subsets,
including Oliver, Seth, Kubinec and Jon. Concretely, 2D skeletons of the image
frames are obtained by OpenPose [10] for baseline methods training. The frames
are cropped to make the speaker locate at the image center. Since the time
span of a meaningful co-speech gesture unit sequence ranges from 4s to 14s
[28, 49], we trim invalid videos and finally obtain 1306, 990, 1294 and 1284
clips for four subsets, respectively. The overall mean clip length is 9.8s. We
randomly split the segments into 90% for training and 10% for evaluation. The
image frames are sampled at 25 fps and further resized to $256\times 256$.

Comparison Methods. We compare with recent SOTA works: 1) _Speech to Gesture_
(S2G) [22], a GAN-based pipeline that maps audio to 2D keypoints with a U-Net;
2) _Hierarchical Audio to Gesture_ (HA2G) [34] which captures the hierarchical
associations between multi-level audio features and tree-like human skeletons;
3) _Speech Drives Template_ (SDT) [41] which relieves the one-to-many mapping
ambiguity by a set of continuous gesture template vectors; 4) _Trimodal
Context_ (TriCon) [65], a representative framework that considers the trimodal
context of audio, text and speaker identity. Note that all methods could drive
2D human skeletons with speech audio. We train baselines on the PATS image
dataset and tune the hyper-parameters by grid search for the best evaluation
result. In particular, we also show direct evaluations on the _Ground Truth_
(GT) skeletons for clearer comparison.

Implementation Details. We sample $T=96$ frame clips with stride $32$ for
training. 1) For the VQ-Motion Extractor: the co-speech gesture pattern
codebook size $M$ for both relative shift-translation $\Delta\mu$ and
factorial covariance change $\Delta L$ are set to $512$. The encoders
$E_{\Delta\mu}$, $E_{\Delta L}$ and the decoders $D_{\Delta\mu}$,
$D_{\Delta\mu}$ are based on 1D-convolution structure. The channel dimension
$\ell$ of each codebook entry $\mathbf{d}_{\Delta\mu}$, $\mathbf{d}_{\Delta
L}$ as well as the encoded latent features $\mathbf{e}_{\Delta\mu}$,
$\mathbf{e}_{\Delta L}$ are $512$, while the temporal dimension $T^{\prime}$
is set as $T/8=12$ to encode contextual features with downsampling rate of
$8$. The $\epsilon$ is set as $1\times 10^{-5}$ to guarantee the positiveness
of diagonal entries in factorial covariance $L$. The commit loss trade-offs in
$\mathcal{L}_{\textit{VQ}}$ are empirically set as $\beta_{1}=\beta_{2}=0.1$.
We optimize the gesture pattern VQ-VAE with Adam optimizer [29] of learning
rate $3\times 10^{-5}$. 2) For the Co-Speech GPT: the Transformer channel
dimension is $768$, and the attention layer is implemented in $12$ heads with
dropout probability of $0.1$. The onset strength audio features
$\mathbf{a}^{\textit{onset}}\in\mathbb{R}^{426}$ are extracted by Librosa,
while the audio mfcc features
$\mathbf{a}^{\textit{mfcc}}\in\mathbb{R}^{28\times 12}$ are computed with the
window size of $10$ ms. During the GPT training, the
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(1:11)}$, $\mathbf{e}^{\mathbf{q}}_{\Delta
L,(1:11)}$ are used as input while
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(2:12)}$, $\mathbf{e}^{\mathbf{q}}_{\Delta
L,(2:12)}$ serve as supervision labels. 3) For the motion representation
$\mathcal{M}$ and image generator $G$: we implement as MRAA [47] to use $K=20$
regions. The motion estimator is pretrained for knowledge distillation. The
overall framework is implemented in PyTorch [39] and trained on one 16G Tesla
V100 GPU for three days.

###  4.2 Quantitative Evaluation

Evaluation Metrics. We adopt 1) _Fréchet Gesture Distance_ (FGD) [65] to
evaluate the distance between the real and synthetic gesture distribution. We
train an auto-encoder on the PATS image dataset and use the encoder to compute
fréchet distance between the real and synthetic gesture in feature space. We
also use the 2) _Beat Consistency Score_ (BC) and 3) _Diversity_ (Div.) [32,
34] to account for the speech-motion alignment and the diversity among
generated gestures. Specifically, BC is computed as the average temporal
distance between each audio beat and its closest gesture beat, and Diversity
indicates the difference of multiple audios’ corresponding gestures in the
latent space. Note that since all metrics are skeleton-based, we downgrade our
method to operate on skeleton data for fair comparison, _i.e._ , we use VQ-VAE
w/o cholesky scheme to create 2D skeletons for evaluation.

Evaluation Results. The results are reported in Table 1. It can be seen that
the proposed ANGIE achieves the best evaluation results on most metrics. Since
our method summarizes reusable co-speech gesture patterns into quantized
codebooks and complements subtle rhythmic dynamics, we can cover richer
gesture patterns and create diverse results. Note that HA2G [34] tends to
generate over-expressive gestures with multi-level audio features, which makes
their results on BC even better than the ground truth in some cases. Despite
of this, we perform comparable to ground truth on BC metric with stable motion
results, showing that we can generate audio-aligned gestures. Besides, we can
find that both SDT [41] and ours perform better on FGD and Diversity metrics
than other methods due to the explicit modeling of co-speech gesture patterns.
However, since the gesture pattern is finite and discrete, our quantized
codebook design is more suitable than continuous representation.

Table 2: User study results on co-speech gesture generation quality. The rating scale is 1-5, with the larger the better. We compare the Realness, Synchrony and Diversity to baselines [22, 34, 41, 65]. Methods | GT | S2G [22] | HA2G [34] | SDT [41] | TriCon [65] | ANGIE (Ours)
---|---|---|---|---|---|---
Realness | 4.29 | 3.27 | 3.92 | 4.01 | 3.74 | 4.08
Synchrony | 4.36 | 3.48 | 4.01 | 3.97 | 3.85 | 4.11
Diversity | 3.97 | 2.49 | 3.31 | 3.88 | 3.02 | 3.95
Figure 3: Image sequence results of ANGIE. We show the co-speech gesture image
generation results of Kubinec, Seth and Jon, respectively. More qualitative
results can be found in demo video.

###  4.3 Qualitative Analysis

User Study. We further conduct a user study to reflect the quality of audio-
driven gestures. Concretely, we sample 25 audio clips from the PATS image test
set for all methods to generate skeleton results, then involve 18 participants
for user study. The Mean Opinion Scores protocol is adopted, which requires
the participants to rate three aspects: (1) Realness; (2) Synchrony; (3)
Diversity. The rating scale is 1 to 5, with 1 being the poorest and 5 being
the best. The results are reported in Table 2, where our method performs the
best on all three aspects. Notably, with the help of motion pattern codebook,
we achieve comparable diversity result to the ground truth, demonstrating that
the quantization design helps to capture common gesture patterns.

Video Generation Results. With the Co-Speech GPT and motion refinement
network, we could predict the future quantization code and complement rhythmic
motion details based on audio features, which enables us to generate video
results. As shown in Fig. 3, the synthesized image sequence results from left
to right contain diverse and meaningful gestures that are aligned with speech
audio.

Table 3: Ablation study of VQ-Motion Extractor and Co-Speech GPT with Motion Refinement. Ablation Settings | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$ |  $\mathcal{L}_{1}$ $\downarrow$ | Perceptual $\downarrow$ | AED $\downarrow$
---|---|---|---|---|---|---
w/o Quantization | 5.86 | 0.54 | 35.6 | 0.071 | 79.2 | 0.095
Quantize $\mu$, $L$ | - | - | - | 0.058 | 67.4 | 0.086
Quantize $\Delta\mu$, $L$ | - | - | - | 0.043 | 52.3 | 0.069
Quantize $\mu$, $\Delta L$ | - | - | - | 0.052 | 63.6 | 0.083
w/o Motion Refinement | 1.39 | 0.58 | 48.3 | 0.041 | 48.1 | 0.063
ANGIE (Ours) | 1.35 | 0.72 | 49.4 | 0.037 | 42.9 | 0.063
Figure 4: Codebook Analysis. We validate that the codebooks contain meaningful
motion patterns.

Codebook Analysis for Co-Speech Gesture Pattern. We analyze the meaning of the
quantized codebooks by asking the question: Does each codebook entry really
correspond to a certain type of motion pattern? To answer this, we validate
two things in Fig. 4: 1) Different entries represent diverse gesture patterns,
hence driving the same image with different quantized codes leads to different
gestures (left). 2) Each codebook entry denotes a fixed motion pattern, hence
driving different images with the same quantized code show same motions
(right). Please refer to demo video for more results.

###  4.4 Ablation Study

In this section, we present ablation study of two modules in our framework.
Note that except for the skeleton-based metrics, we also use video
reconstruction accuracy as a proxy for image quality, including
$\mathcal{L}_{1}$ and perceptual loss [27] between the reconstructed and GT
image; _Average Euclidean Distance_ (AED) evaluates identity preservation by
pretrained re-identification networks [25, 47].

VQ-Motion Extractor. We conduct ablation experiments on five settings: 1) w/o
Quantization, where we directly infer motion representation with audio; 2)
Quantize $\mu$, $L$; 3) Quantize $\Delta\mu$, $L$; 4) Quantize $\mu$, $\Delta
L$, where we quantize the absolute or relative difference of the motion
components; 5) w/o Cholesky Decomposition, which means the covariance matrix
is not guaranteed to be symmetric positive definite. The results are shown in
Table 3, which proves that the quantization design with position-irrelevant
relative motion pattern could improve the performance. We also find that
quantizing $\Delta\mu$ is more effective than $\Delta L$, since the shift
translation is more correlated with region position. Note that when motion
component is invalid, the pipeline fails to generate image due to the
numerical instability in calculating affine matrix. Hence the results for the
setting 5 are not reported.

Motion Refinement Module. To verify the efficacy of motion residuals, we
eliminate the motion refinement for ablation. The results in Table 3 suggest
that this module improves beat consistency by capturing subtle rhythmic
movements, while the improvement on FGD majorly derives from the quantization
design. Such module complements the motion pattern for fine-grained results.

##  5 Discussion

Conclusion. In this paper, we propose a novel framework ANGIE to generate
audio-driven co-speech gesture video in the image domain. To summarize valid
common co-speech gesture patterns, we propose VQ-Motion Extractor with
cholesky decomposition based quantization scheme and position-irrelevant
design to represent relative motion patterns. Then we propose Co-Speech GPT to
refine subtle rhythmic movement details for fine-grained results. Extensive
experiments demonstrate that our framework renders realistic and vivid co-
speech gesture video generation results.

Ethical Consideration. Generating co-speech gesture images facilitates
applications such as digital human. However, it could be misused for malicious
purposes like forgery generation. We believe that the proper use of this
technique will enhance the machine learning research and digital
entertainment.

Limitation and Future Work. As an early work towards audio-driven co-speech
video generation without structural prior, we notice that our method fails for
some challenging cases. For example, if the source image is in a large pose,
it is hard to generalize well in such an out-of-domain data. We will explore
how to develop a model with higher generalization ability in future work.

##  6 Acknowledgment

This work is in part supported by GRF 14205719, TRS T41-603/20-R, Centre for
Perceptual and Interactive Intelligence, and CUHK Interdisciplinary AI
Research Institute; and in part supported by NTU NAP, MOE AcRF Tier 2
(T2EP20221-0033), and under the RIE2020 Industry Alignment Fund – Industry
Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-
kind contribution from the industry partner(s).

## References

  * [1] Chaitanya Ahuja, Dong Won Lee, Ryo Ishii, and Louis-Philippe Morency.  No gestures left behind: Learning relationships between spoken language and freeform gestures.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1884–1895, 2020.
  * [2] Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency.  Style transfer for co-speech gesture animation: A multi-speaker conditional-mixture approach.  In European Conference on Computer Vision, pages 248–265. Springer, 2020.
  * [3] Chaitanya Ahuja and Louis-Philippe Morency.  Language2pose: Natural language grounded pose forecasting.  In 2019 International Conference on 3D Vision (3DV), pages 719–728. IEEE, 2019.
  * [4] Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow.  Style-controllable speech-driven gesture synthesis using normalising flows.  In Computer Graphics Forum, volume 39, pages 487–496. Wiley Online Library, 2020.
  * [5] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.  arXiv preprint arXiv:1803.01271, 2018.
  * [6] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag.  Synthesizing images of humans in unseen poses.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8340–8348, 2018.
  * [7] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishek Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha.  Text2gestures: A transformer-based network for generating emotive body gestures for virtual agents.  In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 1–10. IEEE, 2021.
  * [8] Judee K Burgoon, Thomas Birk, and Michael Pfau.  Nonverbal behaviors, persuasion, and credibility.  Human communication research, 17(1):140–169, 1990.
  * [9] Chen Cao, Qiming Hou, and Kun Zhou.  Displaced dynamic expression regression for real-time facial tracking and animation.  ACM Transactions on graphics (TOG), 33(4):1–10, 2014.
  * [10] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: realtime multi-person 2d pose estimation using part affinity fields.  IEEE transactions on pattern analysis and machine intelligence, 43(1):172–186, 2019.
  * [11] Justine Cassell, David McNeill, and Karl-Erik McCullough.  Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information.  Pragmatics & cognition, 7(1):1–34, 1999.
  * [12] Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone.  Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents.  In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 413–420, 1994.
  * [13] Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore.  Beat: the behavior expression animation toolkit.  In Life-Like Characters, pages 163–185. Springer, 2004.
  * [14] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros.  Everybody dance now.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [15] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.  Hierarchical cross-modal talking face generation with dynamic pixel-wise loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7832–7841, 2019.
  * [16] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black.  Monocular expressive body regression through body-driven attention.  In European Conference on Computer Vision (ECCV), 2020.
  * [17] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black.  Capture, learning, and synthesis of 3d speaking styles.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10101–10111, 2019.
  * [18] Jan P De Ruiter, Adrian Bangerter, and Paula Dings.  The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis.  Topics in cognitive science, 4(2):232–248, 2012.
  * [19] Ylva Ferstl and Rachel McDonnell.  Investigating the use of recurrent motion modelling for speech gesture generation.  In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pages 93–98, 2018.
  * [20] Ylva Ferstl, Michael Neff, and Rachel McDonnell.  Adversarial gesture generation with realistic gesture phasing.  Computers & Graphics, 89:117–130, 2020.
  * [21] Zhenglin Geng, Chen Cao, and Sergey Tulyakov.  3d guided fine-grained face manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9821–9830, 2019.
  * [22] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik.  Learning individual styles of conversational gesture.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3497–3506, 2019.
  * [23] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Elgharib, and Christian Theobalt.  Learning speech-driven 3d conversational gestures from video.  arXiv preprint arXiv:2102.06837, 2021.
  * [24] Dai Hasegawa, Naoshi Kaneko, Shinichi Shirakawa, Hiroshi Sakuta, and Kazuhiko Sumi.  Evaluation of speech-to-gesture generation using bi-directional lstm network.  In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pages 79–86, 2018.
  * [25] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [26] Carlos T. Ishi, Daichi Machiyashiki, Ryusuke Mikata, and Hiroshi Ishiguro.  A speech-driven hand gesture generation method and evaluation in android robots.  IEEE Robotics and Automation Letters, 3(4):3757–3764, 2018.
  * [27] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [28] Adam Kendon.  Gesture: Visible action as utterance.  Cambridge University Press, 2004.
  * [29] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [30] Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellström.  Gesticulator: A framework for semantically-aware speech-driven gesture generation.  In Proceedings of the 2020 International Conference on Multimodal Interaction, ICMI ’20, page 242–250, New York, NY, USA, 2020. Association for Computing Machinery.
  * [31] Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao.  Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders.  arXiv preprint arXiv:2108.06720, 2021.
  * [32] Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa.  Learn to dance with aist++: Music conditioned 3d dance generation.  arXiv preprint arXiv:2101.08779, 2021.
  * [33] Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou.  Visual sound localization in the wild by cross-modal interference erasing.  arXiv preprint arXiv:2202.06406, 2022.
  * [34] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou.  Learning hierarchical cross-modal association for co-speech gesture generation.  arXiv preprint arXiv:2203.13161, 2022.
  * [35] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou.  Semantic-aware implicit neural audio-driven video portrait generation.  arXiv preprint arXiv:2201.07786, 2022.
  * [36] McFee, Brian, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python, 2015.  In Proceedings of the 14th python in science conference, pp. 18-25. 2015\.
  * [37] David McNeill.  Hand and mind.  De Gruyter Mouton, 2011.
  * [38] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.  Learning to listen: Modeling non-deterministic dyadic facial motion.  arXiv preprint arXiv:2204.08451, 2022.
  * [39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [40] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar.  A lip sync expert is all you need for speech to lip generation in the wild.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 484–492, 2020.
  * [41] Shenhan Qian, Zhi Tu, YiHao Zhi, Wen Liu, and Shenghua Gao.  Speech drives templates: Co-speech gesture synthesis with learned templates.  arXiv preprint arXiv:2108.08020, 2021.
  * [42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [43] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh.  Meshtalk: 3d face animation from speech using cross-modality disentanglement.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173–1182, 2021.
  * [44] Najmeh Sadoughi, Yang Liu, and Carlos Busso.  Msp-avatar corpus: Motion capture recordings to study the role of discourse functions in the design of intelligent virtual agents.  In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 7, pages 1–6. IEEE, 2015\.
  * [45] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  Animating arbitrary objects via deep motion transfer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2377–2386, 2019.
  * [46] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [47] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Motion representations for articulated animation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653–13662, 2021.
  * [48] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu.  Bailando: 3d dance generation by actor-critic gpt with choreographic memory.  arXiv preprint arXiv:2203.13055, 2022.
  * [49] Michael Studdert-Kennedy.  Hand and mind: What gestures reveal about thought.  Language and Speech, 37(2):203–209, 1994.
  * [50] Kenta Takeuchi, Souichirou Kubota, Keisuke Suzuki, Dai Hasegawa, and Hiroshi Sakuta.  Creating a gesture-speech dataset for speech-based automatic gesture generation.  In International Conference on Human-Computer Interaction, pages 198–202. Springer, 2017.
  * [51] Taoran Tang, Jia Jia, and Hanyang Mao.  Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1598–1606, 2018.
  * [52] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2387–2395, 2016.
  * [53] Jackson Tolins, Kris Liu, Yingying Wang, Jean E Fox Tree, Marilyn Walker, and Michael Neff.  A multimodal motion-captured corpus of matched and mismatched extravert-introvert conversational pairs.  In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3469–3476, 2016.
  * [54] Lloyd N Trefethen and David Bau III.  Numerical linear algebra, volume 50.  Siam, 1997.
  * [55] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  Advances in neural information processing systems, 30, 2017.
  * [56] Susanne Van Mulken, Elisabeth Andre, and Jochen Müller.  The persona effect: how substantial is it?  In People and computers XIII, pages 53–66. Springer, 1998.
  * [57] Ekaterina Volkova, Stephan De La Rosa, Heinrich H Bülthoff, and Betty Mohler.  The mpi emotional body expressions database for narrative scenarios.  PloS one, 9:e113647, 2014.
  * [58] P. Wagner, Z. Malisz, and S. Kopp.  Gesture and speech in interaction: An overview.  Speech Communication, 57:209–232, 2014.
  * [59] Olivia Wiles, A Koepke, and Andrew Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In Proceedings of the European conference on computer vision (ECCV), pages 670–686, 2018.
  * [60] Jason R Wilson, Nah Young Lee, Annie Saechao, Sharon Hershenson, Matthias Scheutz, and Linda Tickle-Degnen.  Hand gestures and verbal acknowledgments improve human-robot rapport.  In International Conference on Social Robotics, pages 334–344. Springer, 2017.
  * [61] Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, and Pascale Fung.  Lightweight and efficient end-to-end speech recognition using low-rank transformer.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6144–6148. IEEE, 2020.
  * [62] Jing Xu, Wei Zhang, Yalong Bai, Qibin Sun, and Tao Mei.  Freeform body motion generation from speech.  arXiv preprint arXiv:2203.02291, 2022.
  * [63] Yanzhe Yang, Jimei Yang, and Jessica Hodgins.  Statistics-based motion synthesis for social conversations.  In Computer Graphics Forum, volume 39, pages 201–212. Wiley Online Library, 2020.
  * [64] Payam Jome Yazdian, Mo Chen, and Angelica Lim.  Gesture2vec: Clustering gestures using representation learning methods for co-speech gesture generation, 2022.
  * [65] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Speech gesture generation from the trimodal context of text, audio, and speaker identity.  ACM Transactions on Graphics (TOG), 39(6):1–16, 2020.
  * [66] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee.  Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots.  In 2019 International Conference on Robotics and Automation (ICRA), pages 4303–4309. IEEE, 2019.
  * [67] Jianwei Yu, Shi-Xiong Zhang, Jian Wu, Shahram Ghorbani, Bo Wu, Shiyin Kang, Shansong Liu, Xunying Liu, Helen Meng, and Dong Yu.  Audio-visual recognition of overlapped speech for the lrs2 dataset.  In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6984–6988. IEEE, 2020.
  * [68] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9459–9468, 2019.
  * [69] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.  Pose-controllable talking face generation by implicitly modularized audio-visual representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Audio-Driven Co-Speech Gesture Video Generation
(Supplemental Document)

In the supplemental document, we will introduce below contents: 1) proof of
Theorem 1 (unique cholesky decomposition theorem) (Sec. A); 2) more details
about Co-Speech Gesture GPT (Sec. B); 3) model architecture details (Sec. C);
4) analysis on the limitations and future work (Sec. D); 5) analysis on the
model’s generalization ability (Sec. E); 6) the standard deviation of use
study results (Sec. F); 7) the details of audio feature extraction (Sec. G);
8) analysis on extra input modality of text information (Sec. H); 9)
experimental results with error bars (Sec. I); 10) user study details (Sec.
J); 11) ethical considerations with possible prevention measures against
potential negative societal impact (Sec. K); 12) discussions on the dataset
consent and personally identifiable information (Sec. L); 13) the licenses of
existing assets involved in this paper (Sec. M);

##  Appendix A Proof of Theorem 1

In the main paper, to ease the constraint in the quantization process, we use
the _unique cholesky decomposition theorem_ [54] to transform the covariance
matrix $C$ to factorial covariance $L$ by theorem:

######  Theorem 2.

For any real symmetric positive definite matrix $C\in\mathbb{S}^{n}_{++}$,
there exists a unique lower triangular matrix $L$ with positive diagonal
entries, such that $C=LL^{\mathsf{T}}$.

Proof. As $C$ is a symmetric positive definite matrix, it can be represented
as $C=PDP^{\mathsf{T}}$, where $D$ is a diagonal matrix with positive diagonal
entries and $P$ is a unitary matrix. Suppose
$B^{\mathsf{T}}=D^{\frac{1}{2}}P^{\mathsf{T}}$, it can be factorized as the
product of a unitary matrix $Q$ and an upper triangular matrix $R$ (according
to Matrix QR Decomposition), _i.e._ , $B^{\mathsf{T}}=QR$. Thus, we have

$C=PDP^{\mathsf{T}}=BB^{\mathsf{T}}=R^{\mathsf{T}}Q^{\mathsf{T}}QR=R^{\mathsf{T}}R=LL^{\mathsf{T}},$
(11)

where $L=R^{\mathsf{T}}$ is a lower triangular matrix. Because $C$ has the
positive determinants, the diagonal entries of $L$ should be non-zero values.
Therefore, there exists a lower triangular matrix $L$ with positive diagonal
entries such that $C=LL^{\mathsf{T}}$.

Assume $C=L_{1}L_{1}^{\mathsf{T}}=L_{2}L_{2}^{\mathsf{T}}$ where $L_{1},L_{2}$
are two distinct lower triangular matrices with diagonal positive entries, it
shows that
$I=L_{1}^{-1}L_{2}L_{2}^{\mathsf{T}}L_{1}^{-\mathsf{T}}=(L_{1}^{-1}L_{2})(L_{1}^{-1}L_{2})^{\mathsf{T}}$
where $I$ is the identity matrix. It means that
$L_{1}^{-1}L_{2}=(L_{1}^{-1}L_{2})^{-\mathsf{T}}$ which further shows that
$L_{1}^{-1}L_{2}$ should be a diagonal matrix. Because
$I=(L_{1}^{-1}L_{2})^{2}$, it demonstrates that $L_{1}^{-1}L_{2}$ is a
diagonal matrix with $\pm 1$ entries. Since both $L_{1},L_{2}$ have positive
diagonal entries, it means $L_{1}^{-1}L_{2}=I$. Therefore, $L_{1}=L_{2}$. ∎

##  Appendix B More Details about Co-Speech Gesture GPT

The target of the Co-Speech Gesture GPT is to predict the quantized motion
codes from the audio onset features $\mathbf{a}^{\textit{onset}}$. With the
proposed vector quantization scheme of position-irrelevant representation, we
could transform the raw relative shift-translation $\Delta\mu_{(1:T)}$ and
relative factorial covariance change $\Delta L_{(1:T)}$ into corresponding
quantized code sequence $\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(1:T^{\prime})}$
and $\mathbf{e}^{\mathbf{q}}_{\Delta L,(1:T^{\prime})}$. In this way, any co-
speech gesture sequence can be represented by a sequence of quantized motion
codes. The original continuous audio-to-gesture prediction task is reframed as
to select the proper motion code from the codebooks $\mathcal{D}_{\Delta\mu}$
and $\mathcal{D}_{\Delta L}$ according to the previous motion codes and
current speech audio features. The output of the GPT [42] model at the $t$-th
time step is the probability of choosing each codebook entry, where the entry
with the largest probability serves as the predicted motion code of the next
time step.

In particular, given the quantized code sequence
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(1:T^{\prime})}\in\mathbb{R}^{T^{\prime}}$
and $\mathbf{e}^{\mathbf{q}}_{\Delta
L,(1:T^{\prime})}\in\mathbb{R}^{T^{\prime}}$ of temporal length $T^{\prime}$,
two embedding layers map them to learnable features
$\mathbf{f}_{\Delta\mu}\in\mathbb{R}^{T^{\prime}\times\ell}$ and
$\mathbf{f}_{\Delta L}\in\mathbb{R}^{T^{\prime}\times\ell}$, respectively. The
onset audio features $\mathbf{a}^{\textit{onset}}$ are also passed through an
embedding layer to obtain audio features
$\mathbf{f}_{a}\in\mathbb{R}^{T^{\prime}\times\ell}$. After the concatenation
over the temporal dimension and supplement the position embedding information,
we get the latent input features of shape $(3\times T^{\prime})\times\ell$.
Then, $12$ Transformer layers with channel dimension of $768$ and $12$
attention heads of dropout probability $0.1$ are utilized to encode the cross-
attention information. Finally, a linear and softmax layer map the encoded
latent features to normalized motion probabilities of
$\mathbf{p}\in\mathbb{R}^{(3\times T^{\prime})\times M}$, where $M$ is the
codebook size. Notably, $\mathbf{p}_{t,m}$ denote the probability of motion
code $\mathbf{d}_{\Delta\mu,m}$ and $\mathbf{d}_{\Delta L,m}$ predicted at the
time step $t+1$ for relative shift-translation $\Delta\mu$ and relative
factorial covariance change $\Delta L$, respectively. During the
implementation, the probabilities at the audio feature indexes
$\mathbf{p}_{(1:T^{\prime})}$ are discarded, while the probabilities at the
$\Delta\mu$ and $\Delta L$ indexes represent the corresponding motion
probabilities, _i.e._ ,
$\mathbf{p}_{\Delta\mu}=\mathbf{p}_{(T^{\prime}+1:2T^{\prime})}$ and
$\mathbf{p}_{\Delta L}=\mathbf{p}_{(2T^{\prime}+1:3T^{\prime})}$ [48]. The
attention layer can be formulated as:

$\displaystyle\text{Attention}(\mathbf{Q,K,V,M})=\text{softmax}(\frac{\mathbf{QK^{\mathsf{T}}}+\mathbf{M}}{\sqrt{\ell}})\mathbf{V},$
(12)

where $\mathbf{Q,K,V}$ are the query, key and value matrix from input,
$\mathbf{M}$ denote the mask, $\ell$ is the channel dimension and $\mathsf{T}$
is the matrix transpose operation. To encode the cross-conditional features of
audio features, relative shift-translation features and relative covariance
change features, the mask $\mathbf{M}$ is set as a $3\times 3$ repeated block
matrix with the lower triangular matrix as its element.

The Co-Speech GPT is trained via cross-entropy loss as a classification
problem. Specifically, the motion codes of
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(1:11)}$, $\mathbf{e}^{\mathbf{q}}_{\Delta
L,(1:11)}$ are used as input, while
$\mathbf{e}^{\mathbf{q}}_{\Delta\mu,(2:12)}$, $\mathbf{e}^{\mathbf{q}}_{\Delta
L,(2:12)}$ serve as supervision labels. The overall loss function is:

$\displaystyle\mathcal{L}_{\textit{CE}}=\frac{1}{T^{\prime}}\sum_{t=1}^{T^{\prime}}\left\\{\texttt{CrossEntropy}\left[\mathbf{p}_{(\Delta\mu,t)},\mathbf{e}^{\mathbf{q}}_{(\Delta\mu,t+1)}\right]+\texttt{CrossEntropy}\left[\mathbf{p}_{(\Delta
L,t)},\mathbf{e}^{\mathbf{q}}_{(\Delta L,t+1)}\right]\right\\},$ (13)

where CrossEntropy denotes the cross-entropy loss function;
$\mathbf{p}_{(\Delta\mu,t)}$, $\mathbf{p}_{(\Delta L,t)}$ are the predicted
probability of $\Delta\mu$, $\Delta L$ terms at the $t$-th time step,
respectively; the $\mathbf{e}^{\mathbf{q}}_{(\Delta\mu,t+1)}$,
$\mathbf{e}^{\mathbf{q}}_{(\Delta L,t+1)}$ are the vector quantized code of
$\Delta\mu$ and $\Delta L$ at the $(t+1)$-th time step, respectively.

##  Appendix C Architecture Details

VQ-Motion Extractor Encoder and Decoder. The encoders $E_{\Delta\mu}$,
$E_{\Delta L}$ and the decoders $D_{\Delta\mu}$, $D_{\Delta\mu}$ are 1D
temporal convolution networks. In particular, the encoders have three
convolution layers, with each layer’s kernel size of $3$, padding of $1$ and
stride of $2$. In this way, the downsampling scale of the encoder is $8$,
which will compress the temporal contextual information. For the decoder, the
structure is similar, with three convolution layers of kernel size $3$,
padding $1$ and stride $1$, and three upsample layers of upsample scale $2$ to
reconstruct the latent features.

Motion Estimator and Image Generator. For the motion estimator, we follow MRAA
[47] that uses a region predictor to extract the articulated human body
information. Specifically, we similarly use the U-Net architecture with five
“convolution - batch norm - ReLU - pooling” blocks in the encoder and five
“upsample - convolution - batch norm - ReLU” blocks in the decoder for both
the region predictor and pixel-wise flow predictor. Note that since it is
inappropriate to infer the background motion from speech audio cues, we remove
the original background motion predictor. In terms of the image generator, it
is based on the Johnson architecture [27], with two down-sampling blocks, six
residual-blocks, and two up-sampling blocks. The skip connections are added to
warp and weight the confidence maps.

Motion Refinement Network. The motion refinement network maps the audio mfcc
features to the residual terms to complement the subtle rhythmic movements of
the co-speech gesture. Concretely, the audio mfcc features
$\mathbf{a}^{\textit{mfcc}}\in\mathbb{R}^{28\times 12}$ are first pre-
extracted with the window size of $10$ ms. Then a convolution based audio
encoder with a series of linear layers is used to encode the per-frame audio
features into $\mathbf{f}_{a}^{\textit{mfcc}}\in\mathbb{R}^{128}$. The
detailed structure is listed in Table 4.

We first follow the pipeline of MRAA [47] to pretrain the motion estimator and
image generator via self-reconstruction. Then, the motion estimator module is
freezed and serves as supervision in VQ-Motion Estimator training, _i.e._ , we
train the VQ-VAE model to reconstruct the motion representation from
pretrained MRAA motion estimator.

Table 4: Detailed structure of audio encoder. †Note that in the table, Conv2d
($c_{in}$, $c_{out}$, $k$, $s$, $p$) means the conv2d operation of input
channel dimension $c_{in}$, the output channel dimension of $c_{out}$, the
kernel size of $k$, the stride of $s$ and the padding of $p$; MaxPool2d ($k$,
($s_{h}$, $s_{w}$)) means the MaxPool2d operation of kernel size $k$, h
direction stride of $s_{h}$ and w direction stride of $s_{w}$; Linear
($c_{in}$, $c_{out}$) means the linear layer with input feature of dimension
$c_{in}$ and output feature of dimension $c_{out}$; ReLU means the ReLU
operation on the input feature. Audio Encoder
---
Feature | Feature Shapes | Operations
Input | $1\times 28\times 12$ | Conv2d (1, 64, 3, 1, 1)
Layer-1 | $64\times 28\times 12$ | Conv2d (64, 128, 3, 1, 1)
Layer-2 | $128\times 28\times 12$ | MaxPool2d (3, (1, 2))
Layer-3 | $128\times 26\times 5$ | Conv2d (128, 256, 3, 1, 1)
Layer-4 | $256\times 26\times 5$ | Conv2d (256, 256, 3, 1, 1)
Layer-5 | $256\times 26\times 5$ | Conv2d (256, 512, 3, 1, 1)
Layer-6 | $512\times 26\times 5$ | MaxPool2d (3, (2, 2))
Flatten | $512\times 12\times 2$ | flatten
Layer-7 | $12288$ | Linear (12288, 2048), ReLU
Layer-8 | $2048$ | Linear (2048, 256), ReLU
Layer-9 | $256$ | Linear (256, 128), ReLU
Per-frame Audio Feature | $128$ | -

##  Appendix D Analysis on Limitations and Future Work

As an early work towards audio-driven co-speech video generation without
structural prior, we notice that our method fails for some challenging cases.
For example, if the source image is in a large pose, it is hard to generalize
well in such out-of-domain data.

Notably, since our work does not involve any structural human body prior, it
is of high freedom without any human skeleton physical constraint. Therefore,
for some cases of large pose, the model may generate too large motion field
and lead to failure case results. We will explore how to develop a model with
higher generalization ability in future work.

Generalizing co-speech gesture avatars to more complex and general settings
like social conversation is a promising idea of great practical usage.
Currently, the biggest bottleneck could be the lack of high-quality
conversational dataset. Although CMU Panoptic contains the multi-view
conversation videos, the image quality is poor for co-speech image animation.
Besides, since the social co-speech gesture is more diverse, some model
designs like VQ codebook size should be well studied. We will delve into this
interesting problem in future work.

Besides, since our frames are preprocessed to resolution of $256\times 256$,
the scale and resolution of facial regions are rather small. We leverage
Wav2Lip [40] to sync the lip shape in the demo video for better visualization.
Note that such process substantially differs from the post-processing step of
[22, 41] that trains an extra pose2image generator: 1) The focus of co-speech
gesture generation task is to synthesize coherent upper body poses that are
aligned to speech audio, while the lip-sync is not the focus in this work. We
merely implement it for better visualization. 2) We could directly use the
off-the-shelf Wav2Lip [40] to sync the lip shape without any preprocessing
step, post-processing step or re-training. Therefore, it is in essence
different from an additional pose2image generator that requires tedious re-
training. 3) The task of simultaneously generating audio-consistent upper body
gestures and lip shapes is challenging and remains unsolved. The difficulty
lies in a) the low-resolution of mouth regions in the large-scale upper body
image makes it hard to capture the fine-grained lip-sync information; b) the
mouth shape only depends on the phoneme, while the co-speech gesture
correlates to the semantic meanings. We will explore this problem in future
work.

##  Appendix E Analysis on Generalization Ability

As shown in previous studies [2, 22], the co-speech gesture motions and styles
vary a lot for different speakers, which is termed as “individual speaking
style” [22]. Therefore, it is suitable to train a separate co-speech gesture
generation model for each person following the experiment settings of
baselines [2, 22, 41]. However, we explore a more challenging task of co-
speech gesture video generation in a unified framework without structural
prior and achieve superior performance. Even for a single-person subset, it is
non-trivial to animate non-rigid human body in image space by speech audio,
especially with the interference of complex background scenes.

It is hard to generalize to speakers that are not in the training data with
currently available co-speech gesture image datasets. In particular, the
commonly used datasets are TED Gesture [65] and PATS [2, 22]. TED Gesture is
based on TED Talk videos, while PATS contains 25 speakers of talk shows,
lectures, etc. Due to the frequent camera movements and viewpoint shift in TED
videos, there lacks clear co-speech gesture clips for image generation. Hence
we narrow down the experiments to PATS dataset in this work. A dataset with
high-quality co-speech gesture image frames of multiple speakers is needed to
learn a model of novel (unseen) person generalization ability. We will strive
for this in future work.

We verify the potential generalization ability of our approach: a) We could
animate different appearances of a speaker with speech audio (as shown in
codebook analysis part of the demo video, we could animate Oliver’s different
appearances), while previous studies [41, 22] that resort to off-the-shelf
pose2img generator only support a single appearance. b) We additionally
implement the experiments of animating with novel (unseen) audio. The
evaluated results are reported in Table 5. It shows that the model’s
performance is still effective with the input of unseen audio. With the
proposed vector quantize design, each codebook entry corresponds to a
reasonable co-speech gesture pattern. In contrast to directly mapping to the
continuous coordinate space, such technical design guarantees the valid
gesture even when generalizing to the unseen audio.

Table 5: Experiment results when generalize to unseen audio. Methods | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$
---|---|---|---
ANGIE (Novel Audio) | 1.46 | 0.69 | 48.5
ANGIE (Ours) | 1.35 | 0.72 | 49.4

##  Appendix F The Standard Deviation of User Study

We additionally provide the standard deviation of user study results in Table
6, which shows that the agreement among participants is highly consistent.

Table 6: The standard deviation of the user study results. The rating scale is 1-5, with the larger the better. We compare the Realness, Synchrony and Diversity to baselines [22, 34, 41, 65]. Methods | GT | S2G [22] | HA2G [34] | SDT [41] | TriCon [65] | ANGIE (Ours)
---|---|---|---|---|---|---
Realness | 0.492 | 0.413 | 0.293 | 0.206 | 0.385 | 0.312
Synchrony | 0.480 | 0.574 | 0.214 | 0.265 | 0.350 | 0.345
Diversity | 0.252 | 0.629 | 0.471 | 0.313 | 0.221 | 0.287

##  Appendix G Details on Audio Feature Extraction

Audio Onset Strength Feature. The audio onset strength feature of $T$ frames
is of shape $(T,426)$, where $T$ is the temporal dimension (frame number) and
$426$ is the feature channel dimension. It is the concatenation of constant-Q
chromagram, tempogram, onset beat, onset tempo and onset strength. Most
features are derived from the audio onset strength/envelope and the channel
dimension is summed up to $426$. We utilize the librosa onset functions to
extract the features. The audio sample rate is $16000$, the time lag for
computing differences is $1$, the hop length is $512$ and the window length is
$384$.

Audio MFCC Feature. The original audio mfcc was calculated as $12$ mfccs which
has the dimension of $(T^{\prime},12)$ where $T^{\prime}$ is the original
audio frame number. In our implementation, we use a $28$-dim sliding window to
further unfold the mfcc feature into a final shape of $(T,28,12)$, where $T$
denotes the final temporal dimension (video frame number), $28$ denotes the
size of sliding window and $12$ is the mfcc feature dimension. MFCC feature is
extracted with a sample rate of $16000$, window length of $25$ ms, window step
of $10$ ms, cepstrum number of $13$, filters number of $26$ and FFT block size
of $512$. In the motion refinement module, we use the a certain frame’s mfcc
feature of shape $(28,12)$ and forward a series of convolution and linear
layers to extract the per-frame audio feature of dimension $128$.

##  Appendix H Analysis on Extra Input Modality of Text Information

Our main focus and technical contributions are how to animate the co-speech
gesture in the image domain, but not on the influence of input modality. We
choose the audio-driven setting mainly for two reasons: a) We generally follow
the problem setting of baselines [2, 22, 41], where all the compared methods
take only the speech audio as input modality for fair comparison. b) We have
to pre-process the raw transcripts to be temporally aligned with audio using
tools like Gentle. To prevent the potential alignment inaccuracy in the pre-
processing step, and to simplify the problem setting for better focusing on
audio-driven co-speech gesture image animation, we do not involve the text
input in this work.

As proved in Automatic Speech Recognition (ASR) [67, 61] and recent co-speech
gesture studies [34], the speech audio actually contains some high-level
semantic information. Such implicit semantic information in the speech audio
could guide the model to capture some specific co-speech gesture patterns like
metaphorics [34]. Besides, as shown in the Kubinec chemistry lecture setting,
our model manages to synthesize deictic gestures of pointing to the screen by
learning such implicit audio-gesture correlations. We would like to
respectfully claim that the generated gestures of our model are not only
limited to the beat gestures, but some semantic gestures could also be
synthesized.

Typically, language model/text information contains rich semantics, which are
beneficial to the learning of semantics relevant gestures like iconics,
metaphorics, and deictics. Previous study [65] has also verified the influence
of each input modality on co-speech gesture, including audio, text and speaker
identity, etc. Therefore, we additionally complement an experiment of using
text feature: we encode the Gentle aligned transcripts by TextTCN [5] and
further concatenate with audio features. The combined audio text features are
fed to Co-Speech GPT with Motion Refinement network to predict the quantized
code as well as motion residuals. The results are reported in Table 7, which
suggest that the text feature could indeed facilitate the co-speech gesture
generation.

As an early attempt exploring audio-driven co-speech video generation, this
work could serve as a baseline for further studies in the research community.
However, how to effectively fuse the multiple modality information (including
audio and text) and better map to the implicit motion representation remains
an open problem. We will explore this in future work.

Table 7: Experiment results of concatenating text features. Methods | FGD $\downarrow$ | BC $\uparrow$ | Diversity $\uparrow$
---|---|---|---
ANGIE (Extra Text Concat) | 1.30 | 0.73 | 49.8
ANGIE (Ours) | 1.35 | 0.72 | 49.4

##  Appendix I Experimental Results with Error Bars

Since the Diversity metric involves randomness, which may fluctuate due to
different sampling clips. Therefore, we randomly sample $400$ pairs to get
robust evaluation results. To verify the steadiness, we conduct the evaluation
$5$ times (create random samples $5$ times with different random seeds). The
results are listed in Table 8. We can see that the fluctuation is rather
small, which shows the robustness of such evaluation.

Table 8: Randomness of the Diversity metric on the Oliver subset. Group | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
Diversity | 53.29 | 52.88 | 53.62 | 53.59 | 53.36

##  Appendix J User Study Details

The study involves $18$ participants of $9$ females and $9$ males. In
particular, the users are unaware of which generated result corresponds to
which method for a fair comparison. Before they rate the quality of
synthesized results, the participants will first be shown what the Ground
Truth (original raw video) is to help them make accurate scoring. The
participants are asked to judge the three perspectives of the generated
portraits: (1) Realness; (2) Synchrony; (3) Diversity. All participants are
paid for $15$ USD for about $40-60$ minutes’ rating process. We use the
Fleiss’s-Kappa statistic to measure the participants’ scoring disagreement,
which is a statistical measure for assessing the reliability of agreement
between a fixed number of raters when assigning categorical ratings to a
number of items or classifying items.

##  Appendix K Ethical Considerations

Our method could synthesize co-speech gesture images, which is envisioned to
facilitate extensive applications like digital human and human-machine
interaction. On the other hand, however, such techniques could be misused for
malicious purposes such as forgery generation. As part of our responsibility,
we strongly advocate all safeguarding measures against malicious use of co-
speech gesture images and feel obliged to share our generated results with the
deepfake detection community to improve the method’s robustness. We believe
that the proper use of this technique will enhance positive societal
development in both machine learning research and human’s daily entertainment.

##  Appendix L Discussions on Dataset Consent and Information

The PATS Image Dataset is collected and preprocessed based on the PATS
dataset. We download the YouTube videos by the provided video links, shared
under the “CC BY - NC - ND4.0 International” license. This license allows for
non-commercial use. Since all our code, data, models and results are kept for
academic use only, hence despite of the contained personally identifiable
information in the collected data, we will use them for research use only and
will not allow any commercial use.

##  Appendix M Assets License

In the Table 9, we list the licenses of all the existing assets we have used
in this work.

Table 9: Licenses of existing assets we have used in this work. Asset | License Link
---|---
MRAA [47] | https://github.com/snap-research/articulated-animation/blob/main/LICENSE.md
Librosa [36] | https://github.com/librosa/librosa/blob/main/LICENSE.md
Bailando [48] | https://github.com/lisiyao21/Bailando/blob/main/LICENSE
PATS [1, 2, 22] | https://github.com/chahuja/pats


* Meng  _et al._ [2023] X. Meng, Y. Cui, J. Gao, S. Havlin, and A. E. Ruckenstein, Deterministic entanglement distribution on series-parallel quantum networks, Phys. Rev. Res. 5, 013225 (2023).
  * Nielsen [1999] M. A. Nielsen, Conditions for a Class of Entanglement Transformations, Phys. Rev. Lett. 83, 436 (1999).
  * Żukowski  _et al._ [1993] M. Żukowski, A. Zeilinger, M. A. Horne, and A. K. Ekert, “Event-Ready-Detectors” Bell Experiment via Entanglement Swapping, Phys. Rev. Lett. 71, 4287 (1993).
  * Bennett  _et al._ [1996] C. H. Bennett, H. J. Bernstein, S. Popescu, and B. Schumacher, Concentrating partial entanglement by local operations, Phys. Rev. A 53, 2046 (1996).
  * Vidal [1999] G. Vidal, Entanglement of Pure States for a Single Copy, Phys. Rev. Lett. 83, 1046 (1999).
  * Duffin [1965] R. J. Duffin, Topology of series-parallel networks, J. Math. Anal. Appl. 10, 303 (1965).
  * Kesten  _et al._ [1980] H. Kesten _et al._ , The critical probability of bond percolation on the square lattice equals 1/2, Commun. Math. Phys. 74, 41 (1980).
  * Hill and Wootters [1997] S. Hill and W. K. Wootters, Entanglement of a Pair of Quantum Bits, Phys. Rev. Lett. 78, 5022 (1997).
  * Perseguers  _et al._ [2010] S. Perseguers, D. Cavalcanti, G. J. Lapeyre Jr., M. Lewenstein, and A. Acín, Multipartite entanglement percolation, Phys. Rev. A 81, 032327 (2010).
  * Pirandola [2019] S. Pirandola, End-to-end capacities of a quantum communication network, Commun. Phys. 2, 1 (2019).
  * Pant  _et al._ [2019] M. Pant, H. Krovi, D. Towsley, L. Tassiulas, L. Jiang, P. Basu, D. Englund, and S. Guha, Routing entanglement in the quantum internet, npj Quantum Inf. 5, 25 (2019).
  * Kobayashi  _et al._ [2010] H. Kobayashi, F. Le Gall, H. Nishimura, and M. Rötteler, Perfect quantum network communication protocol based on classical network coding, in  _2010 IEEE International Symposium on Information Theory_ (2010) pp. 2686–2690.
  * Cuquet and Calsamiglia [2009] M. Cuquet and J. Calsamiglia, Entanglement Percolation in Quantum Complex Networks, Phys. Rev. Lett. 103, 240503 (2009).
  * Lawler [1976] E. Lawler,  _Combinatorial Optimization: Networks and Matroids_ (Holtz, Rinehart and Winston, 1976).
  * Monma and Sidney [1979] C. Monma and J. Sidney, Sequencing with series-parallel precedence constraints, Math. Oper. Res. 4, 215 (1979).
  * Cohen  _et al._ [2000] R. Cohen, K. Erez, D. ben-Avraham, and S. Havlin, Resilience of the Internet to Random Breakdowns, Phys. Rev. Lett. 85, 4626 (2000).
  * Bern  _et al._ [1987] M. Bern, E. Lawler, and A. Wong, Linear-time computation of optimal subgraphs of decomposable graphs, J. Algorithms 8, 216 (1987).
  * Borie  _et al._ [1992] R. B. Borie, R. G. Parker, and C. A. Tovey, Automatic generation of linear-time algorithms from predicate calculus descriptions of problems on recursively constructed graph families, Algorithmica 7, 555 (1992).
  * Kikuno  _et al._ [1983] T. Kikuno, N. Yoshida, and Y. Kakuda, A linear algorithm for the domination number of a series-parallel graph, Discret. Appl. Math. 5, 299 (1983).
  * Takamizawa  _et al._ [1982] K. Takamizawa, T. Nishizeki, and N. Saito, Linear-time computability of combinatorial problems on series-parallel graphs, J. ACM 29, 623– (1982).
  * Baffi and Petreschi [1995] L. Baffi and R. Petreschi, Parallel maximal matching on minimal vertex series parallel digraphs, in  _Algorithms, Concurrency and Knowledge_ , edited by K. Kanchanasut and J. Lévy (Springer, Berlin, 1995) pp. 34–47.
  * Bertolazzi  _et al._ [1992] P. Bertolazzi, R. F. Cohen, G. Di Battista, R. Tamassia, and I. G. Tollis, How to draw a series-parallel digraph, in  _Algorithm Theory — SWAT ’92_ , edited by O. Nurmi and E. Ukkonen (Springer, Berlin, 1992) pp. 272–283.
  * Rendl [1986] F. Rendl, Quadratic assignment problems on series-parallel digraphs, Z. Oper. Res. 30, A161–A17 (1986).
  * Steiner [1985] G. Steiner, A compact labeling scheme for series-parallel graphs, Discret. Appl. Math. 11, 281 (1985).
  * Bang-Jensen and Gutin [2008] J. Bang-Jensen and G. Gutin,  _Digraphs: Theory, Algorithms and Applications_ (Springer, 2008).
  * Valdes  _et al._ [1982] J. Valdes, R. Tarjan, and E. Lawler, The recognition of series parallel digraphs, SIAM J. Comput. 11, 298 (1982).
  * Bodlaender and van Antwerpen de Fluiter [2001] H. Bodlaender and B. van Antwerpen de Fluiter, Parallel algorithms for series parallel graphs and graphs with treewidth twos, Algorithmica 29, 534 (2001).
  * Eppstein [1992] D. Eppstein, Parallel recognition of series-parallel graphs, Inform. and Comput. 98, 41 (1992).
  * He [1991] X. He, Efficient parallel algorithms for series parallel graphs, J. Algorithms 12, 409 (1991).
  * He and Yesha [1987] X. He and Y. Yesha, Parallel recognition and decomposition of two terminal series parallel graphs, Inform. and Comput. 75, 15 (1987).
  * Kron [1939] G. Kron,  _Tensor Analysis of Networks_ (Wiley, 1939).
  * Versfeld [1970] L. Versfeld, Remarks on star-mesh transformation of electrical networks, Electron. Lett. 6, 597 (1970).
  * Malik  _et al._ [2022] O. Malik, X. Meng, S. Havlin, G. Korniss, B. K. Szymanski, and J. Gao, Concurrence percolation threshold of large-scale quantum networks, Commun. Phys. 5, 1 (2022).
  * Krattenthaler [2015] C. Krattenthaler, Lattice path enumeration, arXiv:1503.05930 (2015).
  * Jensen [2004] I. Jensen, Enumeration of self-avoiding walks on the square lattice, J. Phys. A 37, 5503 (2004).
  * Roberts and Kroese [2007] B. Roberts and D. P. Kroese, Estimating the number of s-t paths in a graph, J. Graph Algorithms Appl. 11, 195 (2007).
  * Erdős and Rényi [1960] P. Erdős and A. Rényi, On the evolution of random graphs, in  _Publications of the Mathematical Institute of the Hungarian Academy of Sciences_ (1960) pp. 17–61.
  * Treinish  _et al._ [2023] M. Treinish, J. Gambetta, S. Thomas, P. Nation, qiskit-bot, P. Kassebaum, D. M. Rodríguez, S. d. l. P. González, J. Lishman, S. Hu, L. Bello, J. Garrison, K. Krsulich, J. Huang, J. Yu, M. Marques, J. Gacon, D. McKay, E. Arellano, J. Gomez, L. Capelluto, Travis-S-IBM, A. Mitchell, A. Panigrahi, lerongil, R. I. Rahman, S. Wood, T. Itoko, A. Pozas-Kerstjens, and C. J. Wood, Qiskit/qiskit: Qiskit 0.42.0, Zenodo (2023).
  * Calderon-Vargas  _et al._ [2019] F. A. Calderon-Vargas, G. S. Barron, X.-H. Deng, A. J. Sigillito, E. Barnes, and S. E. Economou, Fast high-fidelity entangling gates for spin qubits in Si double quantum dots, Phys. Rev. B 100, 035304 (2019).
  * Kandala  _et al._ [2021] A. Kandala, K. X. Wei, S. Srinivasan, E. Magesan, S. Carnevale, G. A. Keefe, D. Klaus, O. Dial, and D. C. McKay, Demonstration of a High-Fidelity cnot Gate for Fixed-Frequency Transmons with Engineered ZZ Suppression, Phys. Rev. Lett. 127, 130501 (2021).
  * Bantysh  _et al._ [2021] B. I. Bantysh, A. Y. Chernyavskiy, and Y. I. Bogdanov, Quantum tomography benchmarking, Quantum Inf. Process. 20, 339 (2021).