# Simple interferometric setup enabling sub-Fourier-scale ultra-short laser
pulses

Enrique G. Neyra, Gustavo A. Torchia, Pablo Vaveliuk and Fabian Videla1,2 1
Centro de Investigaciones Ópticas (CICBA-CONICET-UNLP), Cno. Parque Centenario
y 506, P.O. Box 3, 1897 Gonnet, Argentina  2 Departamento de Ciencias Básicas,
Facultad de Ingeniería UNLP, 1 y 47 La Plata, Argentina
<EMAIL_ADDRESS>
###### Abstract

In this work, we describe an interferometric method to generate ultra-short
pulses below the Fourier limit. It is done by extending concepts first
developed in the spatial domain to achieve sub-diffractive beams through the
addition of a spatial chirp in one of the arms of a Michelson interferometer
using a spherical mirror. To experimentally synthesize sub-Fourier pulses, we
replace the spherical mirror with a water cell, since it produces chirp in the
temporal domain. We also present an alternative procedure, based on
asymmetrical interference between the widened pulse and the original pulse
where the peaks of both pulses exhibit a temporal delay achieving the
narrowing of ultra-short pulses with sub-Fourier scales.

To characterize the performance of the system, we performed a preliminary
assessment considering the percentage of FWHM shrinking obtained for each
scheme. By means of a symmetrical configuration 7 and 12 % pulse reductions
were verified, both experimentally and analytically, while for the non-
symmetrical configuration 10 and 24% reductions were achieved corresponding to
main lobe to side-lobes ratios of 10 and 30% .

The experimental setup scheme is simple, versatile and able to work with high-
power laser sources and ultra-short pulses with a broad bandwidth at any
central wavelength. The results presented in this work are promising and help
to enlighten new routes and strategies in the design of coherent control
systems. We envision that they will become broadly useful in different areas
from strong field domain to quantum information.

##  1 Introduction

The temporal width of an ultra-short laser pulse $E(t)$, is given by the
distribution of the spectral phase $\phi(\omega)$ in the Fourier spectral
components of the field $E(\omega)$. The so-called transform-limited pulses
condition is fulfilled if the spectral phase of the field is independent of
the frequency. This condition implies that the temporal and spectral widths
are minimum and in turn, its product corresponds to a particular constant in
agreement with the pulses shapes. Considering a Gaussian pulse, and their
temporal and spectral full width at half-maximum (FWHM) , this product yields
: $\Delta\omega$ $\Delta t=2\pi c$ with $c=0.441$ [1]. Of course, the c value
changes for other pulse shapes.

The Fourier limit is tied to the quantum Heisenberg uncertainty principle
since the conjugate variables of both phenomena are related by the Fourier
transform. In contrast, if the pulse does not meet this condition, i.e. the
spectral phase $\phi(\omega)$ is frequency dependent, the pulse presents chirp
and the FWHM of the pulse is larger than the FWHM of the pulse without a
chirp.

In this sense, there has been an emerging interest in recent years for a wave
phenomenon called superoscillation [2, 3], in which a band-limited function
can oscillate faster than their fastest Fourier component.

The superoscillatory phenomenon can be seen as an interferometric one, where
each spectral component has a particular phase that originate, in the
conjugate space, small regions whose FWHMs are below the Fourier limit.
Additionally, those regions where the Fourier limit is ”broken”, show poor
efficiency about the initial pulse. In addition, this phenomenon coexists with
the emerging of side-lobes with relative amplitude [4].

The superoscillatory phenomenon is not only an interesting mathematical topic.
Several wave phenomena take advantage of superoscillations such as: Focusing
different kinds of beams for superresolution microscopy [5], in quantum
mechanics [6, 7], in acoustic waves [8], in signals [9], and others[3].
Additionally, for ultra-short pulses, the Fourier limit can be overcome
through devices like the spatial light modulator(SLM) in a prism based pulse
shaper [10], which generates the desired phase mask [11, 12, 13, 14, 15].
However, this experimental setup requires a complex process of alignment and
control, complemented with cumbersome numerical tools. On the other hand,
limitations in the beam shrinking process using pulse shapers (SLM’s), arise
specially from the chromatics aberrations because of the bandwidth of short
pulses. On the other hand pulse shapers need of phase calibration. Likewise,
must be mentioned that SLM’s presents aberrations given by their pixel size
[13].

It is an accepted notion that the minimum reachable FWHM is associated with a
constant spectral phase, but this is not strictly true. The clearest example
is put forward by a $\pi$-phase mask where sub-Fourier pulses were obtained
[13]. Additionally, the ability to manipulate the shape of the optical pulses
is essential for numerous applications [16] such as spectroscopy and coherent
control [17, 18], metrology [19] , microscopy [20] and optical communications
[21].

In this article, we propose a robust, versatile, and simple method to achieve
sub-Fourier ultra-short laser pulses. The current approach is inspired by the
spatial narrowing of a focused beam, recently proposed [4]. In that work, to
print a quadratic phase in the beam spatial profile, a spherical mirror was
introduced modifying a branch of a Michelson interferometer. In this work to
achieve a quadratic phase in the beam but now in the temporal domain, we
replaced the spherical mirror with a water cell (dispersive media) introducing
chirp on the original pulse, As a result, a temporally widened pulse is
obtained after the cell. To assess the performance of the proposed
experimental scheme, we developed two experiments.The first one is based on a
symmetrical interference between the widened pulse and the original one, that
is, when the peaks of both pulses coincide. The second one, is based in an
asymmetrical interference between the widened pulse and the original pulse,
where the peaks of both pulses exhibit a temporal delay.

##  2 Theoretical framework

In order to mathematically describe the interfering process, we call $E(t)$
the pulse originated as a result of the superposition of pulse 1 (unaltered
pulse) and pulse 2 (altered pulse) after passing through the interferometric
device. Pulse 1 is a band-limited ultra-short Gaussian pulse $E_{0}(t)$, with
spectral content
$E_{0}(\omega)=E_{0}e^{-((\omega-\omega_{0})/\delta\omega)^{2}}$, centered at
the carrier frequency $\omega_{0}$ whose bandwidth is represented by
$\delta\omega$.Therefore, the whole field in the frequency domain, composed by
the unaltered and the altered pulse is given by:

$E(\omega)=E_{0}(e^{-(\frac{\omega-\omega_{0}}{\delta\omega})^{2}}+\alpha
e^{-(\frac{\omega-\omega_{0}}{\delta\omega})^{2}}e^{-i\beta(\omega-\omega_{0})^{2}}e^{i\phi})$
(1)

Clearly, the phase of pulse 2 is modified by the chirp parameter $(\beta)$, in
turn the amplitude is modified by $(\alpha)$, a factor that weighs the
relative amplitude between the arms. Besides, there is a third controllable
parameter, $(\phi)$, representing the relative phase between both arms of the
interferometer. The Fourier transform of the Eq. (1) returns the narrowed
pulse in the time domain:

$E(t)=e^{-\frac{1}{4}t^{2}}e^{i\omega_{0}t}+\frac{\alpha}{(1+\beta^{2})^{1/4}}e^{-\frac{1}{4(1+\beta^{2})}t^{2}}e^{i\Phi(t)}e^{i\omega_{0}t},$
(2)

where the $t$-varying phase is:

$\Phi(t)=\frac{\beta}{4(1+\beta^{2})}t^{2}-\frac{1}{2}\arctan{(\beta)}+\phi.$
(3)

The Fourier transform was evaluated taken $\delta\omega$ and $E_{0}$ equal to
the unit, for a sake of clarity to visualize the technique highlights.

Explicitly, Eq. (2) shows the interference between a Fourier-transform-limited
pulse $E_{0}(t)=e^{-\frac{1}{4}t^{2}}$ and the widened version of this pulse,
broadened by the factor $\tau=\sqrt{1+\beta^{2}}$ with relative amplitude
$\alpha/(1+\beta^{2})^{1/4}$ and phase $\Phi(t)$.

It is well-known that the chirp quadratic parameter $\beta$, in the temporal
domain, is originated when the pulse travels through a dispersive medium. This
parameter is related to the group velocity dispersion (GVD) by
$\beta=\frac{1}{2}GVD\times L$, where $L$ is the length of the dispersive
media [1]. The pulse intensity $I(t)$ is then obtained by multiplying Eq.2 by
its complex conjugate to give:

$I(t)=e^{-\frac{1}{2}t^{2}}+\frac{\alpha^{2}}{\sqrt{1+\beta^{2}}}e^{-\frac{t^{2}}{2(1+\beta^{2})}}+\frac{2\alpha}{(1+\beta^{2})^{1/4}}e^{-\frac{t^{2}}{4}(1+\frac{1}{(1+\beta^{2})})}\cos{\Phi}$
(4)

Equation (4) represents the temporal intensity profile of the field $E(t)$,
that for a suitable choice of the external parameters $\beta$, $\alpha$ and
$\phi$, make it possible to shape a pulse whose central lobe shows a temporal
width below the Fourier limit, i.e. narrower than the reference pulse
$E_{0}(t)$. For this purpose, a pure destructive interference condition
between the pulses was necessary, which was achieved for values of the phase
$\phi=\phi(\alpha,\beta)$, $\alpha$ and $\beta$. A complete theoretical
description of this interference was made in Ref [4]. There, in the same plot,
the reduction of the FWHM are represented as a function of $\alpha$ and
$\beta$, as well as, curves bounding regions for different ranges of the ratio
between the side-lobes and the central lobe. Superimposed on the previous
regions a third region with a high efficiency for the central lobe, a trade-
off between the previous zones, making the performance of the technique
suitable for each application.

##  3 Experimental

###  3.1 Setup

The experimental setup used to carry out the proposed approach to synthesize
narrower pulses is sketched in Figure 1. The ultra-short pulse input into a
Michelson Interferometer (MI), then was split into two pulses by means of a
beam-splitter (BS). Pulse 1 traveled through branch 1 of the interferometer
and then was reflected by the fixed mirror (M1), remaining unaltered. The
pulse 2 traveled through branch 2, this arm undergoing the influence of the
different control parameters, $\alpha$, $\beta$ and $\phi$. Mirror (M2) was
mounted in a micro-meter translation stage and controls the relative phase
$\phi$ between the interferometer arms. During the first experiment, MI was
set as a balanced interferometer, while for the second a delay line $\tau_{D}$
was activated to accomplish a non-symmetrical interference. In branch 2, pulse
2 was stretched by passing it through a 6 cm long water cell, due to the group
velocity dispersion ($\beta$ parameter, GVD).The path through the cell
(indicated as E) is shown in Fig.1. In the same branch, pulse 2 passes through
a half-wave plate (HWP) and a polarizer (P) system to control the relative
intensity between both pulses ($\alpha$ parameter), device D in Fig. 1.
Finally, when the two pulses are reflected by the mirrors ( the pulse cross
again the water cell), the resulting interference is detected by a GRENOUILLE
system [22]. GRENOUILLE can be considered as an implementation of FROG
(Frequency Resolved Optical Gating) but without mobile parts, and was
indicated by F in the setup. Its function was to characterise the temporal
profile of the pulse synthesis. The laser system employed to test our
experimental approach was a Ti:Sapphire femtosecond oscillator (S), Mai Tai
model from Spectra Physics (USA), characterised by a spectral bandwidth of
10:5nm, with a band-limited pulse of FWHM $\approx$ 90fs (see inset of Fig.
1).

Figure 1: Our experimental setup for synthesise pulses is based on a Michelson
Interferometer (MI). S is the femtosecond laser source, a Ti:Sapphire
femtosecond oscillator. M1 and M2 are the fixed and mobile mirrors
respectively. BS is the beam splitter. In the delay line, the water cell E
(dispersive medium) is followed by a half-wave plate (HWP) plus a polarizer,
D, at the opposite end, the GRENOUILLE system F. It was indicated the movable
part in the arm for $\tau_{D}$ control

###  3.2 Results and discussions

First of all, we measured the different pulses widths, which are represented
in Figure 2. Panel (a) shows the FROG trace for the pulse into branch 1, while
Fig 2(b) shows the temporal intensity profile and its phase retrieved by the
FROG trace, with temporal width of FWHM=89.8fs. Panel (c) of Figure 2 shows
the measured FROG trace corresponding to the stretched pulse by propagation
through the water cell (branch 2). Panel (d) presents the FROG retrieved
temporal intensity and phase profile of the pulse,on which an FWHM can be
observed of 199,4fs. By properly setting units and considering
$\delta\omega=1$ the relationship between the FWHMs of the pulses obtained in
each branch was $\sqrt{1+\beta^{2}}$ (see Eq.2). In this way the $\beta$ value
could be solved as $\frac{199.4fs}{89.8fs}=\sqrt{1+\beta^{2}}$ which results
in $\beta=1.99$ (in fractions of $\delta\omega$).

Figure 2: FROG traces corresponding to pulses traveling in each arm of the
Michelson Interferometer (left side). Retrieved temporal profiles for the
pulses in each arm of the MI. (right side).

Fig 3 presents the synthesized pulses obtained with the interferometric
technique described previously. The different pulses widths showed in panels
(b) and (d) were obtained by varying the $\alpha$ parameter handling a half-
wave plate plus a polariser system, detailed in the experimental procedure. In
Figure 3, panels (a) and (b) show the FROG trace and its retrieved pulse
profile, respectively. The result of the interference was measured by setting
$\alpha=0.25$. The FWHM retrieved was 84 fs. corresponding to a reduction of
the Fourier limit of 7 percent for the input pulse. On the other hand, in
Figures 3(c) and 3(d), the second set of measurements is shown. In this case,
the interference pulse was conducted by setting $\alpha=0.38$. The retrieved
FWHM was 79 fs, corresponding to a reduction of 12% of the original pulse
limited by Fourier transform.

Figure 3: FROG traces corresponding to synthesised pulses with parameters
$\alpha=0.25$, $\beta=1.99$ and $\alpha=0.38$, $\beta=1.99$ are sketched in
panels (a) and (c), respectively. In panels (b) and (d) the temporal profile
of the Frog traces (a) and (c), where it can be seen a FWHM of 84 fs and 79 fs
respectively. In (e), evolution of FWHM of the synthesised pulses vs $\alpha$
(the amplitude ratio of the interfering pulses), in fractions of the unaltered
FWHM pulse considered as unit, so $\tau_{0}=1$. The chirp parameter $\beta$
was held constant at 1.99. The two red crosses correspond to the measured
experimental data.

In a complementary fashion, Fig 3 (e) shows the evolution of the FWHM
corresponding to the central lobe for the synthesised pulse as a function of
the $\alpha$ parameter (solid line). The FWHM scale was graduated in reference
to the temporal FWHM of the input pulse. This simulation, described by
equation 4, was made considering a fixed value of the $\beta$ parameter equal
to 1.99. The first region from $\alpha$=1 to $\alpha$=0.8, shows a linear
relation between FWHM and $\alpha$. A second region, for $\alpha>1$, presents
a fast reduction of the FWHM. By calculation of equation (4), for large values
of $\alpha$, the synthesised pulse shows side-lobes with larger amplitude
compared to the central lobe, as well as when the FWHM is close to zero value.
As described in [23], each side-lobe has a different spectral content and a
complete theoretical analysis can be found in Ref. [4]. In Figure 3,
experimental points have been indicated by red crosses. In this case, it is
important to mention that we could not take measures for large values of
$\alpha$, to verify the theoretical curve (Fig 3(e)), because of the technical
limitations of the GRENOUILLE system. When measures were made with large
values of $\alpha$, interference rings are generated in the wave-front of the
synthesised pulse. This could be possibly explained by the additional
divergence of the pulse caused when passing through the water cell. It is
necessary to work with Gaussian or continuous wave-fronts to achieve a proper
operation of the GRENOUILLE system [22].

Alternatively to the above method, we propose another way to produce similar
sub-Fourier synthesized pulses controlling the delay times $\tau_{D}$ and the
attenuation parameter $\alpha$ by using the same Michelson interferometer. In
this case, the method is based on a non-symmetrical interference in the time
domain. As we show, Eq. 1 represents the interference between an ultra-short
and a widened pulse centred in $t=0$, which produce a synthesised pulse with
sub-Fourier characteristics. The proposed alternative can be achieved if the
two pulses traveling in each arm of the MI, the original and widened pulse,
have a delay time $\tau_{D}$ at the moment of the interference. We will show
that changes in the delay of the interferometer branches produce similar
behavior that only changes the $\alpha$ parameter. Both lead to sub-Fourier
pulses.

In Fig. 4 two measurements corresponding to different delay times ($\tau_{D}$)
are presented (see Fig. 1), considered as the distance between peaks of the
original and widened pulses, respectively (see fig(4)). The chosen values are
: $\tau_{D}=31.5$fs($\alpha=0.85$) and $\tau_{D}=49.5$fs ($\alpha=0.5$), top
and bottom figures, respectively. In panels 4(a) and 4(b) at the top of Fig. 4
show the FROG trace and the retrieved temporal profile corresponding to a
delay time of $31.5$fs and $\alpha=0.85$; the synthesised pulse retrieved had
an FWHM=68fs ($24\%$ of the Fourier limit reduction). In the middle of Fig 4.
panels 4(c) and 4(d) respectively show the FROG trace and the retrieved
temporal profile corresponding to a delay of $49.5$fs and $\alpha=0.5$, giving
a resultant pulse with an FWHM=81fs ($10\%$ Fourier limit reduction). Note
that $\alpha$ is the relation between the maximum amplitude of both fields.
When different delays are introduced the peak amplitudes are shifted,
resulting in an effective amplitude ratio between pulses smaller than
$\alpha$, since the interference occurs while the peak of the unaltered pulse
overlaps with the tail of the widened pulse. At the bottom of the figure,
panels 4(e) and 4 (f) show the respective simulations of the measurements. The
blue line designates the original pulse and in green the widened pulse, with
the delay $\tau_{D}$. The synthesised pulse after the interference is shown by
the red dashed line. Here, the asymmetric distribution of the two side-lobes
can be observed and are in good agreement with the experimental measurements.

This last alternative to obtain sub-Fourier pulses allows modulating the
temporal width of the pulse as a function of the temporal delay $\tau_{D}$.
This process of FWHM modulation can be described through the next sequence:
suppose we began with two pulses temporally spaced by a large delay
$\tau_{D}$, which is progressively reduced through the movement of the micro-
positioning stage. After that, the Fourier limited pulse will experience a
reduction of FWHM. The minimum FWHM is reached when a maximum of the
destructive interference condition occurs. If $\tau_{D}$ is reduced yet more,
the Fourier limited pulse will begin to increase again. Depending on the FWHM
of the widened pulse, this process is repeated once, twice or $n$ times. In
consequence, the temporal width of the initial pulse, will experience a
sinusoidal variation with the $\tau_{D}$ parameter.

For a better understanding and visualisation of the previous description, two
simulations were uploaded [ref] as supplementary material. Each of them
represents, qualitatively, the change of the FWHM when the delay $\tau_{D}$
varies. Both set the $\alpha$ parameter as one while the $\beta$ parameter as
5 and 10, respectively. Simulations, carefully observed, shows an FHWM
oscillatory behaviour.

Finally, it is worth mentioning that the experimental results presented in
this work are limited by our detection system (GRENOUILLE). We think that a
complete set of measurements, predicted theoretically in Ref. [4], can be
achieved by another detecting system like FROG or SPIDER, as other authors
have shown [11, 14] since in the conjugate space, the different techniques are
similar. In our case, the phase mask necessary to obtain sub-Fourier pulses is
given by the operator $H(\omega)=(1+\alpha
e^{i\beta(\omega-\omega_{0})}e^{i\phi})$ (see Eq.1), either implemented by a
SLM or an interferometric system as in our case.

Figure 4: FROG traces corresponding to synthesised pulses for delays of 31.5
fs and 49.5 fs are sketched in panels (a) and (c), respectively. In panels (b)
and (d) the temporal counterpart of (a) and (c) respectively. On the bottom
panels (e) and (f), detailed for each delay, the resultants pulses (red dashed
line). In blue solid and green solid lines, the pulses traveling through
branches 1 and 2

##  4  Conclusions

In conclusion, two simple approaches we have presented, supported by
experimental results, to obtain ultra-short pulses where whose FWHMs are below
the Fourier limit. This technique, based on an interferometric process, shows
some advantages when compared to those commonly used for this purpose,
including the capability to work with high power laser sources and operate
with ultra-bandwidth pulses centered at any wavelength. The performance of the
system can be summarized taking each scheme into account . For attenuation
based scheme 7 and 12 % of FWHM reduction were both experimentally and
analytically verified, while for phase control scheme 10 and 24% of FWHM
reduction was achieved with main lobe to side-lobes ratios of 10 and 30%
respectively. The limitations can be associated with different devices: beam-
splitter, mirrors, etc. which are affected by thermal fluctuations, mechanical
vibrations, the fluence threshold of mirrors, etc., which are intrinsic
properties of a Michelson interferometer. The technique, theoretically, works
without constraints for arbitrary pulse lengths, however, experimentally it is
difficult to implement the quadratic chirp parameter. For very long pulses the
dispersive media can be an optical fiber. Alternatively, for short pulses in
the few-cycle regime, other dispersive orders (third order, fourth order, etc)
must be taken into account. These high orders could be compensated with a
spherical mirror [24]. We think, that this new strategy for pulse synthesis
could open a new route to address ultra-fast spectroscopy, as well as a
different coherent control process, from weak field to strong field domains.

## Acknowledgments

We wish to thanks Dr. Paula Gago from the Faculty of Engineering Imperial
College UK. E N, G T and P V are with CONICET. F V belongs to the Comisión de
Investigaciones Científicas (Buenos Aires, Argentina).

## References

## References

  * [1] Diels J C and Rudolph W 2006 Ultrashort laser pulse phenomena (Elsevier)
  * [2] Aharonov Y, Colombo F, Sabadini I, Struppa D and Tollaksen J 2011 Journal of Physics A: Mathematical and Theoretical 44 365304
  * [3] Berry M, Zheludev N, Aharonov Y, Colombo F, Sabadini I, Struppa D C, Tollaksen J, Rogers E T, Qin F, Hong M et al. 2019 Journal of Optics 21 053002
  * [4] Neyra E G and Vaveliuk P 2021 Journal of Optics 23 075604
  * [5] Gbur G 2019 Nanophotonics 8 205–225
  * [6] Remez R, Tsur Y, Lu P H, Tavabi A H, Dunin-Borkowski R E and Arie A 2017 Physical Review A 95 031802
  * [7] Berry M and Popescu S 2006 Journal of Physics A: Mathematical and General 39 6965
  * [8] Shen Y X, Peng Y G, Cai F, Huang K, Zhao D G, Qiu C W, Zheng H and Zhu X F 2019 Nature communications 10 1–7
  * [9] Zarkovsky S, Ben-Ezra Y and Schwartz M 2020 Scientific reports 10 1–7
  * [10] Weiner A M 2000 Review of scientific instruments 71 1929–1960
  * [11] Boyko O, Valentin C, Rey G, Antonucci L, Balcou P and Coudreau S 2005 Optics express 13 8222–8230
  * [12] Binhammer T, Rittweger E, Morgner U, Ell R and Kärtner F 2006 Optics letters 31 1552–1554
  * [13] Eliezer Y, Singh B K, Hareli L, Arie A et al. 2018 Optics express 26 4933–4941
  * [14] Eliezer Y, Hareli L, Lobachinsky L, Froim S and Bahabad A 2017 Physical review letters 119 043903
  * [15] Rausch S, Binhammer T, Harth A, Kärtner F X and Morgner U 2008 Optics express 16 17410–17419
  * [16] Keller U 2003 nature 424 831–838
  * [17] Goswami D 2003 Physics Reports 374 385–481
  * [18] Silberberg Y 2009 Annual review of physical chemistry 60 277–292
  * [19] Tanabe T, Tanabe H, Teramura Y and Kannari F 2002 JOSA B 19 2795–2802
  * [20] Bardeen C J, Yakovlev V V, Squier J A, Wilson K R, Carpenter S D and Weber P M 1999 Journal of Biomedical Optics 4 362–367
  * [21] Sardesai H, Chang C C and Weiner A 1998 Journal of Lightwave Technology 16 1953
  * [22] Akturk S, Kimmel M, O’Shea P and Trebino R 2003 Optics Express 11 491–501
  * [23] Neyra E, Videla F, Ciappina M, Pérez-Hernández J, Roso L and Torchia G A 2018 Physical Review A 98 013403
  * [24] Reitze D, Weiner A and Leaird D 1992 Applied physics letters 61 1260–1262


# Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image
Animation

Mingwang Xu1∗ Hui Li1∗ Qingkun Su1∗ Hanlin Shang1 Liwei Zhang1 Ce Liu3
Jingdong Wang2 Yao Yao4 Siyu Zhu1
1Fudan University 2Baidu Inc 3ETH Zurich 4Nanjing University

###### Abstract

The field of portrait image animation, driven by speech audio input, has
experienced significant advancements in the generation of realistic and
dynamic portraits. This research delves into the complexities of synchronizing
facial movements and creating visually appealing, temporally consistent
animations within the framework of diffusion-based methodologies. Moving away
from traditional paradigms that rely on parametric models for intermediate
facial representations, our innovative approach embraces the end-to-end
diffusion paradigm and introduces a hierarchical audio-driven visual synthesis
module to enhance the precision of alignment between audio inputs and visual
outputs, encompassing lip, expression, and pose motion. Our proposed network
architecture seamlessly integrates diffusion-based generative models, a UNet-
based denoiser, temporal alignment techniques, and a reference network. The
proposed hierarchical audio-driven visual synthesis offers adaptive control
over expression and pose diversity, enabling more effective personalization
tailored to different identities. Through a comprehensive evaluation that
incorporates both qualitative and quantitative analyses, our approach
demonstrates obvious enhancements in image and video quality, lip
synchronization precision, and motion diversity. Further visualization and
access to the source code can be found at: https://fudan-generative-
vision.github.io/hallo.

††footnotetext: ∗ indicates equal contribution. Figure 1:  The proposed
methodology aims to generate portrait image animations that are temporally
consistent and visually high-fidelity. This is achieved by utilizing a
reference image, an audio sequence, and optionally, the visual synthesis
weight in conjunction with a diffusion model based on the hierarchical audio-
driven visual synthesis approach. The results of this method showcase improved
fidelity and visual quality in comparison to previous approaches that rely on
intermediate facial representations. Furthermore, the proposed methodology
enhances the accuracy of lip synchronization measurement and enhances the
control over motion diversity.

##  1 Introduction

Portrait image animation, also known as talking head image animation, aims to
generate a speaking portrait from a single static image and a corresponding
speech audio. This technology holds substantial value across various domains
such as video gaming and virtual reality, film and television production,
social media and digital marketing, online education and training, as well as
human-computer interaction and virtual assistants. Significant advancements in
this field are exemplified by works like Stable Diffusion [32] and DiT [28],
which involve the gradual incorporation of noise into the training data in the
latent space, followed by a reverse process that progressively reconstructs
the signal from this noise. By leveraging parametric or implicit
representations of lip movements [29, 53], facial expressions [14, 25, 35],
and head poses [25, 38, 39], combined with diffusion techniques in the latent
space [7, 23, 40, 47, 48], it is possible to generate high-quality, lifelike
dynamic portraits in an end-to-end manner.

This study focuses on two primary challenges in portrait image animation: (1)
The synchronization and coordination of lip movements, facial expressions, and
head poses driven by the audio input. (2) The creation of high-fidelity
animations that are visually appealing and maintain temporal consistency.
Addressing the first challenge, parametric model-based approaches [7, 25, 29,
35, 39, 47, 53] initially rely on input audio to derive sequences of facial
landmarks or parametric motion coefficients, such as 3DMM [2], FLAME [20] and
HeadNeRF [17], as intermediate representations, which are subsequently used
for visual generation. However, the effectiveness of these methods is
constrained by the accuracy and expressiveness of the intermediate
representations. In contrast, another approach involves decoupled
representation learning in latent space [14, 23, 48], which segregates facial
features into identity and non-identity components within a latent space.
These components are independently learned and integrated across frames. The
main obstacle in this approach is effectively disentangling the various
portrait factors while encompassing static and dynamic facial attributes
comprehensively. Moreover, the introduction of intermediary representations or
decoupled representations in the latent space of the aforementioned methods
undermine the production of temporally consistent visual outputs with high
realism and diversity.

To generate visually appealing and temporally consistent high-quality
animations, the utilization of end-to-end diffusion models [40] leverages
recent advancements in diffusion models for synthesizing portrait videos
directly from images and audio clips. This approach eliminates the need for
intermediate representations or complex preprocessing steps, enabling the
production of high-resolution, detailed, and diverse visual outputs. However,
achieving precise alignment between audio and the generated facial visual
synthesis, including lip movements, expressions, and poses, still remains a
challenge. In this study, we adhere to the framework of end-to-end diffusion
models with the objective of addressing the alignment issue while
incorporating the ability to control the diversity of expressions and poses.
To tackle this challenge, we introduce a hierarchical audio-driven visual
synthesis module that employs cross-attention mechanisms to establish
correspondences between audio and visual features associated with lips,
expressions, and poses. Subsequently, these cross-attentions are fused using
adaptive weighting. Building upon this hierarchical audio-driven visual
synthesis module, we propose a network architecture that integrates a
diffusion-based generative model with a UNet-based denoiser [33], temporal
alignment [12] for sequence coherence, and a ReferenceNet to guide visual
generation. As shown in Figure 1, this integrated approach enhances lip and
expression movements to synchronize effectively with the audio signal and
offers adaptive control over the granularity of expressions and poses, thereby
ensuring consistency and realism across various visual identities.

In the experimental section, our proposed approach is comprehensively
evaluated from both qualitative and quantitative perspectives, encompassing
assessments of image and video quality, lip synchronization, and motion
diversity. Specifically, our method demonstrates a significant enhancement in
image and video quality compared to previous methodologies, as quantified by
the FID and FVD metrics. Moreover, our approach exhibits a promising
advancement in lip synchronization when contrasted with previous diffusion-
based techniques. Furthermore, our method offers the flexibility to adjust the
diversity of expressions and poses according to specific requirements. It is
noteworthy that we are dedicated to the dissemination of our source code and
sample data to the open-source community, with the aim of facilitating further
research in related fields.

##  2 Related Work

#### Diffusion-Based Video Generation.

Recent advancements in diffusion-based video generation have shown promising
results [3, 8, 13, 16, 19, 24, 49, 57], leveraging the foundational principles
of text-to-image diffusion models. Notable efforts include Video Diffusion
Models (VDM) [16], which utilize a space-time factorized U-Net for
simultaneous image and video data training, and ImagenVideo [15], which
enhances VDM with cascaded diffusion models for high-definition outputs. Make-
A-Video [37] and MagicVideo [55] extend these concepts to facilitate seamless
text-to-video transformations. For video editing, techniques like VideoP2P
[22] and vid2vid-zero [45] manipulate cross-attention maps to refine outputs,
while Dreamix [26] employs image-video mixed fine-tuning. Additionally, Gen-1
[10] integrates structural guidance through depth maps and cross-attention,
whereas MCDiff [6] and LaMD [18] focus on motion-guided video generation,
enhancing the realism of human actions. VideoComposer [46], AnimateDiff [5],
and VideoCrafter [5] further explore the synthesis of image-to-video
generation by conditioning the diffusion process on images and blending image
latents. Given the remarkable efficacy of diffusion models in achieving high-
fidelity and temporally consistent visual generation outcomes, an end-to-end
diffusion model is adopted in this study.

#### Facial Representation Learning.

Learning facial representations in the latent space, particularly disentangled
from identity-related appearances and non-identity-related motions such as lip
movements, expressions, and poses, represents a significant challenge in the
field of computer vision. Representation learning related to this challenge
generally falls into two categories: explicit and implicit methods. Explicit
methods often employ facial landmarks, which are key points [36, 51] on the
face used for localization and representation of critical regions such as the
mouth, eyes, eyebrows, nose, and jawline. Additionally, work involving 3D
parametric models [11, 31, 52], like the 3D morphable Model (3DMM), captures
the variability in human faces through a statistical shape model coupled with
a texture model. However, these explicit methods are limited by their
expressive capabilities and the precision of reconstructions. Conversely,
implicit methods aim to learn disentangled representations in 2D [4, 21, 27,
42, 50, 56] or 3D [9, 44] latent spaces, focusing on aspects such as
appearance identity, facial dynamics, and head pose. While these approaches
have yielded promising results in expressive facial representations, they face
similar challenges to explicit methods, notably the accurate and effective
disentanglement of various facial factors remains a considerable challenge.

#### Portrait Image Animation.

In recent years, the field of portrait image animation has made substantial
progress, particularly in generating realistic and expressive talking head
animations from static images paired with audio inputs. The advancement began
with LipSyncExpert [29], which improved lip synchronization to speech
segments, achieving high accuracy for static images and videos of specific
identities. This was followed by developments such as SadTalker [53] and
DiffTalk [35], which integrated 3D information and control mechanisms to
enhance the naturalism of head movements and expressions. A major shift
occurred with the introduction of diffusion models, notably Diffused Heads
[38] and GAIA [14], which facilitated the creation of high-fidelity talking
head videos featuring natural head motion and facial expressions without
reliance on additional reference videos. These models demonstrated robust
capabilities for personalized and generalized synthesis across various
identities. Continuing this evolution, DreamTalk [25] and VividTalk [39]
utilized diffusion models to produce expressive, high-quality audio-driven
facial animations, emphasizing improved lip synchronization and diverse
speaking styles. Further advancements were made by Vlogger [7] and AniPortrait
[47], which introduced spatial and temporal controls to accommodate variable
video lengths and customizable character representations, respectively. Recent
innovations such as VASA-1 [48] and EMO [40] have developed frameworks for
creating emotionally expressive and realistic talking faces from static images
and audio, capturing a broad spectrum of facial nuances and head movements.
Lastly, AniTalker [23] introduced a groundbreaking framework focused on
generating detailed and realistic facial movements through a universal motion
representation, minimizing the need for labeled data and highlighting the
potential for dynamic avatar creation in practical applications. In this
study, we adopt the latent diffusion formulation as espoused in EMO [40] and
introduce a hierarchical cross-attention mechanism to augment the correlation
between audio inputs and non-identity-related motions such as lip movements,
expressions, and poses. Such a formulation not only affords adaptive control
over expression and pose diversity but also enhances the overall coherence and
naturalness of the generated animations.

Figure 2:  The overview of the proposed pipeline. Specifically, we integrates
a reference image containing a portrait with corresponding audio input to
drive portrait animation. Optional visual synthesis weights can be used to
balance lip, expression, and pose weights. ReferenceNet encodes global visual
texture information for consistent and controllable character animation. Face
and audio encoders generate high-fidelity portrait identity features and
encode audio as motion information respectively. The module of hierarchical
audio-driven visual synthesis establishes relationships between audio and
visual components (lips, expression, pose), with a UNet denoiser used in the
diffusion process.

##  3 Methodology

###  3.1 Preliminaries

#### Latent Diffusion Models.

Latent diffusion models represent a class of diffusion models that operate
within the encoded latent space produced by an autoencoder, formalized as
$\mathcal{D}(\mathcal{E}(\cdot))$. A prominent instance of an latent diffusion
model is Stable Diffusion, which integrates a VQ-VAE autoencoder [41] and a
time-conditioned U-Net for noise estimation. Moreover, Stable Diffusion [33]
employs a CLIP ViT-L/14 text encoder [30] to transform the input text prompt
into a corresponding text embedding, which serves as a condition during the
diffusion process. During the training phase, given an image
$I\in\mathbb{R}^{H_{I}\times W_{I}\times 3}$ and its associated text condition
$c_{\text{embed}}\in\mathbb{R}^{D_{c}}$, the latent representation:

$z_{0}=\mathcal{E}(I)\in\mathbb{R}^{H_{z}\times W_{z}\times D_{z}}$ (1)

undergoes a diffusion process across $T$ timesteps. This process is modeled as
a deterministic Gaussian process, culminating in $z_{T}\sim\mathcal{N}(0,I)$.
The training objective for Stable Diffusion is encapsulated by the following
loss function:

$L=\mathbb{E}_{\mathcal{E}(I),c_{\text{embed}},\epsilon\sim\mathcal{N}(0,1),t}\left[\lVert\epsilon-\epsilon_{\theta}(z_{t},t,c_{\text{embed}})\rVert_{2}^{2}\right],$
(2)

with $t$ uniformly sampled from $\\{1,...,T\\}$. Here, $\epsilon_{\theta}$
denotes the trainable components of the model, which include a denoising U-Net
equipped with Residual Blocks and layers that facilitate both self-attention
and cross-attention. These components are tasked with processing the noisy
latent variables $z_{t}$ and the conditional embeddings $c_{\text{embed}}$.

Upon the completion of training, the original latent $z_{t}$ is reconstructed
using deterministic sampling methods such as denoising diffusion implicit
models (DDIM) [30]. The latent $z_{t}$ is then decoded by the decoder
$\mathcal{D}$ to generate the final image output. This methodology not only
preserves the fidelity of the generated images but also ensures that they are
contextually aligned with the input auxiliary conditions, demonstrating the
efficacy of integrating conditioned processes in generative models.

#### Cross Attention as Motion Guidance.

Cross attention is a critical component of the latent diffusion model
framework, significantly enhancing the generative process by directing the
flow and focus of information. This mechanism allows the model to concentrate
on specific motion guidance and latent image representations, thus improving
the semantic coherence between the generated videos and the motion condition.
In general video generation tasks, motion conditions may include textual
prompts or dense motion flows. In more specialized domains such as human image
animation, motion conditions may involve skeletons, semantic maps, and 3D
parametric flows. For portrait image animation tasks, audio is typically used
as the motion condition, with cross attention employed to integrate the
relevant conditions effectively. Mathematically, cross attention is
implemented using attention layers that process both the embedding of the
motion condition, denoted as $c_{\text{embed}}$, and the noisy latent
variables $z_{t}$. These layers calculate attention scores that determine the
model’s focus level on various input aspects. The scores are then utilized to
weight the contributions of each component, creating a weighted sum that
serves as the input for subsequent layers. The cross attention mechanism can
be expressed as:

$\text{CrossAttn}(z_{t},c_{\text{embed}})=\text{Attention}(Q,K,V).$ (3)

Here, $\text{CrossAttn}(z_{t},c_{\text{embed}})$ computes the attention scores
using:

$\displaystyle Q$ $\displaystyle=W_{Q}\cdot
z_{t},\;\;W_{Q}\in\mathbb{R}^{D_{e}\times D_{z}}$ (4) $\displaystyle K$
$\displaystyle=W_{K}\cdot c_{\text{embed}},\;\;W_{K}\in\mathbb{R}^{D_{e}\times
D_{c}}$ (5) $\displaystyle V$ $\displaystyle=W_{V}\cdot
c_{\text{embed}},\;\;W_{V}\in\mathbb{R}^{D_{e}\times D_{c}}$ (6)

where $W_{Q}$, $W_{K}$, and $W_{V}$ are learnable projection matrices.

The output is a weighted representation that captures the most pertinent
aspects of the input condition and latent image at the current stage of the
diffusion process. By incorporating cross attention, the latent diffusion
model dynamically adjusts its focus based on the evolving state of the latent
variables and the specified motion condition.

Figure 3: Visualization of the hierarchical audio-driven visual synthesis and
a comparative analysis of the audio-visual cross attention between the
original full method and our proposed hierarchical audio-visual cross
attention.

###  3.2 Hierarchical Audio-Driven Visual Synthesis.

As demonstrated in Figure 2, our task is to generate a video with $S$ frames
given a face image and a clip of audio. For each frame $s$, we denote the
corresponding latent representation at the diffusion time step $t$ as
$z^{(s)}_{t}$.

#### Face Embedding.

The objective of face embedding is to produce high-fidelity portrait identity
features. In contrast to previous approaches utilizing CLIP [30] for generic
visual feature encoding by jointly training on a large dataset of images and
textual descriptions, our method employs a pre-trained face encoder to extract
identity features $c_{\text{exp}}\in\mathbb{R}^{D_{f}}$. These features are
fed into a diffusion network’s cross-attention module, i.e.,
CrossAttn($z_{t}^{(s)}$, $c_{\text{exp}}$), to interact with latents,
generating portrait animations faithful to the input character features. This
approach not only ensures generalization in facial feature extraction but also
accurately preserves and reproduces individual identity traits such as facial
expressions, age, and gender.

#### Audio Embedding.

To enhance the encoding of audio for motion-driven information in animation,
we employed wav2vec [34] as the audio feature encoder. Specifically, we
concatenated the audio embeddings from the final 12 layers of the wav2vec
network to capture richer semantic information across different audio layers.
Given the contextual influence on sequential audio data, we extracted the
corresponding 5-second audio segment for the $S$ frames. Utilizing three
simple linear layers, we transform the pre-trained model’s audio embeddings to
$\\{c_{\text{audio}}^{(s)}\\}_{s=1}^{S}$, where
$c_{\text{audio}}^{(s)}\in\mathbb{R}^{D_{a}}$ is the audio embedding for the
$s$ frame.

#### Hierarchical Audio-Visual Cross Attention.

As shown in Figure 3, we introduce a hierarchical audio-visual cross attention
mechanism to facilitate the model’s learning of relationships between audio
and visual components such as lips, expression and pose.

First, we pre-process the face image to obtain masks
$M_{\text{lip}},M_{\text{exp}},M_{\text{pose}}\in\\{0,1\\}^{H_{z}\times
W_{z}}$ representing the lip, expression and pose areas respectively.
Specifically, we use the off-the-shelf toolbox MediaPipe to predict landmarks
from the face image $I$. The MediaPipe predicts the coordinates of multiple
landmarks for lip and expression. The bounding box masks $Y_{\text{lip}}$,
$Y_{\text{exp}}\in\\{0,1\\}^{H_{z}\times W_{z}}$ are obtained by

$Y_{\text{r}}(i,j)=\begin{cases}1&\text{if $(i,j)$ is inside
$\text{BoundingBox(r)}$}\\\ 0&\text{otherwise}\end{cases},$ (7)

where $r\in\\{\text{lip},\text{exp}\\}$, and BoundingBox(r) denotes the
bounding box of all the landmarks for r. We compute

$\displaystyle M_{\text{lip}}$ $\displaystyle=Y_{\text{lip}},$ (8)
$\displaystyle M_{\text{exp}}$ $\displaystyle=(1-M_{\text{lip}})\odot
Y_{\text{exp}},$ (9) $\displaystyle M_{\text{pose}}$
$\displaystyle=1-M_{\text{exp}},$ (10)

where $\odot$ is the Hadamard product.

Next, we apply the cross-attention mechanism between the latent
representations and the audio embeddings:

$o_{t}^{(s)}=\text{CrossAttn}(z_{t}^{(s)},c_{\text{audio}}^{(s)}).$ (11)

We apply masks and obtain different scaled latent representations by

$\displaystyle b_{t}^{(s)}$ $\displaystyle=o_{t}^{(s)}\odot M_{\text{pose}},$
(12) $\displaystyle f_{t}^{(s)}$ $\displaystyle=o_{t}^{(s)}\odot
M_{\text{exp}},$ (13) $\displaystyle l_{t}^{(s)}$
$\displaystyle=o_{t}^{(s)}\odot M_{\text{lip}}.$ (14)

In the end, to effectively merge these outputs, we add a adaptive module to
handle hierarchical audio guidance. Specifically, the output is obtained by
the convolution layer:

$\sum_{u}\text{Conv}_{u}(u),\;u\in\\{a_{t}^{(s)},f_{t}^{(s)},l_{t}^{(s)}\\}.$
(15)

###  3.3 Network Architecture

Finally, we present the network architecture of the diffusion-based generative
model with the hierarchical audio-visual cross-attention proposed in this
paper. We employed a standard visual generative model for video generation
based on Stable Diffusion, utilizing ReferenceNet to guide the generation
process using existing reference images. Temporal alignment enhances coherence
and consistency in the generated video sequence over time, in addition to the
hierarchical audio-driven visual synthesis module that establishes refined
mappings between audio and lip, expression, and pose, as previously described.

#### Diffusion Backbone.

Stable Diffusion 1.5 is built upon a latent diffusion model that consists of
three primary components: Vector Quantised Variational AutoEncoder (VQ-VAE),
Unet-based denoising model, and a conditioning module. In the context of text-
to-image applications, latent image inputs are generated from random
initialization and processed by the diffusion model in collaboration with the
conditioning module to generate new latent image outputs. In this study, text
features are excluded from the conditioning in Stable Diffusion, as audio
signals are employed as the primary driver for motion.

#### ReferenceNet.

ReferenceNet is designed to guide the generation process by referencing
existing images to enhance the quality of generated videos, including visual
texture information of portraits and backgrounds. It is a Unet-based Stable
Diffusion network with the same number of layers as the denoising model
network. The feature maps generated by these structures at specific layers are
likely to exhibit similarities, aiding in the integration of extracted
features by ReferenceNet into the diffusion backbone. These structures
generate feature maps at specific layers with the same spatial resolution and
potentially similar semantic features. This integration enhances the quality
of generated videos, including visual texture information of portraits and
backgrounds. During model training, the first frame of the video clip serves
as the reference image.

#### Temporal Alignment.

In video generation tasks based on diffusion models, temporal alignment plays
a crucial role in ensuring the coherence and consistency of the generated
video sequences over time. More precisely, a subset of frames (we use 2 in our
implementation) from the preceding inference step are designated as motion
frames, which are then concatenated with the latent noise and manipulated
along the temporal axis. This temporal manipulation is facilitated through
multiple self-attention blocks, each adept at processing the temporal sequence
elements of the video frames.

Datasets | Raw | Filtered
---|---|---
#IDs | #Hours | #IDs | #Hours
HDTF | 190 | 8.42 | 188 | 6.47
Bilibili | 402 | 18.41 | 311 | 8.66
Youtube | 1617 | 137.49 | 1324 | 93.73
Total | 2209 | 164.32 | 1823 | 108.86

Figure 4: Statistics of the dataset for training and inference.

###  3.4 Training and Inference

#### Training.

The training process consists of two distinct stages: (1) In the first
training phase, individual video frames are generated by utilizing reference
image and target video frame pairs. The parameters of VAE encoder and decoder,
along with the facial image encoder, are fixed, while allowing the weights of
the spatial cross-attention modules of ReferenceNet and denoising UNet to be
optimized to improve single-frame generation capability. Video clips
containing 14 frames are extracted for input data, with a random frame from
the facial video clip chosen as the reference frame and another frame from the
same video as the target image. (2) In the second training phase, video
sequences are trained using reference images, input audio, and target video
data. The spatial modules of ReferenceNet and denoising UNet remain static,
focusing on enhancing video sequence generation capability. This phase
predominantly focuses on training hierarchical audio-visual cross-attention to
establish the relationship between audio as motion guidance and the visual
information of lip, expression, and pose. Additionally, motion modules are
introduced to improve model temporal coherence and smoothness, initialized
with pre-existing weights from AnimateDiff [12]. One frame from a video clip
is randomly selected as the reference image during this phase.

#### Inference.

During the inference stage, the network takes a single reference image and
driving audio as input, producing a video sequence that animates the reference
image based on the corresponding audio. To produce visually consistent long
videos, we utilize the last 2 frames of the previous video clip as the initial
$k$ frames of the next clip, enabling incremental inference for video clip
generation.

Method | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---
SadTalker [54] | 22.340 | 203.860 | 7.885 | 7.545 | 9.776
Audio2Head [43] | 37.776 | 239.860 | 8.024 | 7.145 | 17.103
DreamTalk [25] | 78.147 | 790.660 | 6.376 | 8.364 | 15.696
AniPortrait [47] | 26.561 | 234.666 | 4.015 | 10.548 | 13.754
Ours | 20.545 | 173.497 | 7.750 | 7.659 | 7.951
Real video | - | - | 8.700 | 6.597 | -
Table 1:  The quantitative comparisons with the existed portrait image
animation approaches on the HDTF data-set. Our proposed method excels in
generating high-quality, temporally coherent talking head animations with
superior lip synchronization performance.  Figure 5: Qualitative comparison
with existing approaches on HDTF data-set.

##  4 Experiment

###  4.1 Experimental Setups

#### Implementation Details.

Experiments encompassing both training and inference were carried out on a
computational platform featuring 8 NVIDIA A100 GPUs. The initial and
subsequent stages comprised 30,000 training steps each, with a batch size of 4
and video dimensions set at $512\times 512$. Each training instance in the
second stage produced 14 video frames, with the motion module’s latents
concatenated with the first 2 ground truth frames for video continuity. In
both training stages, a learning rate of 1e-5 was employed, and the motion
module was initialized with weights from Animatediff. To enhance video
generation, the reference image, guidance audio, and motion frames were
dropped with a 0.05 probability during training. In inference, continuity
across sequences was ensured by concatenating noisy latents with feature maps
of the last 2 motion frames from the previous step within the motion module.

#### Datasets.

As shown in Figure 4, our dataset comprises HDTF (190 clips, 8.42 hours) and
additional Internet-sourced data (2019 clips, 155.90 hours). As shown in
Figure 4, these videos are consisted of individuals of diverse ages,
ethnicities, and genders in half-body or close-up shots against varied indoor
and outdoor backgrounds. To ensure high-quality training data, we underwent a
data cleaning process that focused on retaining single-person speaking videos
exhibiting strong lip and audio consistency, while excluding videos with scene
changes, significant camera movements, excessive facial motion, and fully
side-facing shots. Mediapipe was utilized to determine expression and lip
activity ranges in training videos, forging the basis for constructing
expression and lip masks employed in both training and inference. Following
data cleaning, our refined training dataset includes HDTF (188 clips, 6.47
hours) and Internet-sourced data (1635 clips, 102.39 hours), with each
training video clip comprising 15 frames at a resolution of $512\times 512$.

#### Evaluation Metrics.

The evaluation metrics utilized in the portrait image animation approach
include Fréchet Inception Distance (FID), Fréchet Video Distance (FVD),
Synchronization-C (Sync-C), Synchronization-D (Sync-D) and E-FID.
Specifically, FID and FVD measure the similarity between generated images and
real data, with lower values indicating better performance and thus more
realistic outputs. Sync-C and Sync-D evaluate the lip synchronization of
generated videos in terms of content and dynamics, with higher Sync-C and
lower Sync-D scores denoting better alignment with the audio. E-FID assesses
the quality of generated images based on the features extracted from the
Inception network, offering a refined evaluation of fidelity.

Method | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---
SadTalker [54] | 50.015 | 471.163 | 6.922 | 7.921 | 95.194
Audio2Head [43] | 84.793 | 457.499 | 6.518 | 8.143 | 153.618
DreamTalk [25] | 109.011 | 988.539 | 5.709 | 8.743 | 153.450
AniPortrait [47] | 46.915 | 477.179 | 2.853 | 11.709 | 88.986
Ours | 44.578 | 377.117 | 7.191 | 7.984 | 78.495
Real video | - | - | 7.372 | 7.518 | -
Table 2: The quantitative comparisons with the existing portrait image
animation approaches on the CelebV data-set. Figure 6: Qualitative comparison
with existing approaches on CelebV data-set.

#### Baseline.

In our quantitative experiments, we conducted a comparative analysis of our
proposed method against publicly available implementations of SadTalker [54],
DreamTalk [25], Audio2Head [43], and AniPortrait [47]. The evaluation was
performed on the HDTF, CelebV and the proposed dataset, utilizing a training
and testing split where 90% of the identity data was allocated for training
purposes. The qualitative comparison encompassed an evaluation of our method
against these selected approaches, taking into consideration reference images,
audio inputs, and the resultant animated outputs provided by each respective
method. This qualitative assessment aimed to provide insights into the
performance and capabilities of our method in generating realistic and
expressive talking head animations.

Method | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---
SadTalker [54] | 24.212 | 249.786 | 6.613 | 8.099 | 37.324
Audio2Head [43] | 61.510 | 383.178 | 5.719 | 8.585 | 66.116
DreamTalk [25] | 128.423 | 964.088 | 5.925 | 8.596 | 58.180
AniPortrait [47] | 24.118 | 250.770 | 3.043 | 10.997 | 37.806
Ours | 23.266 | 239.647 | 6.924 | 7.969 | 34.731
Real video | - | - | 7.011 | 7.606 | -
Table 3: The quantitative comparisons with the existed portrait image
animation approaches on the proposed “wild” data-set. Figure 7: Video
generation results of the proposed approach given different portrait styles.

###  4.2 Quantitative Results

#### Comparison on HDTF Dataset.

Table 1 presents a comprehensive quantitative evaluation of various portrait
image animation techniques on the HDTF dataset. Our proposed method
demonstrates superior performance across several metrics, notably achieving
the lowest FID (20.545), FVD (173.497), and E-FID (7.951). These results
underscore the high quality and temporal coherence of the generated talking
head animations. Additionally, our method exhibits commendable lip
synchronization capabilities, as evidenced by the Sync-C (7.750) and Sync-D
(7.659) metrics, which closely align with real video benchmarks. These
achievements highlight the effectiveness of our approach in enhancing lip
synchronization while maintaining high-fidelity visual generation and temporal
consistency. To supplement these findings, Figure 5 visualizes the comparative
performance of the different methods.

Figure 8: Video generation results of the proposed approach given different
audio styles. Figure 9: Qualitative comparison of head and expression motion
diversity between existing methodologies and our proposed approach.

#### Comparison on CelebV Dataset.

The quantitative evaluations presented in Table 2 provide a comparative
analysis of various portrait image animation techniques using the CelebV
dataset. Our proposed approach achieves superior performance, demonstrating
the lowest FID (44.578) and FVD (377.117) scores, as well as the highest
Sync-C score (7.191). Additionally, our method secures a competitive Sync-D
value (7.984) and the lowest E-FID score (78.495), indicating high-quality
animations with notable temporal coherence and precise lip synchronization.
These results attest to the robustness and efficacy of our technique in
generating realistic animations. To offer a more comprehensive analysis, a
visual comparative examination is presented in Figure 6.

#### Comparison on the Proposed “Wild” Dataset.

The quantitative evaluations depicted in Table 3 offer a comparative analysis
of various portrait image animation techniques on the proposed “wild” dataset
(as shown in statistics of Figure 4 except for HDTF). Our methodology
demonstrates outstanding performance, achieving the lowest FID (23.266) and
FVD (239.647) scores, as well as the highest Sync-C score (6.924).
Additionally, our approach attains the lowest E-FID score (34.731) and a
competitive Sync-D value (7.969), closely approximating benchmarks set by real
video data. These results highlight the robustness and efficacy of our
technique in generating high-quality animations characterized by temporal
coherence and precise lip synchronization under diverse and challenging
conditions.

Figure 10: Visualization of identity fine-tuning personalization. Enhancing
identity-specific expression and pose features through personalized data fine-
tuning facilitates the generation of animations closely resembling the
targeted identity.

###  4.3 Qualitative Results

#### Different Portrait Styles.

The study depicted in Figures 7 delves into the impact of various portrait
styles, including but not limited to sketching, painting, AI generated images,
sculpture, and more, on video synthesis using the proposed methodology. The
results underscore the versatility and robustness of the approach in
generating a diverse range of visual and auditory outputs.

#### Different Audio Styles.

Figure 8 presents qualitative outcomes concerning distinct audio styles. The
findings demonstrate that our method can efficiently handle a range of audio
inputs to generate high-fidelity and visually coherent videos that align
seamlessly with the audio content.

#### Audio-Visual Cross Attention.

Figure 3 illustrates the comparison of audio-visual cross attention between
the original full and the proposed hierarchical audio-visual cross attention.
It is observed that the proposed hierarchical audio-visual cross attention
effectively achieves fine-grained and precise alignment between audio, lip
motion, and facial expressions. This alignment enhances the relevance during
training and enables more precise motion control based on audio in the
inference process.

#### Head and Expression Motion Diversity.

Figure 9 presents a qualitative comparison of head and expression motion
diversity between existing methodologies and our proposed approach. The
analysis demonstrates that our method excels in generating animations
characterized by enhanced diversity in expression and pose motion.

#### Identity Fine-tuning Personalization.

Figure 10 presents the outcomes of fine-tuning diverse identities using
personalized data specific to each identity. The analysis reveals that the
proposed hierarchical audio-driven visual synthesis adeptly captures the
distinctive features of individual identities, leading to personalized
animations that closely reflect the unique traits of each identity following
the fine-tuning procedure.

(a) Expression control by adjusting hierarchical weights.

(b) Qualitative comparison with applying higher coefficient on expression.

(c) Qualitative comparison with applying higher coefficient on pose.

Figure 11: Motion control (pose, expression and lip) by adjusting hierarchical weights. he statistical analysis of these dynamics is conducted exclusively on the HDTF dataset to ensure consistency in the evaluation framework. Full | Lip | Exp. | Pose | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---|---|---|---
$\checkmark$ |  |  |  | 20.581 | 193.062 | 6.499 | 8.691 | 9.133
$\checkmark$ | $\checkmark$ |  |  | 24.605 | 217.417 | 7.187 | 8.002 | 8.334
$\checkmark$ |  | $\checkmark$ |  | 24.003 | 207.352 | 7.072 | 8.127 | 8.282
$\checkmark$ |  |  | $\checkmark$ | 23.452 | 205.636 | 6.436 | 8.502 | 8.375
| $\checkmark$ | $\checkmark$ | $\checkmark$ | 20.545 | 173.497 | 7.750 | 7.659 | 7.951
Table 4: Ablation study of hierarchical audio-visual (lip, expression, and
pose) cross attention. The designation “Full” denotes the baseline
configuration of full visual-audio cross attention. Subsequently, the study
incrementally incorporates regional cross attention between audio and visual
modalities, introducing lip, expression, and pose features individually. In
our experimental setup, we adhere to the configuration specified in the last
row of the table.

(a) Qualitative study of hierarchical audio-visual (lip, expression, and pose)
cross attention.

(b) Comparison of different attention weighting mechanisms in hierarchical
audio-driven visual synthesis.

Figure 12: Qualitative ablation study of (1) hierarchical audio-visual (lip,
expression, and pose) cross attention; (2) different attention weighting
mechanism in hierarchical audio-driven visual synthesis.

###  4.4 Ablation Study

#### Hierarchical Audio-Visual Cross Attention.

The ablation study presented in Table 4 meticulously examines the impact of
hierarchical audio-visual cross attention by systematically integrating lip,
expression, and pose features. Analysis of the results reveals that the
incorporation of lip features in isolation leads to enhancements in specific
metrics, albeit with varying degrees of improvement across evaluation
criteria. Furthermore, the combined inclusion of facial features with lip
features yields mixed results but notably enhances synchronization confidence
and reduces E-FID. Notably, the incorporation of all three modalities—lip,
expression, and pose—exhibits the most substantial overall performance
improvement, elevating the quality and coherence of audio-visual synthesis.
This iterative refinement underscores the efficacy of hierarchical cross-
attention across multiple features in achieving enhanced audio-visual
integration. Figure 12(a) complements this assessment with further qualitative
insights into hierarchical audio-visual (lip, expression, and pose) cross
attention.

#### Different Attention Weighting Mechanism.

Table 5 provides a quantitative assessment of various attention weighting
mechanisms utilized in the hierarchical audio-driven visual synthesis module.
Among the mechanisms examined, the “Direct addition” approach stands out with
the lowest FID score of 19.580, indicating exceptional image quality.
Remarkably, our “Zero convolution” strategy surpasses all others,
demonstrating superior performance across critical metrics. Specifically, it
achieves the highest FVD (173.497), Sync-C (7.750), Sync-D (7.659), and E-FID
(7.951) scores, showcasing its robustness in generating synchronized,
temporally coherent animations with utmost fidelity. Figure 12(b) complements
this analysis by providing additional qualitative comparisons of different
attention weighting mechanisms in hierarchical audio-driven visual synthesis.

Figure 13: Qualitative study of audio and image CFG scales. Mechanism | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---
Direct addition | 19.580 | 177.715 | 7.087 | 8.147 | 8.063
Self attention | 20.984 | 183.113 | 7.568 | 7.947 | 8.292
Zero convolution (ours) | 20.545 | 173.497 | 7.750 | 7.659 | 7.951
Table 5: Quantitative comparison of different attention weighting mechanism in
the module of hierarchical audio-driven visual synthesis.

#### Expression Control by Adjusting Hierarchical Weights.

Figure 11 illustrates the qualitative outcomes of expression control achieved
through the manipulation of hierarchical weights. The evaluation of various
synthesis weight settings reveals a discernible trend: augmenting the lip
weight leads to improved lip-sync accuracy while slightly compromising overall
fidelity and video quality. In contrast, elevating the expression weight
marginally enhances image and video quality but may have a slight adverse
effect on synchronization metrics. Consequently, the selection of weight
settings necessitates a judicious balance between fidelity, video quality, and
synchronization, guided by the specific requirements of the application. For
this study, we adhere to the default setting determined through the adaptive
weighting process.

audio | image | FID$\downarrow$ | FVD$\downarrow$ | Sync-C$\uparrow$ | Sync-D$\downarrow$ | E-FID$\downarrow$
---|---|---|---|---|---|---
$\lambda_{a}$ = 1.0 |  $\lambda_{i}$ = 1.0 | 33.320 | 270.925 | 5.983 | 9.532 | 12.579
$\lambda_{a}$ = 1.0 |  $\lambda_{i}$ = 3.5 | 23.210 | 194.295 | 5.742 | 9.435 | 8.565
$\lambda_{a}$ = 1.0 |  $\lambda_{i}$ = 6.0 | 23.154 | 204.625 | 5.583 | 9.589 | 8.759
$\lambda_{a}$ = 3.5 |  $\lambda_{i}$ = 3.5 | 23.167 | 195.179 | 7.658 | 7.894 | 7.951
$\lambda_{a}$ = 6.0 |  $\lambda_{i}$ = 3.5 | 25.016 | 229.128 | 7.952 | 7.742 | 9.024
Table 6: Quantitative study of audio and image CFG scales. In our
implementation, we adopt the setting ($\lambda_{a}=3.5$ and $\lambda_{i}=3.5$)
for the balance of visual fidelity and motion diversity.

#### Audio and Image CFG Scales.

Table 6 presents a quantitative analysis of the generated videos under various
configurations of audio and image CFG scales. Among the configurations, the
setting of $\lambda_{a}=3.5$ and $\lambda_{i}=3.5$ achieves a balanced
performance, with competitive FID (23.167), FVD (195.179), Sync-C (7.658),
Sync-D (7.894), and E-FID (7.951) scores. This balance between visual fidelity
and motion diversity underscores the effectiveness of this configuration in
producing high-quality, synchronized videos.

Method | GPU memory (GB) | Time (sec)
---|---|---
Inference w. HADVS | 9.77 | 1.63
Inference w.o. HADVS | 9.76 | 1.63
Inference ($256\times 256$) | 6.62 | 0.46
Inference ($1024\times 1024$) | 20.66 | 10.29
Table 7: Efficiency analysis of different steps and setting of the proposed
approach. “HADVS” denotes the proposed hierarchical audio-driven visual
synthesis. “$256\times 256$” and “$1024\times 1024$” represent the inference
video resoluton.

#### Efficiency Analysis.

Table 7 presents an efficiency analysis of various configurations of our
proposed approach. The inference with hierarchical audio-driven visual
synthesis requires 9.77 GB of GPU memory and takes 1.63 seconds. In contrast,
without hierarchical audio-driven visual synthesis, the GPU memory usage is
slightly reduced to 9.76 GB while maintaining the same inference time of 1.63
seconds. Variations in video resolution significantly affect both GPU memory
usage and inference time; processing at $256\times 256$ resolution requires
6.62 GB of GPU memory and 0.46 seconds, whereas $1024\times 1024$ resolution
necessitates 20.66 GB of GPU memory and 10.29 seconds.

###  4.5 Limitation and Future Work

Despite the advancements proposed in this study regarding portrait image
animation, there exist several limitations that necessitate further
exploration and consideration. These limitations underscore areas where future
research endeavors can contribute to the refinement and augmentation of the
presented methodology: (1) Enhanced visual-audio synchronization. Future
investigations could delve into more advanced synchronization techniques,
potentially integrating sophisticated audio analysis methods or harnessing
deeper cross-modal learning strategies. These advancements have the potential
to refine the alignment of facial movements with audio inputs, particularly in
contexts characterized by intricate speech patterns or nuanced emotional
expressions. (2) Robust temporal coherence. There is a need for further
exploration of advanced temporal coherence mechanisms to address
inconsistencies in sequences featuring rapid or intricate movements.
Developing robust alignment strategies, possibly guided by long-term
dependencies in sequential data or leveraging recurrent neural network models,
could bolster frame-to-frame stability. (3) Computational efficiency. It is
imperative to optimize the computational efficiency of the diffusion-based
generative model and UNet-based denoiser. Research into lightweight
architectures, model pruning, or efficient parallelization techniques holds
promise in rendering the approach more viable for real-time applications while
minimizing resource utilization. (4) Improved diversity control. The balance
between the diversity of expressions and poses and the preservation of visual
identity integrity remains a critical challenge. Future endeavors could
concentrate on refining adaptive control mechanisms, potentially through the
integration of more nuanced control parameters or sophisticated diversity
metrics. Such refinements would ensure a more natural and diverse animation
output while upholding the authenticity of facial identities.

###  4.6 Social Risks and Mitigations

In the context of the research presented in the paper, there are social risks
associated with the development and implementation of portrait image animation
technologies driven by audio inputs. One potential risk is the ethical
implications of creating highly realistic and dynamic portraits that could
potentially be misused for deceptive or malicious purposes, such as deepfakes.
To mitigate this risk, it is essential to establish ethical guidelines and
responsible use practices for the technology. Additionally, there may be
concerns regarding privacy and consent related to the use of individuals’
images and voices in the creation of animated portraits. Mitigating these
concerns involves ensuring transparent data usage policies, obtaining informed
consent, and safeguarding the privacy rights of individuals. By addressing
these social risks and implementing appropriate mitigations, the research aims
to promote the responsible and ethical development of portrait image animation
technologies within the societal context.

##  5 Conclusion

This paper introduces a novel method for portrait image animation using end-
to-end diffusion models, addressing challenges in audio-driven facial dynamics
synchronization and high-quality animation generation with temporal
consistency. The proposed hierarchical audio-driven visual synthesis module
enhances audio-visual alignment through cross-attention mechanisms and
adaptive weighting. By integrating diffusion-based generative modeling, UNet
denoising, temporal alignment, and ReferenceNet, the method improves animation
quality and realism. Experimental evaluations demonstrate superior image and
video quality, enhanced lip synchronization, and increased motion diversity,
validated by superior FID and FVD metrics. The method allows flexible control
over expression and pose diversity to accommodate diverse visual identities.

## References

  * [1]
  * Blanz and Vetter [2003] Volker Blanz and Thomas Vetter. 2003.  Face recognition based on fitting a 3D morphable model.  _IEEE Transactions on pattern analysis and machine intelligence_ 25, 9 (2003), 1063–1074.
  * Blattmann et al. [2023] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023.  Align your latents: High-resolution video synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22563–22575.
  * Burkov et al. [2020] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. 2020.  Neural head reenactment with latent pose descriptors. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13786–13795.
  * Chen et al. [2023b] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. 2023b.  Videocrafter1: Open diffusion models for high-quality video generation.  _arXiv preprint arXiv:2310.19512_ (2023).
  * Chen et al. [2023a] Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. 2023a.  Motion-conditioned diffusion model for controllable video synthesis.  _arXiv preprint arXiv:2304.14404_ (2023).
  * Corona et al. [2024] Enric Corona, Andrei Zanfir, Eduard Gabriel Bazavan, Nikos Kolotouros, Thiemo Alldieck, and Cristian Sminchisescu. 2024.  VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis.  _arXiv preprint arXiv:2403.08764_ (2024).
  * Dai et al. [2023] Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. 2023.  AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance.  _arXiv preprint arXiv:2311.12886_ (2023).
  * Drobyshev et al. [2022] Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. 2022.  Megaportraits: One-shot megapixel neural head avatars. In _Proceedings of the 30th ACM International Conference on Multimedia (ACM MM)_. 2663–2671.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. 2023\.  Structure and content-guided video synthesis with diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 7346–7356.
  * Gao et al. [2023] Yue Gao, Yuan Zhou, Jinglu Wang, Xiao Li, Xiang Ming, and Yan Lu. 2023\.  High-fidelity and freely controllable talking head video generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5609–5619.
  * Guo et al. [2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. 2023.  AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. In _International Conference on Learning Representations (ICLR)_.
  * Harvey et al. [2022] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. 2022\.  Flexible diffusion modeling of long videos.  _Advances in Neural Information Processing Systems (NeurIPS)_ 35 (2022), 27953–27965.
  * He et al. [2023] Tianyu He, Junliang Guo, Runyi Yu, Yuchi Wang, Jialiang Zhu, Kaikai An, Leyi Li, Xu Tan, Chunyu Wang, Han Hu, et al. 2023\.  GAIA: Zero-shot Talking Avatar Generation.  _arXiv preprint arXiv:2311.15230_ (2023).
  * Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. 2022a.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ (2022).
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. 2022b.  Video diffusion models.  _Advances in Neural Information Processing Systems (NeurIPS)_ 35 (2022), 8633–8646.
  * Hong et al. [2022] Yang Hong, Bo Peng, Haiyao Xiao, Ligang Liu, and Juyong Zhang. 2022.  Headnerf: A real-time nerf-based parametric head model. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 20374–20384.
  * Hu et al. [2023] Yaosi Hu, Zhenzhong Chen, and Chong Luo. 2023.  Lamd: Latent motion diffusion for video generation.  _arXiv preprint arXiv:2304.11603_ (2023).
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. 2023\.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 15954–15964.
  * Li et al. [2017] Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Transactions Graphics._ 36, 6 (2017), 194–1.
  * Liang et al. [2022] Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. 2022\.  Expressive talking head generation with granular audio-visual control. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 3387–3396.
  * Liu et al. [2023] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. 2023.  Video-p2p: Video editing with cross-attention control.  _arXiv preprint arXiv:2303.04761_ (2023).
  * Liu et al. [2024] Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. 2024.  AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding.  _arXiv preprint arXiv:2405.03121_ (2024).
  * Luo et al. [2023] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. 2023.  Videofusion: Decomposed diffusion models for high-quality video generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10209–10218.
  * Ma et al. [2023] Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. 2023\.  Dreamtalk: When expressive talking head generation meets diffusion probabilistic models.  _arXiv preprint arXiv:2312.09767_ (2023).
  * Molad et al. [2023] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. 2023\.  Dreamix: Video diffusion models are general video editors.  _arXiv preprint arXiv:2302.01329_ (2023).
  * Pang et al. [2023] Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. 2023.  Dpe: Disentanglement of pose and expression for general video portrait editing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 427–436.
  * Peebles and Xie [2023] William Peebles and Saining Xie. 2023.  Scalable diffusion models with transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 4195–4205.
  * Prajwal et al. [2020] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. 2020.  A lip sync expert is all you need for speech to lip generation in the wild. In _Proceedings of the 28th ACM international conference on multimedia (ACM MM)_. 484–492.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning (ICML)_. 8748–8763.
  * Ren et al. [2021] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu. 2021.  Pirenderer: Controllable portrait image generation via semantic neural rendering. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 13759–13768.
  * Rombach et al. [2022a] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022a.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10684–10695.
  * Rombach et al. [2022b] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022b.  High-resolution Image Synthesis with Latent Diffusion Models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10684–10695.
  * Schneider et al. [2019] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _arXiv preprint arXiv:1904.05862_ (2019).
  * Shen et al. [2023] Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. 2023.  Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1982–1991.
  * Siarohin et al. [2019] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019\.  First order motion model for image animation.  _Advances in Neural Information Processing Systems (NeurIPS)_ 32 (2019).
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022\.  Make-A-Video: Text-to-Video Generation without Text-Video Data. In _The Eleventh International Conference on Learning Representations (ICLR)_.
  * Stypułkowski et al. [2024] Michał Stypułkowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. 2024.  Diffused heads: Diffusion models beat gans on talking-face generation. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. 5091–5100.
  * Sun et al. [2023] Xusen Sun, Longhao Zhang, Hao Zhu, Peng Zhang, Bang Zhang, Xinya Ji, Kangneng Zhou, Daiheng Gao, Liefeng Bo, and Xun Cao. 2023\.  Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior.  _arXiv preprint arXiv:2312.01841_ (2023).
  * Tian et al. [2024] Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. 2024\.  EMO: Emote Portrait Alive-Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions.  _arXiv preprint arXiv:2402.17485_ (2024).
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al. 2017\.  Neural discrete representation learning.  _Advances in Neural Information Processing Systems (NeurIPS)_ 30 (2017).
  * Wang et al. [2023a] Duomin Wang, Yu Deng, Zixin Yin, Heung-Yeung Shum, and Baoyuan Wang. 2023a.  Progressive disentangled representation learning for fine-grained controllable talking head synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 17979–17989.
  * Wang et al. [2021a] S Wang, L Li, Y Ding, C Fan, and X Yu. 2021a.  Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion. In _International Joint Conference on Artificial Intelligence (IJCAI)_. 1098–1105.
  * Wang et al. [2021b] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021b.  One-shot free-view neural talking-head synthesis for video conferencing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 10039–10049.
  * Wang et al. [2023b] Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. 2023b.  Zero-shot video editing using off-the-shelf image diffusion models.  _arXiv preprint arXiv:2303.17599_ (2023).
  * Wang et al. [2024] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2024.  Videocomposer: Compositional video synthesis with motion controllability.  _Advances in Neural Information Processing Systems (NeurIPS)_ 36 (2024).
  * Wei et al. [2024] Huawei Wei, Zejun Yang, and Zhisheng Wang. 2024.  Aniportrait: Audio-driven synthesis of photorealistic portrait animation.  _arXiv preprint arXiv:2403.17694_ (2024).
  * Xu et al. [2024] Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. 2024.  VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time.  _arXiv preprint arXiv:2404.10667_ (2024).
  * Yang et al. [2023] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. 2023\.  Diffusion probabilistic modeling for video generation.  _Entropy_ 25, 10 (2023), 1469.
  * Yin et al. [2022] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. 2022\.  Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan. In _European Conference on Computer Vision (ECCV)_. Springer, 85–101.
  * Zakharov et al. [2020] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky. 2020.  Fast bi-layer neural synthesis of one-shot realistic head avatars. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_. Springer, 524–540.
  * Zhang et al. [2023c] Bowen Zhang, Chenyang Qi, Pan Zhang, Bo Zhang, HsiangTao Wu, Dong Chen, Qifeng Chen, Yong Wang, and Fang Wen. 2023c.  Metaportrait: Identity-preserving talking head generation with fast personalized adaptation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 22096–22105.
  * Zhang et al. [2023a] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023a.  Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 8652–8661.
  * Zhang et al. [2023b] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. 2023b.  Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 8652–8661.
  * Zhou et al. [2023] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. 2023\.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv preprint arXiv:2211.11018_ (2023).
  * Zhou et al. [2021] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu. 2021\.  Pose-controllable talking face generation by implicitly modularized audio-visual representation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4176–4186.
  * Zhu et al. [2024] Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, , and Siyu Zhu. 2024\.  Champ: Controllable and consistent human image animation with 3d parametric guidance.  _arXiv preprint arXiv:2403.14781_ (2024).