# Exploring Kondo effect by quantum energy teleportation

Kazuki Ikeda<EMAIL_ADDRESS>Co-design Center for Quantum
Advantage, Department of Physics and Astronomy, Stony Brook University, Stony
Brook, New York 11794-3800, USA  Center for Nuclear Theory, Department of
Physics and Astronomy, Stony Brook University, Stony Brook, New York
11794-3800, USA  Rajeev Singh<EMAIL_ADDRESS>Center for Nuclear
Theory, Department of Physics and Astronomy, Stony Brook University, Stony
Brook, New York 11794-3800, USA  C.N. Yang Institute for Theoretical Physics,
Stony Brook University, Stony Brook, New York, 11794-3840, USA  Robert-Jan
Slager<EMAIL_ADDRESS>Department of Physics, TCM, University of Cambridge,
19 JJ Thomson Avenue, CB3 0HE, Cambridge, UK

###### Abstract

We consider a quantum energy teleportation (QET) method to replicate the phase
diagram of a one-dimensional $XXZ$ spin chain featuring a Kondo effect
coupling. In this setup, the energy supplier and receiver are spatially
separated from the point impurity and do not interact directly with it.
Nonetheless, they may successfully generate phase diagrams that closely mirror
those produced via exact diagonalization. This can be achieved using only
local operations on their respective subsystems, supplemented by classical
communication. This feat is made possible due to a critical connection between
the energy obtained through the QET approach and the system’s quantum
entanglement entropy. To substantiate these findings, we initially demonstrate
that the quantum entanglement entropy serves as the relevant order parameter
for the system. Intriguingly, changes in the gap spacing of the entanglement
spectra align with the locations of peaks in both entanglement entropy and
energy, as determined by QET. We hypothesize that this theoretical framework
could, for example, be validated experimentally using a one-dimensional chain
of Rydberg atoms.

##  I Introduction

The study of quantum many-body systems through local operations has gained
significant attention, especially in the context of quantum computation and
quantum information theory Islam  _et al._ (2015); Huang  _et al._ (2020);
Ayral  _et al._ (2023). In quantum computation, a large system is implemented
as a set of qubits, and the physical properties of the system are determined
as a binary bit string obtained by measuring each qubit. Quantum computers
have the potential to provide a significant advantage in simulating quantum
many-body systems, as they can naturally capture quantum behavior that is
inherent to these systems. However, the difficulty in simulating large-scale
quantum many-body systems on a quantum computer gives rise to considerable
challenges Beck  _et al._ (2023). Quantum systems are sensitive to noise and
errors, and maintaining the coherence of qubits over a simulation’s duration
becomes increasingly challenging as the number of qubits grows. Simulating
large quantum systems requires a large number of qubits and a high degree of
gate fidelity. As of now, the number of qubits and the level of control needed
for accurately simulating large scale quantum many-body systems might exceed
the capabilities of current quantum hardware Beck  _et al._ (2023). One may
wonder, whether is it possible to obtain global information on quantum many-
body systems using limited quantum resources. In some previous works by one of
the authors, this question was addressed by applying a QET strategy to
relativistic quantum field theory and symmetry-protected topological phases
Ikeda (2023a, b). Within our context, QET entails a protocol for extracting
energy from local systems using only local operations and classical
communications (LOCCs). The quantum entanglement in the ground state of the
system and the existence of quantum zero-point oscillations of energy are
intrinsically important for QET. In addition to the aforementioned analysis of
phase transitions, QET has been applied to quantum networks and quantum
crypto-systems Ikeda (2023c); Ikeda and Lowe (2023). The methods used in the
series of studies have been implemented using IBM’s real quantum computers
which is publicly available, see Ikeda (2023d, e). As a subsequent study, the
central question we would like to address here is the following: Is it
possible to know the presence or absence of a point defect/impurity at a point
far from the observer without directly observing the point defect?

Defects and impurities play a crucial role in shaping the physical properties
of materials and systems Alloul  _et al._ (2009); Slager  _et al._ (2015);
Slager (2019); Teo and Kane (2010); Slager  _et al._ (2014); Ran  _et al._
(2009). They can significantly impact various material characteristics and
behaviors, often leading to unique effects. In this work, we turn to the Kondo
effect Kondo (1964); Cronenwett  _et al._ (1998), which is a phenomenon in
condensed matter physics, that arises when a magnetic impurity is introduced
into a conducting material. In simple terms, the Kondo effect can be explained
as follows. A magnetic impurity typically has an unpaired electron spin, which
creates a local magnetic moment. This spin interacts with the spins of the
conducting electrons in the surrounding material. At higher temperatures when
thermal fluctuations dominate, the interaction between the impurity spin and
the conducting electrons is weak. The conducting electrons scatter off the
impurity without significantly affecting its spin orientation. However, as the
temperature is lowered, quantum mechanical effects become more pronounced. At
sufficiently low temperatures, the conducting electrons near the impurity are
able to form bound states with the impurity spin. These bound states involve
complex quantum entanglement between the impurity spin and the conducting
electrons. The formation of these bound states has an intriguing consequence.
The conducting electrons “screen” the impurity spin effectively shielding it
from the external magnetic field. This screening leads to a reduction in the
local magnetic moment of the impurity. The presence of the impurity spin and
the associated electron scattering contribute to an increase in the electrical
resistivity of the material at low temperatures. This increase is opposite to
the usual behavior in metals, where resistivity typically decreases with
decreasing temperature due to the decrease in electron scattering Kondo
(1964); Cronenwett  _et al._ (1998).

The Kondo effect has been experimentally observed and has other far-reaching
implications, from understanding the behavior of strongly correlated electron
systems to applications in quantum computing and spintronics Cronenwett  _et
al._ (1998); Neupane  _et al._ (2015); Zahid Hasan  _et al._ (2015); Neupane
_et al._ (2013); Beri and Cooper (2012); Buccheri  _et al._ (2016); Dzsaber
_et al._ (2017). It also provides insights into how quantum entanglement and
many-body interactions can lead to emergent behavior in condensed matter
systems. There has been renewed interest in the Kondo effect within one-
dimensional strongly correlated systems Tsvelik and Yevtushenko (2019). In
interacting one-dimensional systems characterized by the Tomonaga-Luttinger
universality class Tomonaga (1950); Luttinger (1963); Lieb (1965); Haldane
(1981), the influence of a static impurity potential is significantly altered,
being renormalized to either infinite or zero values based on the attractive
or repulsive nature of interactions. This peculiar response of Tomonaga-
Luttinger liquids to static impurities has garnered significant attention
Furusaki (2005); Lee (2016), prompting further investigations into the impact
of dynamic impurities, often involving magnetic impurities in these liquids.
Many researchers have examined variations of the generalized Hubbard model
that incorporate an impurity spin (with spin value $1/2$) yu Hou  _et al._
(1999); Zvyagin and Schlottmann (1997); Öz and Klümper (2019). Notably, the
Kondo temperature, a key energy scale governing the screening of an impurity
spin by host electrons, exhibits a power-law relationship with the Kondo
exchange coupling. The characteristics of low-energy fixed points have been
explored through perturbative renormalization-group analysis and the boundary
conformal field theory approach Furusaki and Hikihara (1998a); Xavier and
Pereira (2022); Kimura and Ozaki (2019).

The rest of the paper is organized as follows. In Sec. II we explain the
theory of QET followed by Sec. III, where we define our model. We provide our
results in Sec. IV, where the phase diagram is reproduced by QET. We also
perform a numerical study of the phase transition using the entanglement
spectrum. Before concluding, we address the experimental feasibility in Sec. V
followed by the conclusion in Sec. VI.

Figure 1: The phase diagram of the $N=8$ Kondo system. Alice at $n_{A}=4$ has
no interaction with the impurity at $n=1$ which is interacting with the
nearest site $n=2$. She also has no interaction with Bob at $n_{B}=7$. The
left two panels (upper and lower) correspond to the entanglement entropy and
the right two panels (upper and lower) correspond to the teleported energy to
Bob’s local system. We keep $J_{K}=0.2$ for upper panels and $B=0.4$ for lower
panels.

##  II Quantum Energy Teleportation

In this section, we elucidate the standard procedure of the QET protocol. We
note that references to quantum circuit implementations for a variety of
configurations can be found in Ikeda (2023d, c, a); Ikeda and Lowe (2023). We
begin with the premises that one can express the Hamiltonian as
$H=\sum_{n=0}^{N-1}H_{n}$, where each $H_{n}$ represents the local Hamiltonian
that encompasses the interactions of the nearest neighbors and complies with
the constraint

$\displaystyle\begin{aligned}
\bra{g}H\ket{g}=\bra{g}H_{n}\ket{g}=0,~{}\forall\,n\in\\{0,1,\cdots,N-1\\}\,.\end{aligned}$
(1)

In the above, $\ket{g}$ signifies the ground state of the overall Hamiltonian
$H$, it’s crucial to recognize that $\ket{g}$ is not invariably the ground
state of the local Hamiltonian $H_{n}$. Engaging in non-trivial local
operations, like measuring the ground state, may lead to excited states and
augment the energy expectation value, an effect facilitated by the
experimental arrangement. Furthermore, $\ket{g}$ is commonly an entangled
state.

Here we describe the QET protocol in a spin chain consisting of spin 1/2
particles. We note that the outlined two level setup in principle can be
extended to larger on-site Hilbert spaces Hotta (2008). To begin, Alice
performs a measurement of her Pauli operator $\sigma_{n_{A}}$ utilizing
$P_{n_{A}}(\mu)=(1+\mu\,\sigma_{n_{A}})/2$, resulting in either $\mu=-1$ or
$\mu=+1$. When measuring the quantum state at subsystem $A$, it disturbs the
ground state entanglement and introduces energy $E_{A}$ from the measurement
apparatus into the entire system. In the beginning, the imparted energy
$E_{A}$ is localized around subsystem $A$, but Alice is unable to retrieve it
from the system merely through her manipulations at $n_{A}$. This phenomenon
transpires as the entanglement that existed before the measurement ensures
that information concerning $E_{A}$ is preserved in distant parts of the
system. To remove the energy from a location other than $n_{A}$, the quantum
many-body characteristic of the system may be employed through LOCC, a process
that is explained next.

Through a classical channel, Alice transmits the result of her measurement
$\mu$ to Bob, who then implements an operation $U_{n_{B}}(\mu)$ on his qubit
and separately measures his local Pauli operators $X_{n_{B}}$, $Y_{n_{B}}$,
and $Z_{n_{B}}$. Following Bob’s application of $U_{n_{B}}(\mu)$ to
$P_{n_{A}}(\mu)\ket{g}$, the resulting density matrix is denoted as
$\rho_{\text{QET}}$, where $\rho_{\text{QET}}$ is

$\rho_{\rm
QET}=\sum_{\mu\in\\{-1,1\\}}U_{n_{B}}(\mu)P_{n_{A}}(\mu)\ket{g}\bra{g}P_{n_{A}}(\mu)U^{\dagger}_{n_{B}}(\mu).$
(2)

With the utilization of $\rho_{\rm QET}$, the anticipated local energy at
Bob’s localized system is calculated as $\langle
E_{n_{B}}\rangle=\mathrm{Tr}[\rho_{\text{QET}}H_{n_{B}}]$, a value that is
typically negative. Owing to the principle of energy conservation,
$E_{B}=-\langle E_{n_{B}}\rangle(>0)$ is withdrawn from the system by the
apparatus conducting the operation $U_{n_{B}}(\mu)$ Hotta (2008). Through this
process, Alice and Bob are able to convey the energy of the quantum system
solely by performing operations on their individual local systems.

What’s most crucial is that, the technique for measuring $\langle
E_{n_{B}}\rangle$ is well-founded and has been authenticated through actual
quantum hardware. Stated differently, the findings and conclusions of this
paper do not engage with the question of whether the positive net energy can
be retrieved from the system.

In the subsequent, we provide specifics regarding the operations performed by
Alice and Bob. The definition of $U_{n_{B}}(\mu)$ is given by

$U_{n_{B}}(\mu)=\cos\theta I-i\mu\sin\theta\sigma_{n_{B}},$ (3)

where $\theta$ obeys the relations

$\displaystyle\cos(2\theta)=\frac{\xi}{\sqrt{\xi^{2}+\eta^{2}}}\,,\quad\sin(2\theta)=-\frac{\eta}{\sqrt{\xi^{2}+\eta^{2}}}\,,$
(4)

with

$\displaystyle\xi=\bra{g}\sigma_{n_{B}}H\sigma_{n_{B}}\ket{g}\,,$
$\displaystyle\qquad\eta=\bra{g}\sigma_{n_{A}}\dot{\sigma}_{n_{B}}\ket{g}\,,$
$\displaystyle\dot{\sigma}_{n_{B}}$ $\displaystyle=i[H,\sigma_{n_{B}}]\,.$ (5)

The local Hamiltonian must be selected in such a way that
$[H,\sigma_{n_{B}}]=[H_{n_{B}},\sigma_{n_{B}}]$. After Bob applies the
operation $U_{n_{B}}(\mu)$ to $P_{n_{A}}(\mu)\ket{g}$, the average quantum
state $\rho_{\text{QET}}$ is achieved. Subsequently, the average energy that
Bob measures is

$\langle
E_{n_{B}}\rangle=\mathrm{Tr}\left[\rho_{\text{QET}}H_{n_{B}}\right]=\frac{1}{2}\left[\xi-\sqrt{\xi^{2}+\eta^{2}}\right]\,.$
(6)

The value is negative when $\eta\neq 0$. It is anticipated that if there is no
energy loss, the positive energy of $-\langle E_{n_{B}}\rangle$ is conveyed to
Bob’s device following the measurement, in accordance with the principle of
energy conservation.

Figure 2: Magnetic field (left panel, upper and lower) and the energy gap
(right panel, upper and lower). We keep $J_{K}=0.2$ for upper panels and
$B=0.4$ for lower panels.

##  III Spin chains of Kondo effect

In this section, we would like to explore the Kondo effect through QET and
therefore, we examine the subsequent $XXZ$ spin chain model Laflorencie  _et
al._ (2008); Furusaki and Hikihara (1998b)

$\displaystyle H$ $\displaystyle=$ $\displaystyle
J\sum_{i}(S^{x}_{i}S^{x}_{i+1}+S^{y}_{i}S^{y}_{i+1}+\Delta
S^{z}_{i}S^{z}_{i+1})$ (7) $\displaystyle+$ $\displaystyle
J_{K}(S^{x}_{1}S^{x}_{n}+S^{y}_{1}S^{y}_{n}+\Delta
S^{z}_{1}S^{z}_{n})+B\sum_{i}S^{z}_{i}\,.$

The term $J\sum_{i}(S^{x}_{i}S^{x}_{i+1}+S^{y}_{i}S^{y}_{i+1}+\Delta
S^{z}_{i}S^{z}_{i+1})$ describes the interaction between neighboring spins
$S_{i}$ in the host material, with $\Delta$ allowing for an anisotropic
interaction and $J_{K}(S^{x}_{1}S^{x}_{n}+S^{y}_{1}S^{y}_{n}+\Delta
S^{z}_{1}S^{z}_{n})$ is the Kondo term, which couples, through coupling
$J_{K}$, the impurity spin $S_{1}$ with a spin $S_{n}$ in the host material.
$B\sum_{i}S^{z}_{i}$ represents the influence of an external magnetic field
$B$ on all the spins in the system. Here, $n$ is an indicative of the position
of an impurity.

We first recall the phase diagram of the model as shown in Fig. 1. In the
Kondo system phase diagram with system size $N=8$, see Fig. 1, Alice, located
at $n_{A}=4$, does not interact with the point impurity at $n=1$, which itself
is in interaction with the adjacent site at $n=2$. Additionally, Alice has no
interaction with Bob, who is at $n_{B}=7$. The left-hand side panels, both
upper and lower, display the entanglement entropy, while the right-hand side
panels, also upper and lower, illustrate the energy teleported to Bob’s local
system. The parameters utilized are $J_{K}=0.2$ for upper panels and $B=0.4$
for lower panels. We observe that both entanglement entropy and teleported
energy agree and share similar features.

The magnetic field profile used for our analysis is

$m=\frac{1}{N}\bra{\psi}\sum_{i=1}^{N}Z_{i}\ket{\psi}\,,$ (8)

which is shown in the left panel plots (upper and lower) in Fig. 2. Crucially,
we observe the correspondence between the teleported energy of Fig. 1 and
energy gap of Fig. 2.

##  IV Detecting phase transition through QET

An intriguing connection exists between the entanglement entropy and the
energy teleported. The variation in entanglement entropy before and after
Alice’s measurement can be computed as follows

$\displaystyle\Delta S_{AB}$ $\displaystyle=S_{AB}-\sum_{\mu\in\\{\pm
1\\}}p_{\mu}S_{AB}(\mu)\,.$ (9)

The first term on the right hand side of the equation, $S_{AB}$, depicts the
entanglement entropy between Alice and Bob prior to her measurement, whereas,
in the second term, $S_{AB}(\mu)$ is the entanglement entropy after Alice’s
measurement and probability distribution of $\mu$ is represented by $p_{\mu}$.

Within the context of the minimal model
$H=h(Z_{0}+Z_{1})+2kX_{0}X_{1}+\text{const.}$ Hotta (2011), we know that the
variation of the entanglement entropy satisfies

$\Delta
S_{AB}\geq-\frac{1+\sin^{2}\lambda}{2\cos^{3}\lambda}\ln\frac{1+\cos\lambda}{1-\cos\lambda}\frac{E_{B}}{\sqrt{h^{2}+k^{2}}}\,,$
(10)

with $\lambda=\arctan\left(k/h\right)$ implying that entanglement is expended
to facilitate energy teleportation, a consequence of Bob carrying out
conditional operations based on Alice’s measurement outcomes. In this regard,
an analog of Maxwell’s daemons circumstance can be obtained Lloyd (1997); Ito
and Sagawa (2015). While it remains uncertain whether this relationship is
universally applicable, similar outcomes have at least been confirmed
numerically Ikeda (2023a). Additionally, it is established that a specific
relationship strictly holds

$S_{A\bar{A}}\geq\frac{E^{2}_{B}}{4\|H_{B}\|^{2}}\,,$ (11)

where $S_{A\bar{A}}$ denotes the entanglement entropy between the subsystem
$A$ and its complement $\bar{A}$. In the denominator, $H_{B}$ represent the
local Hamiltonian which allows to extract energy $E_{B}$ Hotta (2013). This
will be studied extensively in the future publication Ikeda and Lowe . To
investigate the relation between QET and entanglement entropy on a deeper
level, we also examined into the role of entanglement entropy as an order
parameter of the phase transition. We use the entanglement entropy $S_{LR}$
between the left and right sides of the system. The results are shown in Figs.
1 and 2.

Additionally, in the following we investigate the entanglement spectrum, see
Fig. 3. Let’s first assume that the dimension of the Hilbert space of
subsystem $L$ is not greater than that of subsystem $R$:
$\text{dim}\mathcal{H}_{L}\leq\text{dim}\mathcal{H}_{R}$. The Gram–Schmidt
process for orthonormalizing a quantum state is given as

$\ket{\psi}=\sum_{\alpha=1}^{\text{dim}\mathcal{H}_{L}}\xi_{\alpha}\ket{\psi^{L}_{\alpha}}_{L}\ket{\psi^{R}_{\alpha}}_{R}\,,$
(12)

and the coefficients are denoted by

$\xi_{\alpha}\,,\quad\rm{where}\quad\alpha=1,\cdots,\text{dim}\mathcal{H}_{L}\,.$

These coefficients then make up the entanglement spectrum Pollmann  _et al._
(2010); Ikeda  _et al._ (2023) depicted in Fig. 3 where the upper panel is for
the parameters, $N=8,\,B=0.4,\,J_{K}=0.5$, whereas, for the lower panel we
have $N=8,\,B=0.9,\,J_{K}=0.2$. We observe that the entanglement spectrum is
enhanced for lower value of magnetic field and higher value of Kondo coupling.

Figure 3: Entanglement spectrum for the Kondo chain as obtained by the
Gram–Schmidt process after dividing the system.

##  V Experimental feasibility

While Kondo physics arises in many experimentally relevant platforms, a rather
direct route to implementing the described QET scheme arises in the context of
cold atoms systems, see e.g. Cronenwett  _et al._ (1998); Neupane  _et al._
(2015); Zahid Hasan  _et al._ (2015); Neupane  _et al._ (2013); Beri and
Cooper (2012); Buccheri  _et al._ (2016); Dzsaber  _et al._ (2017). A
particular appealing platform in this regard entails Rydberg atoms. In such
systems one considers (transitions to) highly excited states in e.g. Rubidium
atoms Wu  _et al._ (2021); Saffman  _et al._ (2010); Browaeys and Lahaye
(2020). As such, these electrons in these large orbits can be well manipulated
with light-matter interactions, but more importantly have an intrinsically
large dipole moment that connect neighboring atoms. As a result of the mutual
interactions within an ensemble of Rydberg atoms, only one atom can be excited
as the interaction ensures the other levels get off resonance. In particular
within a volume, set by a critical distance, $R_{b}$ any two atoms cannot be
excited at the same time. This Rydberg blockade enables to incorporate
effective interactions and can, for example, be used to simulate topological
spin liquids Verresen and Vishwanath (2022). In the limiting case where the
dipole interactions are much smaller than the difference in energy levels of
the atoms involved, and thus can be treated perturbatively, the effective
Rydberg blockade interaction terms is proportional to $R^{-6}$ in terms of the
inter-atomic distance $R$.

To investigate the compatibility with an effective $1/R^{6}$ coupling, as in
Refs. Wu  _et al._ (2021); Verresen and Vishwanath (2022); Verresen  _et al._
(2021), we study the dependence of our model, Eq. (7), on the distance $d$
from the impurity, that is shown in Fig. 4. Here we put the impurity at $n=1$
that is coupled with an atom at $n=2~{}(d=1)$ or $n=3~{}(d=2)$ with the
coupling constant $J_{K}$. In both the cases, Alice $(n_{A}=5)$ and Bob
$(n_{B}=8)$ are away from these atoms. As is evident from Eq. 7, the effect of
Rydberg atoms may become significant when $d=2$. We observe, from Fig. 4 that
the teleported energy at Bob’s location for constant $J_{K}$ overlaps for
$d=1$ and $d=2$ at lower value of magnetic field $B$ but differs at higher
values of $B$. We however observe that for our model, Eq. (7), the energy
teleportation does not depend on how far an atom is from the impurity. We also
note that it does not depend on the potential, for example $1/R^{6}$, and
therefore, the energy will be teleported. Indeed, we have corroborated these
statements also numerically within the same setup for general larger
separations $n>3$. This result implies that even without impurity information,
Alice and Bob can know the impurity effects and phase diagram through QET.

Figure 4: The plot shows the dependence of teleported energy at Bob’s location
$(y-axis)$ on the magnetic field $B$ $(x-axis)$ with different position of the
impurity coupling in the Rydberg-like potential:
$J_{K,d=2}/J_{K,d=1}=1/d^{6}$, where $d$ is the distance from the impurity. We
used the following parameters $N=8,n_{A}=5,n_{B}=8,n=1,J=0.5,J_{K,d=1}=0.2$
for our analysis.

##  VI Conclusion and discussion

In conclusion, our study has utilized a recent quantum energy teleportation
strategy to faithfully reproduce the phase diagram of a one-dimensional spin
chain governed by Kondo effect coupling. Remarkably, this was accomplished
even though the entities supplying and receiving energy are located at a
spatial distance from the point impurity, without any direct interaction with
it. By leveraging only local operations on their individual subsystems, and
provided by classical communication, these entities are able to generate phase
diagrams that are in direct agreement with those derived from exact
diagonalization techniques. The underpinning principle that enables this
achievement is an essential correlation between the energy procured via QET
and the quantum entanglement entropy of the system. To corroborate this, we
initially identified that the quantum entanglement entropy operates as the
system’s pertinent order parameter. Intriguingly, shifts in the entanglement
spectra’s gap spacing are found to coincide with peaks in both the quantum
entanglement entropy and the energy metrics acquired through QET. We propose
that these theoretical insights could be substantiated in an experimental
setting, potentially through one-dimensional atomic simulators, such as
Rydberg arrays.

## Acknowledgements

The work of K.I. was supported by the U.S. Department of Energy, Office of
Science, National Quantum Information Science Research Centers, Co-design
Center for Quantum Advantage (C2QA) under Contract No. DESC0012704. R.S. is
supported by the Polish NAWA Bekker program No. BPN/BEK/2021/1/00342 and
Polish NCN Grant No. 2018/30/E/ST2/00432. R.S. also thanks the Institute for
Nuclear Theory at the University of Washington for its kind hospitality and
stimulating research environment. R. J. S acknowledges funding from Trinity
College, University of Cambridge and the Winton programme. This research was
supported in part by the INT’s U.S. Department of Energy grant No. DE-
FG02-00ER41132.

## References

  * Islam  _et al._ (2015) R. Islam, R. Ma, P. M. Preiss, M. E. Tai, A. Lukin, M. Rispoli, and M. Greiner,  (2015), 10.1038/nature15750, arXiv:1509.01160 [cond-mat.quant-gas] .
  * Huang  _et al._ (2020) H.-Y. Huang, R. Kueng, and J. Preskill, Nature Phys. 16, 1050 (2020).
  * Ayral  _et al._ (2023) T. Ayral, P. Besserve, D. Lacroix, and E. A. Ruiz Guzman, (2023), arXiv:2303.04850 [quant-ph] .
  * Beck  _et al._ (2023) D. Beck _et al._ (2023) arXiv:2303.00113 [nucl-ex] .
  * Ikeda (2023a) K. Ikeda, Phys. Rev. D 107, L071502 (2023a), arXiv:2301.11712 [hep-th] .
  * Ikeda (2023b) K. Ikeda, AVS Quantum Science 5, 035002 (2023b).
  * Ikeda (2023c) K. Ikeda, (2023c), arXiv:2301.11884 [quant-ph] .
  * Ikeda and Lowe (2023) K. Ikeda and A. Lowe, (2023), arXiv:2306.08242 [quant-ph] .
  * Ikeda (2023d) K. Ikeda, Phys. Rev. Applied 20, 024051 (2023d), arXiv:2301.02666 [quant-ph] .
  * Ikeda (2023e) K. Ikeda, “Quantum Energy Teleportation with Quantum Computers,”  (2023e).
  * Alloul  _et al._ (2009) H. Alloul, J. Bobroff, M. Gabay, and P. J. Hirschfeld, Rev. Mod. Phys. 81, 45 (2009), arXiv:0711.0877 [cond-mat.str-el] .
  * Slager  _et al._ (2015) R.-J. Slager, L. Rademaker, J. Zaanen, and L. Balents, Phys. Rev. B 92, 085126 (2015).
  * Slager (2019) R.-J. Slager, Journal of Physics and Chemistry of Solids 128, 24 (2019).
  * Teo and Kane (2010) J. C. Y. Teo and C. L. Kane, Phys. Rev. B 82, 115120 (2010).
  * Slager  _et al._ (2014) R.-J. Slager, A. Mesaros, V. Juričić, and J. Zaanen, Phys. Rev. B 90, 241403 (2014).
  * Ran  _et al._ (2009) Y. Ran, Y. Zhang, and A. Vishwanath, Nature Physics 5, 298 (2009).
  * Kondo (1964) J. Kondo, Progress of Theoretical Physics 32, 37 (1964).
  * Cronenwett  _et al._ (1998) S. M. Cronenwett, T. H. Oosterkamp, and L. P. Kouwenhoven, Science 281, 540 (1998).
  * Neupane  _et al._ (2015) M. Neupane, N. Alidoust, I. Belopolski, G. Bian, S.-Y. Xu, D.-J. Kim, P. P. Shibayev, D. S. Sanchez, H. Zheng, T.-R. Chang, H.-T. Jeng, P. S. Riseborough, H. Lin, A. Bansil, T. Durakiewicz, Z. Fisk, and M. Z. Hasan, Physical Review B 92 (2015), 10.1103/physrevb.92.104420.
  * Zahid Hasan  _et al._ (2015) M. Zahid Hasan, S.-Y. Xu, and M. Neupane, “Topological insulators, topological dirac semimetals, topological crystalline insulators, and topological kondo insulators,” in  _Topological Insulators_ (John Wiley & Sons, Ltd, 2015) Chap. 4, pp. 55–100.
  * Neupane  _et al._ (2013) M. Neupane, N. Alidoust, S.-Y. Xu, T. Kondo, Y. Ishida, D. J. Kim, C. Liu, I. Belopolski, Y. J. Jo, T.-R. Chang, H.-T. Jeng, T. Durakiewicz, L. Balicas, H. Lin, A. Bansil, S. Shin, Z. Fisk, and M. Z. Hasan, Nature Communications 4 (2013), 10.1038/ncomms3991.
  * Beri and Cooper (2012) B. Beri and N. R. Cooper, Physical Review Letters 109 (2012), 10.1103/physrevlett.109.156803.
  * Buccheri  _et al._ (2016) F. Buccheri, G. D. Bruce, A. Trombettoni, D. Cassettari, H. Babujian, V. E. Korepin, and P. Sodano, New Journal of Physics 18, 075012 (2016).
  * Dzsaber  _et al._ (2017) S. Dzsaber, L. Prochaska, A. Sidorenko, G. Eguchi, R. Svagera, M. Waas, A. Prokofiev, Q. Si, and S. Paschen, Physical Review Letters 118 (2017), 10.1103/physrevlett.118.246601.
  * Tsvelik and Yevtushenko (2019) A. M. Tsvelik and O. M. Yevtushenko, Phys. Rev. B 100, 165110 (2019).
  * Tomonaga (1950) S.-i. Tomonaga, Progress of Theoretical Physics 5, 544 (1950).
  * Luttinger (1963) J. Luttinger, Journal of mathematical physics 4, 1154 (1963).
  * Lieb (1965) D. M. E. Lieb, Journal of Mathematical Physics 6, 304 (1965).
  * Haldane (1981) F. Haldane, Journal of Physics C: Solid State Physics 14, 2585 (1981).
  * Furusaki (2005) A. Furusaki, Journal of the Physical Society of Japan 74, 73 (2005).
  * Lee (2016) T. Lee, “The tomonaga-luttinger liquid with quantum impurity revisited: Critical line and phase diagram,” (2016), arXiv:1512.08842 [hep-th] .
  * yu Hou  _et al._ (1999) B. yu Hou, X. qiang Xi, and R. hong Yue, Physics Letters A 257, 189 (1999).
  * Zvyagin and Schlottmann (1997) A. A. Zvyagin and P. Schlottmann, Phys. Rev. B 56, 300 (1997).
  * Öz and Klümper (2019) Y. Öz and A. Klümper, Journal of Physics A: Mathematical and Theoretical 52, 325001 (2019).
  * Furusaki and Hikihara (1998a) A. Furusaki and T. Hikihara, Physical Review B 58, 5529 (1998a).
  * Xavier and Pereira (2022) H. B. Xavier and R. G. Pereira, Physical Review B 106 (2022), 10.1103/physrevb.106.064429.
  * Kimura and Ozaki (2019) T. Kimura and S. Ozaki, Phys. Rev. D 99, 014040 (2019).
  * Hotta (2008) M. Hotta, Physics Letters A 372, 5671 (2008), arXiv:0803.1512 [quant-ph] .
  * Hotta (2008) M. Hotta, Phys. Rev. D 78, 045006 (2008).
  * Laflorencie  _et al._ (2008) N. Laflorencie, E. S. Sørensen, and I. Affleck, Journal of Statistical Mechanics: Theory and Experiment 2008, P02007 (2008).
  * Furusaki and Hikihara (1998b) A. Furusaki and T. Hikihara, Phys. Rev. B 58, 5529 (1998b).
  * Hotta (2011) M. Hotta, arXiv e-prints , arXiv:1101.3954 (2011), arXiv:1101.3954 [quant-ph] .
  * Lloyd (1997) S. Lloyd, Phys. Rev. A 56, 3374 (1997).
  * Ito and Sagawa (2015) S. Ito and T. Sagawa, Nature Communications 6, 7498 (2015), arXiv:1406.5810 [cond-mat.stat-mech] .
  * Hotta (2013) M. Hotta, Phys. Rev. A 87, 032313 (2013).
  * (46) K. Ikeda and A. Lowe, To appear .
  * Pollmann  _et al._ (2010) F. Pollmann, A. M. Turner, E. Berg, and M. Oshikawa, Phys. Rev. B 81, 064439 (2010).
  * Ikeda  _et al._ (2023) K. Ikeda, D. E. Kharzeev, R. Meyer, and S. Shi, (2023), arXiv:2305.00996 [hep-ph] .
  * Wu  _et al._ (2021) X. Wu, X. Liang, Y. Tian, F. Yang, C. Chen, Y.-C. Liu, M. K. Tey, and L. You, Chinese Physics B 30, 020305 (2021).
  * Saffman  _et al._ (2010) M. Saffman, T. G. Walker, and K. Mølmer, Rev. Mod. Phys. 82, 2313 (2010).
  * Browaeys and Lahaye (2020) A. Browaeys and T. Lahaye, Nature Physics 16, 132 (2020).
  * Verresen and Vishwanath (2022) R. Verresen and A. Vishwanath, Phys. Rev. X 12, 041029 (2022).
  * Verresen  _et al._ (2021) R. Verresen, M. D. Lukin, and A. Vishwanath, Phys. Rev. X 11, 031005 (2021).


(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: 1McGill University 2The Chinese University of Hong Kong
11email<EMAIL_ADDRESS>11email<EMAIL_ADDRESS>11email:
<EMAIL_ADDRESS>
# FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video
Editing

Youyuan Zhang 11 Xuan Ju Corresponding author.22 James J. Clark ⋆ 11

###### Abstract

Diffusion models have demonstrated remarkable capabilities in text-to-image
and text-to-video generation, opening up possibilities for video editing based
on textual input. However, the computational cost associated with sequential
sampling in diffusion models poses challenges for efficient video editing.
Existing approaches relying on image generation models for video editing
suffer from time-consuming one-shot fine-tuning, additional condition
extraction, or DDIM inversion, making real-time applications impractical. In
this work, we propose _FastVideoEdit_ , an efficient zero-shot video editing
approach inspired by Consistency Models (CMs). By leveraging the self-
consistency property of CMs, we eliminate the need for time-consuming
inversion or additional condition extraction, reducing editing time. Our
method enables direct mapping from source video to target video with strong
preservation ability utilizing a special variance schedule. This results in
improved speed advantages, as fewer sampling steps can be used while
maintaining comparable generation quality. Experimental results validate the
state-of-the-art performance and speed advantages of _FastVideoEdit_ across
evaluation metrics encompassing editing speed, temporal consistency, and text-
video alignment.

###### Keywords:

Video Editing Diffusion Models Consistency Models

Figure 1: Editing Results of _FastVideoEdit_. _FastVideoEdit_ offers
efficient, consistent, high-quality, and text-aligned editing capabilities for
both artificial (left col) and natural (right col) videos. The top row
displays the source video, while the second and third rows showcase two edited
videos. Each row features a text prompt at the top, with the edited words
highlighted in red. This visual representation effectively demonstrates how
our method can successfully achieve desired edits such as attribute change,
object change, background change, and style change.

##  1 Introduction

Diffusion models [15, 35, 14, 1] have gained significant attention due to
their remarkable capabilities in text-to-image [31, 35, 15, 17] and text-to-
video generation [14, 34, 3, 12, 4]. Leveraging the capabilities of these
models, it becomes feasible to manipulate videos [4] based on textual input,
holding great potential for various applications in areas such as film
production and content creation.

However, the computational cost associated with sequential sampling in
diffusion models presents a significant challenge for efficient inference,
especially in video editing scenarios where a set of frames need to be
processed. Moreover, the absence of high-quality open-source video diffusion
models [10, 28, 10] that can generate consistent editing results within a
single test time inference, combined with the constraints on video duration of
video diffusion models, has led to the adoption of existing image generation
models for achieving accurate video editing [2, 29, 11]. To align the
distribution between image and video models and perform accurate video
editing, some methods employ a test-time one-shot fine-tuning for inflated
image generation model on each input video [38, 33, 23, 37, 26]. However, this
process further exacerbates the time-consuming nature of the editing process,
which makes it impractical for real-time applications.

To enable faster video editing, three types of zero-shot methods have been
proposed in the literature: (1) Layer-atlas-based methods [2, 21, 7], which
involve editing the video on a flattened texture map and ensuring the temporal
consistency by guaranteeing texture map consistency. However, the absence of a
3D motion prior in the 2D atlas approach results in suboptimal performance.
(2) Dual-branch methods [29, 6, 11, 9], which leverage Denoising Diffusion
Implicit Models (DDIM) [35] to extract source video features and generate
novel content based on the target diffusion branch. The use of DDIM inversion
leads to a doubling of the inference time required for video editing. (3)
Additional conditional constraints incorporating methods [41, 37, 8, 43],
which involve directly adding noise to the source video and denoising the
noisy video using a conditioned diffusion model for preserving essential
content while imposing restrictions on the editing process. While these
methods are efficient during diffusion model inference, they do require
additional information extraction, which slows down the overall speed of the
process.

To address the issue of long computational times encountered in previous video
editing methods, we introduce _FastVideoEdit_ , which is inspired by recent
advances in Consistency Models (CMs) [36]. Specifically, _FastVideoEdit_ is a
zero-shot video editing approach that not only achieves state-of-the-art
performance but also significantly reduces editing time by eliminating the
need for time-consuming inversion or additional condition extraction steps.
The key insight of our proposed method is that the self-consistency property
of CMs enables a special variance schedule that facilitates the editing
process, transforming it from a process of adding noise and then denoising to
one of a direct mapping from source video to target video. Furthermore, the
content preservation capability of CMs enables the use of fewer sampling steps
while maintaining comparable generation quality, which results in an improved
speed advantage of _FastVideoEdit_.

To evaluate _FastVideoEdit_ , we consider metrics that encompass editing
speed, temporal consistency, and text-video alignment. We compare the
performance of _FastVideoEdit_ with previous video editing methods using the
TGVE 2023 open-source dataset [39] as our benchmark. The results demonstrate
the superior performance of _FastVideoEdit_ in terms of editing quality.
Additionally, _FastVideoEdit_ achieves this superior performance while
requiring significantly less time for video editing tasks. This shows the
efficiency and effectiveness of our approach, making it a standout choice for
efficient high-quality video editing.

##  2 Related Work

###  2.1 Video Editing with Diffusion Models

The remarkable success of diffusion-based text-to-image [31, 35, 15, 19, 18]
and text-to-video generation models [14, 34, 3, 12, 4] has opened up new
possibilities for exciting opportunities in text-based image [13, 17] and
video editing [10]. Although editing video directly through video diffusion
models [10, 28, 10] show high temporal consistency, the challenges associated
with extensive video model training, unstable generation quality, and video
duration time limit make using inflated off-the-shelf image generation models
a preferable choice for video editing, which inflating 2D model to 3D with an
additional temporal channel.

Specifically, several works require a test-time one-shot fine-tuning on the
inflated image generation model with each input video [38, 33, 23, 37, 26],
which is time-consuming and too slow for real-time applications. Zero-shot
video editing methods [2, 21, 7, 29, 13, 37, 6, 41, 11, 43, 9, 8] leverage
training-free editing techniques with specialized modules to enhance temporal
consistency across frames, which provide a practical and efficient solution
for editing videos without the need of extensive training. Specifically,
layer-atlas-based methods [2, 21, 7] edit the video on a flattened texture
map, however the lack of 3d motion prior in 2d atlas leads to suboptimal
performance. FateZero [29] solves this problem with a two-branch inflated
image diffusion model that merges attention features of the structural
preservation branch and editing branch. Similarly, Text2Video-Zero [22] and
Pix2Video [6] align the feature of the source image and target image via an
attention operation. To enhance pixel-level temporal consistency, Rerender A
Video [41], TokenFlow [11], and Flatten [9] extract temporal-aware inter-frame
features to propagate the edits throughout the video. However, previous zero-
shot methods that relied on flattened image diffusion were limited by the need
for DDIM inversion or additional conditional constraints (e.g., optical flow),
resulting in a long runtime. In contrast, our proposed _FastVideoEdit_
directly incorporates editing into the inference process by leveraging
consistency models [36], which ensures both runtime efficiency and effective
modifications.

###  2.2 Efficient Diffusion Models

To tackle the computational time limitations of diffusion models caused by the
sequential sampling strategy, faster numerical ODE solvers [35, 42, 24] or
distillation techniques [25, 32, 27, 44] have been employed. While these
methods can be integrated into existing diffusion-based video editing
techniques, they still face the challenge of requiring DDIM inversion or
additional conditional constraints for essential content preservation.

Recently, the introduction of Consistency Models (CMs) [36, 40] has enabled
faster generation by sampling along a trajectory map, thereby opening up
exciting possibilities for more efficient video editing techniques. The few-
step sampling strategy is particularly suitable for efficient video editing
with a fast sampling speed and strong reconstruction ability. _FastVideoEdit_
leverages the self-consistency characteristic of CMs, where the improved
essential content preservation ability eliminates the need for accurate DDIM
inversion and additional conditional constraints. Concurrent to our approach,
OCD [20] separates diffusion sampling for edited objects and background areas,
focusing most denoising steps on the former to enhance efficiency.
_FastVideoEdit_ can be directly combined with OCD to further enhance the
overall efficiency of video editing.

##  3 Preliminaries

Diffusion models include a forward process that adds Gaussian noise $\epsilon$
to convert clean sample $z_{0}$ to noise sample $z_{T}$, and a backward
process that iteratively performs denoising from $z_{T}$ to $z_{0}$, where $T$
represents the total number of timesteps. The denoising process of DDPM
sampling [15] at step $t$ can be formulated as:

$\displaystyle z_{t-1}=$
$\displaystyle\sqrt{{{\alpha}}_{t-1}}\left(\frac{z_{t}-\sqrt{1-{{\alpha}}_{t}}\varepsilon_{\theta}(z_{t},t)}{\sqrt{{\alpha}}_{t}}\right)$
(predicted $z_{0}$) (1)
$\displaystyle+\sqrt{1-{\alpha}_{t-1}-\sigma_{t}^{2}}\cdot\varepsilon_{\theta}(z_{t},t)$
(direction to $z_{t}$)
$\displaystyle+\sigma_{t}\varepsilon_{t}\quad\text{where
}\varepsilon_{t}\sim\mathcal{N}(\bm{0},\bm{I})$ (random noise).

By setting $\sigma_{t}$ to zero, DDIM sampling [35] results in an implicit
probabilistic model with a deterministic forward process:

$\bar{z}_{0}=f_{\theta}(z_{t},t)=\left(z_{t}-\sqrt{1-{\alpha}_{t}}\cdot\varepsilon_{\theta}(z_{t},t)\right)/\sqrt{{\alpha}_{t}}.$
(2)

Following DDIM, we can use the function $f_{\theta}$ to predict and
reconstruct $\bar{z_{0}}$ given noise sample $z_{t}$, where
$t\sim\left[1,T\right]$, $\alpha$ is the hyper-parameter,
$\varepsilon_{\theta}$ is a learnable network, and $T$ represents the total
number of timesteps.

Sampling in CMs [36] is carried out through a sequence of timesteps
$\tau_{1:n}\in[t_{0},T]$. Starting from an initial noise $\hat{z}_{T}$ and
$z_{0}^{(T)}=f_{\theta}(\hat{z}_{T},T)$, at each time-step $\tau_{i}$, the
process samples $\varepsilon\sim\mathcal{N}(\bm{0},\bm{I})$ and iteratively
updates the Multistep Consistency Sampling process through the following
equation:

$\displaystyle\hat{z}_{\tau_{i}}$
$\displaystyle=z_{0}^{(\tau_{i+1})}+\sqrt{\tau_{i}^{2}-t_{0}^{2}}\varepsilon$
(3) $\displaystyle z_{0}^{(\tau_{i})}$
$\displaystyle=f_{\theta}(\hat{z}_{\tau_{i}},\tau_{i}).$

When combined with a condition $c$ with classifier-free guidance [16],
sampling in CMs at $\tau_{i}$ starts with
$\varepsilon\sim\mathcal{N}(\bm{0},\bm{I})$ and updates through:

$\displaystyle\hat{z}_{\tau_{i}}$
$\displaystyle=\sqrt{{\alpha}_{\tau_{i}}}z_{0}^{(\tau_{i+1})}+\sigma_{\tau_{i}}\varepsilon,$
(4) $\displaystyle z_{0}^{(\tau_{i})}$
$\displaystyle=f_{\theta}(\hat{z}_{\tau_{i}},\tau_{i},c).$

Consider a special case of Eq. 1 where $\sigma_{t}$ is chosen as
$\sqrt{1-\alpha_{t-1}}$ at all times $t$. Then the DDPM forward process
naturally aligns with the Multistep Consistency Sampling, and the second term
of Eq. 1 vanishes:

$\displaystyle z_{t-1}=$
$\displaystyle\sqrt{{{\alpha}}_{t-1}}\left(\frac{z_{t}-\sqrt{1-{{\alpha}}_{t}}\varepsilon_{\theta}(z_{t},t)}{\sqrt{{\alpha}}_{t}}\right)$
(predicted $z_{0}$) (5)
$\displaystyle+\sqrt{1-\alpha_{t-1}}\varepsilon_{t}\quad\varepsilon_{t}\sim\mathcal{N}(\bm{0},\bm{I})$
(random noise).

Consider
$f(z_{t},t;z_{0})=\left(z_{t}-\sqrt{1-{\alpha}_{t}}\varepsilon^{\prime}(z_{t},t;z_{0})\right)/\sqrt{{\alpha}_{t}}$,
where the initial $z_{0}$ is available and we replace the parameterized noise
predictor $\varepsilon_{\theta}$ with $\varepsilon^{\prime}$ more generally.
Eq. 5 turns into the following expression:

$\displaystyle
z_{t-1}=\sqrt{{{\alpha}}_{t-1}}f(z_{t},t;z_{0})+\sqrt{1-\alpha_{t-1}}\varepsilon_{t}$
(6)

which is in the same form as the Multistep Consistency Sampling step in Eq 4.

In order to make $f(z_{t},t)$ self-consistent so that it can be considered as
a consistency function, i.e., $f(z_{t},t;z_{0})=z_{0}$, we can directly solve
the equation and $\varepsilon^{\prime}$ can be computed without
parameterization:

$\varepsilon^{\text{cons}}=\varepsilon^{\prime}(z_{t},t;z_{0})=\frac{z_{t}-\sqrt{{\alpha}_{t}}z_{0}}{\sqrt{1-{\alpha}_{t}}}.$
(7)

We arrive at a non-Markovian forward process, in which $z_{t}$ directly points
to the ground truth $z_{0}$ without neural prediction, and $z_{t-1}$ does not
depend on the previous step $z_{t}$ like a consistency model.

##  4 Method

The task of video editing can be described as the following: Given an ordered
set of $m$ source video frames
$\mathcal{I}_{src}=\\{I_{src}^{1},I_{src}^{2},...,I_{src}^{m}\\}$ and a source
prompt $\mathcal{P}_{src}$ describing the source video, we aim to generate an
edited video with temporally consistent frames
$\mathcal{I}_{edit}=\\{I_{edit}^{1},I_{edit}^{2},...,I_{edit}^{m}\\}$
according to a target prompt $\mathcal{P}_{tgt}$.

This paper introduces _FastVideoEdit_ , an end-to-end video edit framework
that edits video efficiently while producing high-quality and temporally
consistent editing content. Notably, our method achieves better background
preservation compared with existing methods when editing foreground object-
level attributes. Unlike many existing methods that depend on additional
estimations such as depth control, edge control, or optical flow,
_FastVideoEdit_ requires only the source video frames and prompts as input
throughout the editing process.

###  4.1 Video Reconstruction with Consistency Model

To our knowledge, _FastVideoEdit_ is the first method in video editing that
eliminates the need for the DDIM inversion process while simultaneously
performing a complete denoising process on individual video frames. To enable
direct editing of the source video without the need for the inversion process,
we leverage a consistency model inspired by InfEdit [40]. The key idea to
reconstruct source latent is to start with randomly sampled reconstruction
noise rather than randomly initialized noisy latents. Following the Multistep
Consistency Sampling in Eq 3, we sample a noise
$\varepsilon_{t}^{\text{cons}}$ at each timestep $t$ and the noisy latent
$z_{t}^{\text{src}}$ becomes directly tractable when $z_{0}^{\text{src}}$ is
given in the editing problem. Instead of denoising the randomly initialized
noisy latent $z_{T}^{\text{src}}$, the whole trajectory of
$\\{z_{t}^{\text{src}}\\}$ is obtained directly from the sampled noise
trajectory $\\{\varepsilon_{t}^{\text{cons}}\\}$, and in the reverse direction
each $\varepsilon_{t}^{\text{cons}}$ can be used to reconstruct
$z_{0}^{\text{src}}$ given $z_{t}^{\text{src}}$. The mappings between
$z_{t}^{\text{src}}$ and $\\{\varepsilon_{t}^{\text{cons}}\\}$ given
$z_{0}^{\text{src}}$ are given by:

$\displaystyle z_{t}^{\text{src}}$
$\displaystyle=\sqrt{{\alpha}_{t}}z_{0}^{\text{src}}+\sqrt{1-{\alpha}_{t}}\varepsilon_{t}^{\text{cons}}$
(8) $\displaystyle\varepsilon^{\text{cons}}_{t}$
$\displaystyle=(z_{t}^{\text{src}}-\sqrt{{\alpha}_{t}}z_{0}^{\text{src}})/\sqrt{1-{\alpha}_{t}}.$

where $\varepsilon_{t}^{\text{cons}}\sim\mathcal{N}(\bm{0},\bm{I})$ is sampled
independently at each timestep. As a result, the reconstructed latent
$z_{t}=z_{0}$ is guaranteed at each timestep using Eq (2).

Figure 2: Overview of _FastVideoEdit_. Our model directly denoises three
branches of batch frames using three attention control methos: CF-Masa, Re-CA
and Bg-Masa. The model uses batch consistency sampling (BCS) with LCMs to
improve efficiency, background latent replacement to align editing content
with source video and TokenFlow propagation to further improve temporal
consistency.

###  4.2 Video Editing with Consistency Model

This section introduces the method to compute $z_{0}^{\text{edit}}$ given
$z_{0}^{\text{src}}$. In addition to $z_{t}^{\text{src}}$ and
$\varepsilon^{\text{cons}}_{t}$ obtained from Eq (8), we need to predict the
editing noise
$\varepsilon_{\theta}(z_{t}^{\text{edit}},t,\mathcal{P}_{\text{tgt}})$ to
generate the editing latent $z_{0}^{\text{edit}}$ according to target prompt
$\mathcal{P}_{\text{tgt}}$. Due to the self-consistency property of LCMs, the
gap between
$\varepsilon_{\theta}(z_{t}^{\text{edit}},t,\mathcal{P}_{\text{tgt}})$ and
$\varepsilon_{t}^{\text{edit}}$ is small. Therefore, using the noise
calibration $\Delta\varepsilon_{t}^{\text{cons}}$ from
$\varepsilon_{\theta}(z_{t}^{\text{src}},t,\mathcal{P}_{\text{src}})$ to the
ground-truth source reconstruction noise $\varepsilon^{\text{cons}}_{t}$, we
can estimate the editing reconstruction noise as well as the editing latent
$z_{0}^{\text{edit}}$ at each timestep $t$:

$\displaystyle\Delta\varepsilon_{t}^{\text{cons}}$
$\displaystyle=\varepsilon^{\text{cons}}_{t}-\varepsilon_{\theta}(z_{t}^{\text{src}},t,\mathcal{P}_{s})$
(9) $\displaystyle\varepsilon_{t}^{\text{edit}}$
$\displaystyle=\varepsilon_{\theta}(z_{t}^{\text{edit}},t,\mathcal{P}_{t})+\Delta\varepsilon_{t}^{\text{cons}}$
$\displaystyle z_{0}^{\text{edit}}$
$\displaystyle=\left(z_{t}^{\text{edit}}-\sqrt{1-{\alpha}_{t}}\cdot\varepsilon_{t}^{\text{edit}}\right)/\sqrt{{\alpha}_{t}}.$

Compared with editing a single frame, we impose the constraints that the
initial latent and random noise sampled at each timestep are identical across
all frames. Since the forward process of the denoising network
$\varepsilon_{\theta}(\cdot,\cdot,\cdot)$ as well as the calibration process
of noise and the updating process of latent are all deterministic relative to
their inputs, identical initial latents and noise samples at each timestep
result in identical output latents when source latents are also identical. In
practice, if source latents are temporally consistent and close to each other,
the output latents should also maintain good temporal consistency.

###  4.3 Batch Attention Control

As an end-to-end inference-based editing framework _FastVideoEdit_ starts with
directly denoising the batched latent $\mathcal{Z}_{t}^{\text{edit}}$
according to the target prompt $\mathcal{P}_{\text{tgt}}$. A naive way of
editing the target frame latent $z_{0}^{\text{src}}$ by the target prompt is
to denoise the DDIM inversion $z_{T}^{\text{inv}}$ of $z_{0}^{\text{src}}$
iteratively through
$\varepsilon_{\theta}(z_{t}^{\text{inv}},t,\mathcal{P}_{\text{tgt}})$. In
section 4.2, we introduced consistency model-based batch editing which
leverages the property of LCMs to skip the time-consuming DDIM inversion
process and directly denoise randomly initialized latent while keeping content
aligned faithfully with source frames. However, without additional control,
denoising conditioned on a target prompt $\mathcal{P}_{\text{tgt}}$ can still
produce editing content distinct from the source content.

Inspired by MasaCtrl [5] and Prompt-to-prompt [13], we propose Cross-Frame
Mutual Self-Attention (CF-Masa) and Re-weighted Cross Attention (Re-CA) to
allow further attention control when denoising the $z_{t}^{\text{edit}}$
conditioned on $\mathcal{P}_{\text{tgt}}$. Specifically, we concurrently
denoise two batched latents
$[\mathcal{Z}_{t}^{\text{src}},\mathcal{Z}_{t}^{\text{edit}}]$ conditioned on
$[\mathcal{P}_{\text{src}},\mathcal{P}_{\text{tgt}}]$ respectively. The
proposed CF-Masa and Re-CA can be directly applied in the forward process of
$\varepsilon_{\theta}([\mathcal{Z}_{t}^{\text{src}},\mathcal{Z}_{t}^{\text{edit}}],t,[\mathcal{P}_{\text{src}},\mathcal{P}_{\text{tgt}}])$.

####  4.3.1 Cross-Frame Mutual Self-Attention

The denoising UNet consists of different size downsample/upsample blocks and a
middle block, which have four resolution levels in the latent space. Each
resolution level incorporates a 2D convolution layer followed by self-
attention and cross-attention layers. The attention mechanism can be
formulated as:

$\displaystyle\text{attn}(Q,K,V)=\text{softmax}(\frac{QK^{T}}{\sqrt{d}}V).$
(10)

In self-attention layers, $Q,K,V$ are the query, key, and value features
obtained by projecting the same spatial features. Without attention control,
the self-attention output of source branch
$\text{attn}(Q^{\text{src}},K^{\text{src}},V^{\text{src}})$ and editing branch
$\text{attn}(Q^{\text{edit}},K^{\text{edit}},V^{\text{edit}})$ are computed
concurrently and independently of each other. We make two changes on self-
attention layers to preserve content consistency as well as temporal
consistency between and within editing latent and source latent. In contrast
to MasaCtrl [5], the preservation of content consistency in _FastVideoEdit_ is
achieved by replacing $Q^{\text{edit}}$ and $K^{\text{edit}}$ with
$Q^{\text{src}}$ and $K^{\text{src}}$ after a fixed step $t_{s}$ and the
editing branch remains unchanged before $t_{s}$. To further maintain temporal
consistency across batched latents within a branch, we concatenate the key
features $[K_{1},K_{2},...,K_{m}]$ and value features
$[V_{1},V_{2},...,V_{m}]$ along their sequence length dimension resulting in
the final format becomes:

$\displaystyle\text{CF-
Masa}(\\{Q_{i}^{\text{edit}},K_{i}^{\text{edit}},V_{i}^{\text{edit}}\\},t)$
(11)
$\displaystyle:=\begin{cases}\\{Q_{i}^{\text{src}},\text{concat}\\{K^{\text{src}}\\},\text{concat}\\{V^{\text{edit}}\\}\\}&t\geq
t_{s}\\\
\\{Q_{i}^{\text{edit}},\text{concat}\\{K^{\text{edit}}\\},\text{concat}\\{V^{\text{edit}}\\}\\}&t<t_{s}\end{cases}.$

####  4.3.2 Re-weighted Cross Attention

The forward process of cross-attention can be edited in a similar way to self-
attention. In cross-attention layers, $Q$ is the set of query features
obtained obtaining by projecting spatial features coming from self-attention
layer, $K,V$ are obtained from the prompt embeddings. By replacing the cross-
attention map of the editing branch with that of the source branch [13], the
scattering from source prompt mutual content to the source spatial features
can be maintained on editing spatial features. To further enhance the effect
of the editing token, the corresponding attention map of the editing token can
be multiplied by a replace scale $r\geq 1$. The resulting formulation of the
Re-weighted Cross Attention is given by:

$\displaystyle\text{Refine}(A^{\text{src}},A^{\text{edit}})_{i,j}=\begin{cases}\left(A^{\text{edit}}\right)_{i,j}&\text{if}\
f_{\mathcal{P}}(j)=\text{None}\\\
\left(A^{\text{edit}}\right)_{i,f_{\mathcal{P}}(j)}&\text{otherwise}\end{cases}$
(12) $\displaystyle\text{Re-
CA}(A^{\text{src}},A^{\text{edit}},t):=\begin{cases}r\cdot\text{Refine}(A^{\text{src}},A^{\text{edit}})&t\geq
t_{c}\\\ A^{\text{edit}}&t<t_{c}\end{cases}$

where $f_{\mathcal{P}}(\cdot)$ is the alignment function indicating the source
prompt token index of the $j^{th}$ token in the target prompt and None if
missing.

###  4.4 Background Preservation via Latent Replacement

There is a trade-off in existing video editing methods between the editing
effect of foreground objects and content preservation of background. Changing
the attributes of an object in the foreground usually makes the background
more consistent with the change. This is because the control methods that are
applied to the forward process are not strict control over the latent space.
Therefore the change of tokens in the target prompt also influences irrelevant
regions of editing latent through attention mechanisms. Compared with state-
of-the-art video editing methods, a significant advantage of _FastVideoEdit_
is the accuracy of foreground editing. This is shown in both quantative and
qualitative results in Sec. 5. We achieve this by multiple designs of
_FastVideoEdit_. Consistent initial latents and noise in Batch Consistency
Sampling algorithm and attention control both provide faithful editing
concerning the source video. In addition to this, we propose further
background preservation strategies to enhance the faithfulness of the edited
content to the source content. Specifically, we propose to simultaneously
denoise a background branch that maintains the structure information of the
editing branch while aligning content with the source branch. Based on the
background branch, we additionally propose a latent replacement algorithm that
replaces the background part in the editing latent with the corresponding part
in the background latent.

####  4.4.1 Background Branch

By simultaneously denoising a background branch conditioned on
$\mathcal{P}_{src}$ and imposing self-attention control from the source branch
and editing branch, we expect the background branch to maintain the structure
of the editing branch and the content of the source branch. We modify the
self-attention process of the background branch as follows:

$\displaystyle\text{Bg-
Masa}(\\{Q_{i}^{\text{bg}},K_{i}^{\text{bg}},V_{i}^{\text{bg}}\\},t)$ (13)
$\displaystyle:=\begin{cases}\\{Q_{i}^{\text{src}},\text{concat}\\{K^{\text{src}}\\},\text{concat}\\{V^{\text{src}}\\}\\}&t\geq
t_{bg}\\\
\\{Q_{i}^{\text{edit}},\text{concat}\\{K^{\text{src}}\\},\text{concat}\\{V^{\text{src}}\\}\\}&t<t_{bg}.\end{cases}$

To maintain the editing structure and source content, we employ a similar
editing approach to MasaCtrl [5] since query features from the edit branch are
used to maintain structure information. Meanwhile, the key and value features
are copied from the source branch to maintain consistency with the source
content. Note that the joint attention is working at early timestep instead of
later timesteps as described in MasaCtrl [5] because our observation is that
the structure is formed at early steps and content details are refined at
later steps.

####  4.4.2 Latent Replacement

At the end of each denoising step, we employ the latent replacement operation
to replace the background region of the editing latent with the corresponding
region of the source latent. The region is determined by computing the
relative region from a cross-attention map. Specifically, given a cross-
attention map $(A^{\text{edit}})_{m\times n}$, we obtain a replacement map
$(M^{\text{edit}})_{m}$ where $m$ is the sequence length of the attention map
or the size of the feature map, and $n$ is the number of tokens in
$\mathcal{P}_{tgt}$. The replacement map is computed as follows:

$\displaystyle(\hat{A}^{\text{edit}})_{i}$
$\displaystyle=\frac{\Sigma_{j}(A^{\text{edit}})_{i,j}\cdot\mathbf{I}_{f_{\mathcal{P}}(j)\neq\text{None}}}{\Sigma_{j}(A^{\text{edit}})_{i,j}}$
(14) $\displaystyle(M^{\text{edit}})_{i}$
$\displaystyle=\mathbf{I}_{(\hat{A}^{\text{edit}})_{i}}\geq\text{thresh}_{\text{edit}}.$

Intuitively, the replacement map has $1$ at positions where the edited tokens
receive high attention scores among all the tokens, and $0$ anywhere else. In
practice, $A^{\text{edit}}$ is obtained by averaging among all the cross-
attention maps of the same size in a fixed resolution level. The replaced
edited latent at the end of denoising step $t$ is:

$\displaystyle(z_{t}^{\text{edit}})=M_{t}^{\text{edit}}\odot(z_{t}^{\text{edit}})+(1-M_{t}^{\text{edit}})\odot(z_{t}^{\text{bg}}).$
(15)

###  4.5 Frame Consistency with Tokenflow

Following [11], we apply tokenflow to improve temporal consistency across
frames. Tokenflow is a plug-and-play module that can be applied at each layer
of the denoising network. The idea of Tokenflow is to first select and denoise
a group of keyframes, and then replace the original spatial features with the
weighted sum of the two most similar spatial features from two adjacent
keyframes when denoising each frame latent. In the first stage, Tokenflow
selects a group of keyframes of indices $\kappa$ and in each layer at each
step and store $\mathbf{T}_{base}=\\{\phi(z^{i})\\}_{i\in\kappa}$, where
$\phi(\cdot)$ maps the latent to its spatial features $(z^{i})$. When
computing the features of an arbitrary frame latent $z^{i}$, the method
queries its two adjacent frames latent of indices $i-$ and $i+$, and gets the
closest feature index $\gamma^{i\pm}[p]$ for each of its feature indexed $p$
as follows:

$\gamma^{i\pm}[p]=\operatorname*{arg\,min}_{q}{\mathcal{D}\left({\phi({z}^{i})[p]},{\phi({z}^{i\pm})[q]}\right)}$
(16)

where $\mathcal{D}$ represents cosine distance of two features. The output
weighted spatial features of frame latent $z_{i}$ therefore become:

$\mathcal{F}_{\gamma}(\mathbf{T}_{base},i,p)=w_{i}\cdot\phi(z^{i+})[\gamma^{i+}[p]]\;+\;(1-w_{i})\cdot\phi(z^{i-})[\gamma^{i-}[p]].$
(17)

In practice, Tokenflow is a plug-and-play operation that can be applied after
the self-attention layer. It replaces the original output of spatial features
$\phi(z^{i})$ of the original frame latent with the features of weighted sum
of two adjacent key frames
$\\{\mathcal{F}_{\gamma}(\mathbf{T}_{base},i,p)\\}_{p}$.

The overall _FastVideoEdit_ algorithm is shown in Algorithm 1 and Figure 2.

Algorithm 1 _FastVideoEdit_ editing

1:For abbreviation, we denote $\mathcal{A}\sim\mathcal{P}$ as every element in
the $\mathcal{A}$ has the same value sampled from distribution $\mathcal{P}$.

2:Input:

3: Latent Consistency Model $\varepsilon_{\theta}(\cdot,\cdot,\cdot)$

4: Sequence of timesteps $\tau_{1}>\tau_{2}>\cdots>\tau_{N-1}$

5: Batched source latents
$\mathcal{Z}_{0}^{\text{src}}=\\{z_{0}^{\text{src},(i)}\ |\ 1\leq i\leq m\\}$

6: Source and target prompts $\mathcal{P}_{src},\mathcal{P}_{tgt}$

7:Set batch attention control on $\varepsilon_{\theta}(\cdot,\cdot,\cdot)$

8:Set Tokenflow propagation on $\varepsilon_{\theta}(\cdot,\cdot,\cdot)$

9:Initial batched latents
$\mathcal{Z}_{\tau_{1}}^{\text{src}}=\mathcal{Z}_{\tau_{1}}^{\text{edit}}=\mathcal{Z}_{\tau_{1}}^{\text{bg}}\sim\mathcal{N}(\bm{0},\bm{I})$

10:Compute $\\{\varepsilon^{\text{cons}}_{\tau_{1}}\\}$ using Eq 8

11:for $n=1$ to $N-1$ do

12: Compute $\mathbf{T}_{\text{base}}^{\text{edit}}$ and
$\mathbf{T}_{\text{base}}^{\text{edit}}$

13: Denoise three branches
$\\{\varepsilon_{\theta}(\\{z^{\text{src}}_{\tau_{n}},z^{\text{edit}}_{\tau_{n}},z^{\text{bg}}_{\tau_{n}}\\},{\tau_{n}},\\{\mathcal{P}_{src},\mathcal{P}_{tgt},\mathcal{P}_{src})\\};\mathbf{T}_{\text{base}}\\}$

14: Update $\mathcal{Z}^{\text{src}}_{\tau_{n+1}}$ using Eq 8

15: Update $\mathcal{Z}^{\text{edit}}_{0}$ and $\mathcal{Z}^{\text{bg}}_{0}$
using Eq 9

16: Sample new reconstruction noise
$\\{\varepsilon_{\tau_{n+1}}^{\text{cons}}\\}\sim\mathcal{N}(\bm{0},\bm{I})$

17: Update $\mathcal{Z}^{\text{edit}}_{\tau_{n+1}}$ and
$\mathcal{Z}^{\text{bg}}_{\tau_{n+1}}$ using Eq 8

18: Replace latents $\mathcal{Z}^{\text{edit}}_{\tau_{n+1}}$ using Eq 14 and
15

19:end for

20:Output: $\mathcal{Z}_{0}^{\textrm{edit}}$

##  5 Experiments

In this section, we first introduce the evaluation benchmark and evaluation
metrics used in our experiment in Sec. 5.1. Following that, we present a
quantitative comparison of our methods in Sec. 5.2 and a qualitative
comparison in Sec. 5.3.

###  5.1 Evaluation Benchmark and Metrics

##### Evaluation Dataset.

For the evaluation of video editing, we utilize the TGVE 2023 open-source
dataset [39] as our benchmark. This dataset consists of 76 videos, each
containing 32 frames with a resolution of 480x480 pixels.

##### Evaluation Metrics.

Following previous work [29, 11], we evaluate the temporal consistency of our
approach by utilizing clip similarity [30] among frames (‘Tem-Con’).
Additionally, we measure the frame-wise editing accuracy through two metrics.
‘Txt-Sim’ for clip similarity between the embeddings of text and image and
‘Clip-Acc’ for the percentage of frames where the edited image has a higher
CLIP similarity to the target prompt compared to the source prompt.
Furthermore, as an additional evaluation metric, we measure the time
consumption of editing $32$ frames’ video using _FastVideoEdit_ and previous
methods in both the inversion and forward processes to evaluate the speed.

###  5.2 Quantitative Comparison

In Tab. 1 we compare _FastVideoEdit_ with two additional conditional
constraints incorporating methods Rerender [41] and Text2Video-Zero [22] as
well as three dual-branch methods FateZero [29], Pix2Video [6], and TokenFlow
[11].

The results demonstrate that _FastVideoEdit_ achieves state-of-the-art
performance in terms of temporal consistency and per-frame editing accuracy,
while significantly reducing the time required for the editing process.
Comparatively, our method outperforms previous additional conditional
constraints incorporating methods and dual-branch methods in terms of
efficiency, delivering high-quality results in less time. The reduction in
runtime originates from two aspects: the elimination of inversion and
additional condition feature extraction, and the use of fewer sampling steps.
This highlights the effectiveness and efficiency of _FastVideoEdit_ in video
editing tasks.

Table 1: Comparison of _FastVideoEdit_ with previous video editing methods.
Bold indicates the best. Underline indicates the second best.

Model |  CLIP Metrics$\uparrow$ |  Time$\downarrow$
---|---|---
Tem-Con | Txt-Sim | Clip-Acc | Inversion | Forward | Sum
Rerender [41] | 95.7 | 25.0 | 48.5 | - | 174.3 | 174.3
Text2Video-Zero [22] | 96.9 | 27.1 | 70.7 | - | 131.0 | 131.0
FateZero [29] | 95.7 | 24.9 | 35.8 | 233.7 | 347.0 | 581.7
Pix2Video [6] | 96.0 | 27.5 | 68.5 | 185.3 | 213.0 | 399.3
TokenFlow [11] | 96.5 | 25.5 | 54.7 | 176.5 | 115.9 | 292.4
Ours | 96.5 | 27.7 | 71.1 | - | 61.7 | 61.7

###  5.3 Qualitative Comparison

Qualitative comparison of _FastVideoEdit_ and previous video editing methods
is shown in Fig. 3. We compare additional conditional constraints
incorporating methods Rerender [41] and Text2Video-Zero [22] as well as three
dual-branch methods FateZero [29], Pix2Video [6], and TokenFlow [11].

Figure 3: Qualitative comparison of _FastVideoEdit_ with previous video
editing methods. The top row displays the source video, while the following
rows showcase edited videos by previous editing methods and _FastVideoEdit_.
Source and target text prompt at shown the top, with the edited words
highlighted in red.

The results show that _FastVideoEdit_ effectively performs video editing
aligned with the text prompt while preserving the essential content of the
source video. Through attention control, latent replacement, and leveraging
the preservation ability of the consistency model, _FastVideoEdit_
successfully performs video foreground editing while preserving the
background. This approach enables targeted editing of the foreground elements
in the video while ensuring that the background remains intact. By selectively
focusing on specific regions of interest and employing latent replacement
techniques, _FastVideoEdit_ achieves accurate and consistent editing results,
maintaining the integrity of the background content. It is worth noting that
_FastVideoEdit_ achieves superior performance compared to other methods while
requiring significantly less time. This highlights the efficiency and
effectiveness of our approach in delivering high-quality results in a more
time-efficient manner.

###  5.4 Ablation Study

Figure 4: Illustration of ablation on model architecture.

We ablate the use of Bg-Masa, CF-Masa, Re-CA and TokenFlow propagation.
Quantitative results and qualitative results are shown in Tab. 2 and Fig. 4.
Without background preservation, the background dirt is changed. Results show
that removing CF-Masa and TokenFlow results in a worse temporal consistency.
Moreover, replacing our attention control with PnP results in a worse editing
effect (See left rabbit’s ears and right rabbit’s tail).

Tab. 2 shows that without latent replacement the temporal consistency and CLIP
accuracy metrics rise, which illustrates that latent replacement protects
background but does not help with either temporal consistency or CLIP
accuracy. The improvement in background preservation is observed evidently in
qualitative results which is not reflected on CLIP metrics. Imposing
background preservation prevents the adaption of background to the editing
prompt which is negatively reflected on CLIP based similarity evaluation.
However, visual observation by eyes can hardly capture the negative impact it
causes in terms of content editing. Apart from background preservation
designs, the rest of our proposed attention controls achieve better
performance in all the three metrics, which shows the effectiveness of our
proposed methods.

Table 2: Ablation study for architecture design of _FastVideoEdit_. Bold
indicates the best. Underline indicates the second best.

Model |  CLIP Metrics$\uparrow$
---|---
Tem-Con | Txt-Sim | Clip-Acc
Ours | 96.5 | 27.7 | 71.1
w/o Bg-Masa | 96.7 | 27.5 | 72.3
w/o CF-Masa | 96.3 | 26.7 | 69.3
w/ PnP | 96.5 | 25.8 | 60.0

##  6 Conclusion

##### Conclusion.

In this work, we have introduced _FastVideoEdit_ , a zero-shot video editing
approach that addresses the computational challenges faced by previous
methods. By leveraging the self-consistency property of Consistency Models
(CMs), our method eliminates the need for time-consuming inversion or
additional condition extraction steps. We have also introduced a novel
approach for maintaining background preservation via latent replacement, which
simultaneously denoises a background branch while imposing self-attention
control from the source and editing branches. Our experimental results
demonstrate the superior performance of _FastVideoEdit_ in terms of editing
quality while requiring significantly less time for video editing tasks.

##### Limitations and future work.

However, _FastVideoEdit_ still has some limitations: (1) _FastVideoEdit_ may
require tuning its hyperparameters to achieve optimal performance on each
video. This dependency on hyperparameter adjustment adds complexity to the
editing process and may require expertise or extensive experimentation to
achieve satisfactory results. (2) While _FastVideoEdit_ demonstrates state-of-
the-art performance in video editing, there is no guarantee of success for
every editing case. The effectiveness of the approach may vary depending on
factors such as input data quality, the complexity of the editing task, and
the suitability of chosen hyperparameters. (3) The performance of
_FastVideoEdit_ relies on the quality and capabilities of the underlying
consistency models. In our future work, we are committed to making further
improvements to address the challenges highlighted in our research.

##### Possible negative social impact.

Video editing approaches may pose privacy risks if used to alter videos
without appropriate consent or to create content that invades someone’s
privacy. Moreover, the convenience and speed offered by _FastVideoEdit_ may
inadvertently encourage irresponsible editing practices, leading to ethical
dilemmas in areas such as journalism, entertainment, and personal
communication. Addressing these possible negative social impacts and
continuously improving our models are key focuses for our future releases.

## References

  * [1] Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al.: ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 (2022)
  * [2] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. In: European conference on computer vision. pp. 707–723. Springer (2022)
  * [3] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22563–22575 (2023)
  * [4] Brooks, T., Peebles, B., Homes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., Ramesh, A.: Video generation models as world simulators (2024), https://openai.com/research/video-generation-models-as-world-simulators
  * [5] Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., Zheng, Y.: Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465 (2023)
  * [6] Ceylan, D., Huang, C.H.P., Mitra, N.J.: Pix2video: Video editing using image diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23206–23217 (2023)
  * [7] Chai, W., Guo, X., Wang, G., Lu, Y.: Stablevideo: Text-driven consistency-aware diffusion video editing. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 23040–23050 (2023)
  * [8] Chen, W., Wu, J., Xie, P., Wu, H., Li, J., Xia, X., Xiao, X., Lin, L.: Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840 (2023)
  * [9] Cong, Y., Xu, M., Simon, C., Chen, S., Ren, J., Xie, Y., Perez-Rua, J.M., Rosenhahn, B., Xiang, T., He, S.: Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint arXiv:2310.05922 (2023)
  * [10] Esser, P., Chiu, J., Atighehchian, P., Granskog, J., Germanidis, A.: Structure and content-guided video synthesis with diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7346–7356 (2023)
  * [11] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373 (2023)
  * [12] Gupta, A., Yu, L., Sohn, K., Gu, X., Hahn, M., Fei-Fei, L., Essa, I., Jiang, L., Lezama, J.: Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 (2023)
  * [13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022)
  * [14] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)
  * [15] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33, 6840–6851 (2020)
  * [16] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [17] Ju, X., Zeng, A., Bian, Y., Liu, S., Xu, Q.: Direct inversion: Boosting diffusion-based editing with 3 lines of code. arXiv preprint arXiv:2310.01506 (2023)
  * [18] Ju, X., Zeng, A., Wang, J., Xu, Q., Zhang, L.: Human-art: A versatile human-centric dataset bridging natural and artificial scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 618–629 (2023)
  * [19] Ju, X., Zeng, A., Zhao, C., Wang, J., Zhang, L., Xu, Q.: Humansd: A native skeleton-guided diffusion model for human image generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15988–15998 (2023)
  * [20] Kahatapitiya, K., Karjauv, A., Abati, D., Porikli, F., Asano, Y.M., Habibian, A.: Object-centric diffusion for efficient video editing. arXiv preprint arXiv:2401.05735 (2024)
  * [21] Kasten, Y., Ofri, D., Wang, O., Dekel, T.: Layered neural atlases for consistent video editing. ACM Transactions on Graphics (TOG) 40(6), 1–12 (2021)
  * [22] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 (2023)
  * [23] Liu, S., Zhang, Y., Li, W., Lin, Z., Jia, J.: Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761 (2023)
  * [24] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., Zhu, J.: Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems 35, 5775–5787 (2022)
  * [25] Luhman, E., Luhman, T.: Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 (2021)
  * [26] Ma, Y., Cun, X., He, Y., Qi, C., Wang, X., Shan, Y., Li, X., Chen, Q.: Magicstick: Controllable video editing via control handle transformations. arXiv preprint arXiv:2312.03047 (2023)
  * [27] Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., Salimans, T.: On distillation of guided diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14297–14306 (2023)
  * [28] Molad, E., Horwitz, E., Valevski, D., Acha, A.R., Matias, Y., Pritch, Y., Leviathan, Y., Hoshen, Y.: Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329 (2023)
  * [29] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535 (2023)
  * [30] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (ICML). pp. 8748–8763. PMLR (2021)
  * [31] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [32] Salimans, T., Ho, J.: Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 (2022)
  * [33] Shin, C., Kim, H., Lee, C.H., Lee, S.g., Yoon, S.: Edit-a-video: Single video editing with object-aware consistency. arXiv preprint arXiv:2303.07945 (2023)
  * [34] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022)
  * [35] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
  * [36] Song, Y., Dhariwal, P., Chen, M., Sutskever, I.: Consistency models. In: Proceedings of the 40th International Conference on Machine Learning (2023)
  * [37] Wang, W., Jiang, Y., Xie, K., Liu, Z., Chen, H., Cao, Y., Wang, X., Shen, C.: Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599 (2023)
  * [38] Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7623–7633 (2023)
  * [39] Wu, J.Z., Li, X., Gao, D., Dong, Z., Bai, J., Singh, A., Xiang, X., Li, Y., Huang, Z., Sun, Y., et al.: Cvpr 2023 text guided video editing competition. arXiv preprint arXiv:2310.16003 (2023)
  * [40] Xu, S., Huang, Y., Pan, J., Ma, Z., Chai, J.: Inversion-free image editing with natural language (2024)
  * [41] Yang, S., Zhou, Y., Liu, Z., Loy, C.C.: Rerender a video: Zero-shot text-guided video-to-video translation. arXiv preprint arXiv:2306.07954 (2023)
  * [42] Zhang, Q., Chen, Y.: Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 (2022)
  * [43] Zhang, Y., Wei, Y., Jiang, D., Zhang, X., Zuo, W., Tian, Q.: Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077 (2023)
  * [44] Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K., Anandkumar, A.: Fast sampling of diffusion models via operator learning. In: International Conference on Machine Learning. pp. 42390–42402. PMLR (2023)