$\lvert Dv\rvert\geq\lvert D_{s}v\rvert\geq\lambda/4\quad\mbox{holds a.e.\,in
$B/2$}\,.$ (5.74)

In this case

$\textnormal{{osc}}_{\tau B}\,Dv\leq
c_{1}\tau^{\beta_{1}}\lambda\qquad\mbox{holds for every $\tau\in(0,1)$,}$
(5.75)

where $c_{1}\geq 1$ and $\beta_{1}\in(0,1)$ both depend on
$n,\mu,q,\nu,L,\mathfrak{m},g(\mathfrak{m})$, but are otherwise independent of
$\bar{a}$. The constants $c_{1}$ and $\beta_{1}$ are non-decreasing and non-
increasing functions of $\mathfrak{m}$, respectively. On the other hand there
exists
$\tilde{\eta}\equiv\tilde{\eta}(n,\mu,q,\nu,L,\mathfrak{m},g(\mathfrak{m}))\in(1/2,1)$
such that, if $\textnormal{{ex}}(B,\lambda,s)$ fails for every
$s\in\\{1,\ldots,n\\}$, then

$\|Dv\|\leq\tilde{\eta}\lambda\quad\mbox{holds a.e.\,in $B/2$}\,.$ (5.76)

The constant $\tilde{\eta}$ is a non-decreasing function of $\mathfrak{m}$.

###### Proof

For the proof of the first assertion we use and extend some of the arguments
in (KuM, , Proposition 3.7); see also dibe  for the case of the classical
$p$-Laplacian operator. We can of course confine ourselves to the case the
first inequality in (5.73) occurs (for a $\sigma$ to be determined in the
course of the proof), the other being similar. By scaling, as in (5.14), we
can assume that $B\equiv\mathcal{B}_{1}$. Since conditions (3.10) are
satisfied by $H_{\textnormal{i}}(\cdot)$, it is standard to prove that $v\in
W^{2,2}_{{\rm loc}}(\mathcal{B}_{1})$. We can then differentiate the equation
$\,{\rm div}\,\partial_{z}H_{\textnormal{i}}(Dv)=0$ in the $x_{s}$ direction,
thereby obtaining $\,{\rm
div}\,(\partial_{zz}H_{\textnormal{i}}(Dv)DD_{s}v)=0$, that is an analogue of
(4.8). Recalling that $v\in W^{1,\infty}(\mathcal{B}_{1})\cap W^{2,2}_{{\rm
loc}}(\mathcal{B}_{1})$ (as in (4.4) and by (5.72)), we then use as test
function $(D_{s}v-\kappa)_{-}\phi^{2}$, where $\phi\in
C^{\infty}_{0}(\mathcal{B}_{1})$ is non-negative and $0\leq\kappa\leq\lambda$;
this is still possible by (3.10)). Integrating by parts the resulting equation
we find

$\displaystyle\int_{\mathcal{B}_{1}}\langle
D_{s}(\partial_{z}H_{\textnormal{i}}(Dv)),D(D_{s}v-\kappa)_{-}\rangle\phi^{2}\,dx$
$\displaystyle\qquad=\int_{\mathcal{B}_{1}}\langle\partial_{z}H_{\textnormal{i}}(Dv),D_{s}D\phi^{2}\rangle(D_{s}v-\kappa)_{-}\,dx$
$\displaystyle\qquad\quad-2\int_{\mathcal{B}_{1}\cap\\{D_{s}v<\kappa\\}}\langle\partial_{z}H_{\textnormal{i}}(Dv),D\phi\rangle
D_{ss}v\phi\,dx\,.$ (5.77)

We define the functions

$\begin{cases}G_{0}(t):=\tilde{g}(t)+\bar{a}(t^{2}+\omega^{2})^{(q-1)/2}\\\
G(t):=[\tilde{g}(t)]^{2}(t^{2}+1)^{\mu/2}+\bar{a}(t^{2}+\omega^{2})^{q/2}\\\
\lambda_{\bar{a}}(t)=(t^{2}+1)^{-\mu/2}+(q-1)\bar{a}(t^{2}+\omega^{2})^{(q-2)/2}\\\
\end{cases}$ (5.78)

for $t\geq 0$, so that

$G_{0}(\lambda)\lambda+\lambda_{\bar{a}}(\lambda)\lambda^{2}+[G_{0}(\lambda)]^{2}/\lambda_{\bar{a}}(\lambda)\leq
cG(\lambda)$ (5.79)

holds for $c\equiv c(q)$. By also using (3.8), (5.69) and (5.72), and the
monotonicity features of $\lambda_{\bar{a}}(\cdot)$ and $G_{0}(\cdot)$ in
(5.77), we easily find

$\displaystyle\lambda_{\bar{a}}(\lambda)\int_{\mathcal{B}_{1}}|D(D_{s}v-\kappa)_{-}|^{2}\phi^{2}\,dx$
$\displaystyle\qquad\leq
cG_{0}(\lambda)\int_{\mathcal{B}_{1}}(\phi|D^{2}\phi|+|D\phi|^{2})(D_{s}v-\kappa)_{-}\,dx$
$\displaystyle\qquad\quad+cG_{0}(\lambda)\int_{\mathcal{B}_{1}}\phi|D\phi||D(D_{s}v-\kappa)_{-}|\,dx\,.$

By then using Young’s inequality and (5.79), and yet recalling that
$(D_{s}v-\kappa)_{-}\leq 2\lambda$, we find

$\displaystyle\lambda_{\bar{a}}(\lambda)\|D(\phi(D_{s}v-\kappa)_{-})\|_{L^{2}(\mathcal{B}_{1})}^{2}$
$\displaystyle\qquad\leq
cG(\lambda)|\mathcal{B}_{1}\cap\\{D_{s}v<\kappa\\}\cap\\{\phi>0\\}|(\|D\phi\|_{L^{\infty}}^{2}+\|D^{2}\phi\|_{L^{\infty}})\,.$
(5.80)

For integers $m\geq 0$ we choose levels
$\\{\kappa_{m}=\lambda(1+1/2^{m})/4\\}$, radii
$\\{\varrho_{m}=1/2+1/2^{m+1}\\}$ and cut-off functions $\phi_{m}\in
C^{\infty}_{0}(\mathcal{B}_{\varrho_{m}})$, with $\phi_{m}\equiv 1$ on
$\mathcal{B}_{\varrho_{m+1}}$,
$\|D\phi_{m}\|_{L^{\infty}}^{2}+\|D^{2}\phi_{m}\|_{L^{\infty}}\lesssim 4^{m}$.
With

$A_{m}:=\\{D_{s}v<\kappa_{m}\\}\cap B_{\varrho_{m}}$ (5.81)

estimate (5.80) becomes

$\lambda_{\bar{a}}(\lambda)\|D(\phi_{m}(D_{s}v-\kappa_{m})_{-})\|_{L^{2}(\mathcal{B}_{1})}^{2}\leq
c4^{m}G(\lambda)|A_{m}|\,.$

We then find, via Sobolev embedding

$\displaystyle\lambda_{\bar{a}}(\lambda)(\kappa_{m}-\kappa_{m+1})^{2}|A_{m+1}|$
$\displaystyle\leq\lambda_{\bar{a}}(\lambda)\|\phi_{m}(D_{s}v-\kappa_{m})_{-}\|_{L^{2}(\mathcal{B}_{1})}^{2}$
$\displaystyle\leq
c\lambda_{\bar{a}}(\lambda)\|D(\phi_{m}(D_{s}v-\kappa_{m})_{-})\|_{L^{2}(\mathcal{B}_{1})}^{2}|A_{m}|^{1/n}$
$\displaystyle\leq c4^{m}G(\lambda)|A_{m}|^{1+1/n}\,.$

Observing that $\kappa_{m}-\kappa_{m+1}\approx 2^{-m}\lambda$ and that (5.72)
implies $\lambda^{2}\approx\lambda^{2}+\omega^{2}$, the above display gives

$\displaystyle|A_{m+1}|$ $\displaystyle\leq$
$\displaystyle\frac{c2^{4m}G(\lambda)}{\lambda_{\bar{a}}(\lambda)\lambda^{2}}|A_{m}|^{1+1/n}$
(5.82)
$\displaystyle\stackrel{{\scriptstyle(\ref{ragnar}),(\ref{ragnar2})}}{{\leq}}$
$\displaystyle
c\left[(\lambda^{2}+1)^{\mu-1}[g(\lambda)]^{2}+1\right]2^{4m}|A_{m}|^{1+1/n}$
$\displaystyle\stackrel{{\scriptstyle(\ref{limitatio})}}{{\leq}}$
$\displaystyle
c\left[\mathfrak{m}^{2(\mu-1)}[g(\mathfrak{m})]^{2}+1\right]2^{4m}|A_{m}|^{1+1/n}$

with $c\equiv c(n,\mu,q,\nu,L,c_{g}(1))$ which is independent of $\bar{a}$.
Inequality (5.82) allows to perform a standard geometric iteration (see for
instance (giu, , Lemma 7.1)) leading to $D_{s}v\geq\lambda/4$ once the first
inequality in (5.73) is verified with
$\sigma\equiv\sigma(n,\mu,q,\nu,L,c_{g}(1),\mathfrak{m},g(\mathfrak{m}))$
small enough (that corresponds to require that $|A_{0}|$ is small, see
(5.81)). This allows to prove (5.75) by considering

$-\,{\rm
div}\,(\texttt{a}(x)DD_{t}v)=0\,,\quad\texttt{a}(x):=[\lambda_{\bar{a}}(\lambda)]^{-1}\partial_{zz}H_{\textnormal{i}}(Dv(x))$
(5.83)

this time for every $t\in\\{1,\ldots,n\\}$. Recalling (3.8)-(3.9) it follows
that $\texttt{a}(\cdot)$ satisfies

$|\xi|^{2}\stackrel{{\scriptstyle(\ref{limitatio})}}{{\leq}}c\langle\texttt{a}(x)\xi,\xi\rangle\,,\quad\lvert\texttt{a}(x)\rvert\stackrel{{\scriptstyle(\ref{dasotto})}}{{\leq}}c[\mathfrak{m}^{\mu-1}g(\mathfrak{m})+1]$
(5.84)

a.e. in $x\in\mathcal{B}_{1/2}$ and for every $\xi\in\mathbb{R}^{n}$, where
$c\equiv c(n,\mu,q,\nu,L)$. Therefore a standard application of De Giorgi-
Nash-Moser theory to $D_{t}v$ and (5.72) imply the validity of (5.75) (note
that for this we obtain (5.75) for $\tau\leq 1/8$, and then the case
$1/8<\tau\leq 1$ follows trivially by (5.72)). We now turn to (5.76); the
proof is a variant of the one in (KuM, , Proposition 3.11) apart from the
parabolic case considered there. The only remark we need to do is that the
analogue of equation in (KuM, , (3.53)) is here given by $\,{\rm
div}\,(\texttt{b}(x)Dz)=0$, where
$\texttt{b}(x):=[\lambda_{\bar{a}}(\lambda)]^{-1}\partial_{zz}H_{\textnormal{i}}(Dv(x))=\texttt{a}(x)$
if $x$ belongs to the support of $\tilde{v}:=(D_{s}v-\lambda/2)_{+}$, and
$\texttt{b}(x)\equiv\mathds{I}_{d}$ (the identity matrix) otherwise. Then
$\tilde{v}$ turns out to be a weak subsolution. Moreover, as
$|Dv|\geq\lambda/2$ on the support of $\tilde{v}$, it follows that
$\texttt{b}(x)$ also satisfies (5.84). We can now use the arguments in (KuM, ,
pp. 784-786) to conclude with (5.76). Finally, as for the monotone dependence
on $\mathfrak{m}$ of the constants $c_{1},\beta_{1},\tilde{\eta},\sigma$, this
is classical, and it is a consequence of the fact that all the estimates above
feature constants, usually denoted by $c$, that are non-decreasing functions
of $\mathfrak{m}$ and $g(\mathfrak{m})$; in turn, $g(\cdot)$ is a non-
decreasing function too.

###### Lemma 12

Let $v$ be as in Lemma 11, assume (5.72) and that $m\omega\geq\lambda$ holds
for some integer $m\geq 1$. Then

$\textnormal{{osc}}_{\tau B}\,Dv\leq
c_{2}\tau^{\beta_{2}}\lambda\qquad\mbox{holds for every $\tau\in(0,1)$,}$
(5.85)

where $c_{2}\geq 1$ and $\beta_{2}\in(0,1)$ both depend on
$n,\mu,q,\nu,L,\mathfrak{m},g(\mathfrak{m}),m$, but are otherwise independent
of $\bar{a}$. The constants $c_{2}$ and $\beta_{2}$ are non-decreasing and
non-increasing functions of $\mathfrak{m}$ and $m$, respectively.

###### Proof

As for Lemma 11 we reduce to the case $B\equiv\mathcal{B}_{1}$ and consider
(5.83). The first inequality in (5.84) follows as $\|Dv\|\leq\lambda$ a.e.;
using $m\omega\geq\lambda$ and (3.8)3 we instead have

$\lvert\texttt{a}(x)\rvert\leq
c[\mathfrak{m}^{\mu}g(\mathfrak{m})+m^{2-q}]\quad\mbox{a.e.\,in
$x\in\mathcal{B}_{1}$}\,,$

where $c\equiv c(n,\mu,q,\nu,L)$. Now (5.85) follows from standard De Giorgi-
Nash-Moser theory.

######  Proof (of Proposition 3)

We take
$\sigma,\tilde{\eta}\equiv\sigma,\tilde{\eta}(n,\mu,q,\nu,L,\mathfrak{m},g(\mathfrak{m}))\in(0,1)$
from Lemma 11 and determine an integer $m\equiv
m(n,\mu,q,\nu,L,\mathfrak{m},g(\mathfrak{m}))\geq 1$ such that
$\eta:=\tilde{\eta}+1/m<1$; notice that $m$ can be determined as a non-
decreasing function of $\mathfrak{m}$. All in all, once $\mathfrak{m}$ is
fixed so is $m$ and therefore $\eta$. We set
$\lambda_{0}:=\sup_{B}\,\|Dv\|+\omega$, $\lambda_{k}:=\eta^{k}\lambda_{0}$,
$B_{k}:=B/2^{k}$ for every $k\in\mathbb{N}$, and define $\bar{k}\in\mathbb{N}$
as the smallest non-negative integer for which $m\omega>\lambda_{\bar{k}}$.
This implies that

$\omega\leq\frac{\lambda_{k}}{m},\quad\mbox{for \ $0\leq k<\bar{k}$}\,.$
(5.86)

Note that (5.86) is empty when $\bar{k}=0$. We further define

$A:=\\{k\in\mathbb{N}\,\colon 0\leq k<\bar{k}\ \mbox{and}\
\mbox{$\textnormal{{ex}}(B_{k},\lambda_{k},s)$ holds for some
$s\in\\{1,\ldots,n\\}$}\\}$

and $\tilde{k}$ as $\tilde{k}:=\min A$ if $A\not=\emptyset$, and
$\tilde{k}=\bar{k}$ if $A=\emptyset$. Using (5.76) and (5.86) iteratively, we
find

$\sup_{B_{k}}\|Dv\|+\omega\leq\lambda_{k}=\eta^{k}\lambda_{0}\quad\mbox{for
every $0\leq k\leq\tilde{k}$}\,.$ (5.87)

On the other hand, by the very definition of $\tilde{k}$ and Lemmas 11-12 (the
latter is only needed when $\tilde{k}=\bar{k}$ and therefore when
$A=\emptyset$), we find that

$\textnormal{{osc}}_{\tau
B_{\tilde{k}}}\,Dv\leq\max\\{c_{1},c_{2}\\}\tau^{\min\\{\beta_{1},\beta_{2}\\}}\lambda_{\tilde{k}}\qquad\mbox{holds
for \ $0<\tau<1$}\,.$ (5.88)

At this stage (5.71) follows via a standard interpolation argument combining
(5.87)-(5.88).

###### Remark 4

As a consequence of the local gradient boundedness obtained in Section 5.8,
the arguments developed for Proposition 3 do not require any upper bound on
$\mu$. Moreover, the specific structure in (5.67) is not indispensable and the
methods used here can be used as a starting point to treat general
nonautonomous functionals with $(p,q)$-growth of the type considered in M2 .

###  5.11 Improved Hölder exponent when $\textnormal{{s}}>0$

The $C^{1,\tilde{\alpha}/2}_{{\rm loc}}(\Omega)$-regularity proof follows
almost verbatim the one in (piovra, , Section 10) since now we already know
that $Du$ is locally Hölder continuous in $\Omega$ with exponent $\beta$. We
confine ourselves to give a few remarks on the main modifications and to
facilitate the adaptation we use the same notation introduced in piovra . We
set $M:=\|Du\|_{L^{\infty}(\Omega)}+[Du]_{0,\beta;\Omega}+1$ (we can assume
this is finite since our result is local). This time we take
$A_{r}(z):=\partial_{z}H_{r^{\tilde{\alpha}},\textnormal{i}}(z)=\partial_{z}F(x_{\rm
c},z)+qa_{r^{\tilde{\alpha}},\textnormal{i}}(B_{r})[\ell_{\textnormal{{s}}}(z)]^{q-2}z$
and recall that
$|\partial_{z}H_{r^{\tilde{\alpha}},\textnormal{i}}(z)-\partial_{z}H(x,z)|\leq
cr^{\tilde{\alpha}}[\ell_{1}(z)]^{q-1}$ holds whenever $x\in B_{r}$. With
$v\in u+W^{1,q}_{0}(B_{r})$ to solve $\,{\rm div}\,A_{r}(Dv)=0$ in $B_{r}$, as
in (5.61) we gain $\lVert Dv\rVert_{L^{\infty}(B_{r/2})}\leq cM^{2}$ with
$c\equiv c(\textnormal{{data}})$. With these informations estimate $\lVert Du-
Dv\rVert_{L^{2}(B_{r/2})}^{2}\leq cr^{n+\tilde{\alpha}}$ follows as in (5.65)
(in turn as in (9)), with $c\equiv c(\textnormal{{data}},M)$. This is the
analogue of (piovra, , (10.3)), but for the fact that here we find
$\tilde{\alpha}$ rather than $2\tilde{\alpha}$ and the integral is supported
in $B_{r/2}$ instead of $B_{r}$. Next, by defining the matrix
$[\mathbb{A}(x)]_{ij}:=\partial_{z_{j}}A_{r}^{i}(Dv(x))$, to use piovra  we
need to prove that there exists
$\lambda\equiv\lambda(\textnormal{{data}},\textnormal{{s}},M)>0$, independent
of $r$, such that $\lambda\mathds{I}_{\rm
d}\leq\mathbb{A}(x)\leq(1/\lambda)\mathds{I}_{\rm d}$ holds for a.e. $x\in
B_{r/2}$. As a consequence of (1.11)2 and of $|Dv|\lesssim M^{2}$ in
$B_{r/2}$, we have

$\frac{|\xi|^{2}}{(M^{4}+1)^{\mu/2}}\leq
c\langle\mathbb{A}(x)\xi,\xi\rangle\,,\qquad|\mathbb{A}(x)|\leq
c[g(M^{2})+\textnormal{{s}}^{q-2}]$

for a.e. $x\in B_{r/2}$ and $z,\xi\in\mathbb{R}^{n}$, where
$c\equiv(\textnormal{{data}},M)$. This is the crucial point where we use After
this, the proof follows exactly as (piovra, , (10.6)) and the proof of the
entire Theorem 1.2 is finally complete.

##  6 Theorem 1.1, Corollary 1 and model examples (1.10)

For every integer $k\geq 0$, we consider the integrand

$F(x,z):=\mathfrak{c}(x)|z|L_{k+1}(|z|)+1,$

with $\mathfrak{c}(\cdot)$ as in (1.7)2 and $L_{k+1}(\cdot)$ as in (1.10)2.
Direct computations show that $F(\cdot)$ satisfies (1.11) with $g(t)\equiv
L_{k+1}(t)+1$, with $\mu=1$ when $k=0$, and with any choice of $\mu>1$ when
$k>0$; moreover, it also satisfies (1.13). The constants $\nu$ and $L$ depend
on $\mu,k$ and $\Lambda$. Theorem 1.2 therefore applies to local minimizers of

$w\mapsto\int_{\Omega}[\mathfrak{c}(x)|Dw|L_{k+1}(|Dw|)+a(x)(Dw|^{2}+\textnormal{{s}}^{2})^{q/2}+1]\,dx\,.$

On the other hand the functional in the above display and

$w\mapsto\int_{\Omega}[\mathfrak{c}(x)|Dw|L_{k+1}(|Dw|)+a(x)(Dw|^{2}+\textnormal{{s}}^{2})^{q/2}]\,dx$

share the same local minimizers, and therefore the regularity results stated
in Theorem 1.2 hold for local minimizers of the last functional too. In this
way the model case in (1.10) is covered. Theorem 1.1 and Corollary 1 then
follow as special cases taking $k=0$ and $\textnormal{{s}}=0$ and
$\textnormal{{s}}=1$, respectively. Note that, in the spirit of Corollary 1,
local minimizers of the functional

$w\mapsto\int_{\Omega}\mathfrak{c}(x)|Dw|L_{k+1}(|Dw|)\,dx$

are locally $C^{1,\alpha_{0}/2}$-regular in $\Omega$, for every $k\geq 0$,
provided (1.7)2 is assumed.

##  7 Vectorial cases and further generalizations

When functionals as in (1.1) and (1.9) are considered in the vectorial case,
i.e. minima are vector valued $u\colon\Omega\to\mathbb{R}^{N}$ and $N>1$, we
can still obtain Lipschitz continuity results. In this situation it is
unavoidable to impose a so-called Uhlenbeck structure uh , that is

$F(x,z)=\tilde{F}(x,|z|)\,,$ (7.1)

where $\tilde{F}\colon\Omega\times[0,\infty)\to[0,\infty)$ is a continuous
function such that $t\mapsto\tilde{F}(x,t)$ is $C^{2}$-regular for every
choice of $x\in\Omega$. This is obviously satisfied by the models in (1.1) and
(1.10).

###### Theorem 7.1

Let $u\in W^{1,1}_{{\rm loc}}(\Omega;\mathbb{R}^{N})$ be a local minimizer of
the functional in (1.9) under assumptions (1.7)1, (1.11)-(1.12) and (7.1).
There exists $\mu_{\textnormal{max}}\in(1,2)$, depending only
$n,q,\alpha,\alpha_{0}$, such that, if $1\leq\mu<\mu_{\textnormal{max}}$, then
(1.14) holds as in Theorem 1.2.

The proof of Theorem 7.1 is essentially the same of the one given for Theorem
1.2, once the content of Proposition 1 is available. This is in fact the only
point where (7.1) enters the game. Inspecting the proof of Proposition 1 shows
that this works in the vectorial case provided (4.9) holds. In turn, this
inequality follows along the lines of the estimates in (BM, , Lemma 5.6).
Notice that, without an additional structure assumption as in (7.1), Theorem
7.1 cannot hold and counterexamples to Lipschitz regularity emerge already
when considering uniformly elliptic systems SY . We also remark that we expect
gradient Hölder continuity in the vectorial case as well; the proof must be
different from the one given in Section 5.10, and based on linearization
methods as those originally introduced in uh  and developed in large parts of
the subsequent literature. Another direct generalization, this time in the
scalar case, occurs for functionals of the type

$w\mapsto\mathcal{N}(w,\Omega):=\int_{\Omega}\left[F(x,Dw)+a(x)G(Dw)\right]\,dx\,,$
(7.2)

where $F(\cdot)$ is as Theorem 1.2 and $G\colon\mathbb{R}^{n}\to[0,\infty)$
belongs to $C^{1}(\mathbb{R}^{n})\cap C^{2}(\mathbb{R}^{n}\setminus\\{0\\})$
and satisfies

$\begin{cases}\ \nu(|z|^{2}+\textnormal{{s}}^{2})^{q/2}\leq G(z)\leq
L(|z|^{2}+\textnormal{{s}}^{2})^{q/2}\\\ \
\displaystyle\nu(|z|^{2}+\textnormal{{s}}^{2})^{(q-2)/2}|\xi|^{2}\leq\langle\partial_{zz}G(z)\xi,\xi\rangle\\\
\ |\partial_{zz}G(z)|\leq
L(|z|^{2}+\textnormal{{s}}^{2})^{(q-2)/2}\end{cases}$

for every $z,\xi\in\mathbb{R}^{n}$, $|z|\not=0$. Theorem 1.2 continuous to
hold in this last case, with essentially the same proof. The only difference
worth pointing out is the new shape of the integrands
$H(x,z):=F(x,z)+a(x)G(z)$ in (3.1) and $H_{\omega,\sigma}(\cdot)$ (and the
related minimal one $H_{\omega,\sigma,\textnormal{i}}(\cdot)$) in (3.2),
accordingly defined as

$\begin{cases}\,H_{\omega,\sigma}(x,z):=F(x,z)+a_{\sigma}(x)G_{\omega}(z)\\\
\,\displaystyle
G_{\omega}(z):=\int_{\mathcal{B}_{1}}G(z+\omega\lambda)\phi(\lambda)\,{\rm
d}\lambda\,.\end{cases}$

Here $\\{\phi_{\varepsilon}\\}\subset C^{\infty}(\mathbb{R}^{n})$, denotes a
family of radially symmetric mollifiers, defined as
$\phi_{\varepsilon}(x):=\phi(x/\varepsilon)/\varepsilon^{n}$, where $\phi\in
C^{\infty}_{c}(\mathcal{B}_{1})$,
$\lVert\phi\rVert_{L^{1}(\mathbb{R}^{n})}=1$, $\mathcal{B}_{3/4}\subset\,{\rm
supp}\,\phi$. The definition of
$\mathbb{H}_{\omega,\sigma,\textnormal{i}}(\cdot)$, instead, remains the same.
Following for instance (dm, , Section 4.5) and (piovra, , Section 6.7) it is
possible to show that the newly defined integrands $H_{\omega,\sigma}(\cdot)$
still have the properties described in Section 3. In particular (3.5) and
(3.6) still hold for a suitable constant $c$ with the same dependence upon the
various parameters described there. Finally, when again considering the
vectorial case for functionals as in (7.2) we can still obtain local Lipschitz
regularity of minima provided (7.1) holds together with
$G(z)\equiv\tilde{G}(|z|)$ and $\tilde{G}(\cdot)$ is $C^{2}$-regular outside
the origin (see for instance BM ; ciccio  for precise assumptions).

Acknowledgments. The first author is supported by INdAM-GNAMPA via the project
“Problemi non locali: teoria cinetica e non uniforme ellitticità", and by the
University of Parma via the project “Local vs nonlocal: mixed type operators
and nonuniform ellipticity". Both the authors are grateful to the referees for
the careful reading of the original version of the manuscript and for the many
suggestions and comments that eventually led to a better presentation.

Funding Open access funding provided by Università degli Studi di Parma within
the CRUI-CARE Agreement.

Data Availibility No data is attached to this paper.

Declarations

Conflict of interests and data. The authors declare to have no conflict of
interests. No data are attached to this paper.

Open Access This article is licensed under a Creative Commons Attribution 4.0
International License, which permits use, sharing, adaptation, distribution
and reproduction in any medium or format, as long as you give appropriate
credit to the original author(s) and the source, provide a link to the
Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s
Creative Commons licence, unless indicated otherwise in a credit line to the
material. If material is not included in the article’s Creative Commons
licence and your intended use is not permitted by statutory regulation or
exceeds the permitted use, you will need to obtain permission directly from
the copyright holder. To view a copy of this licence, visit
creativecommons.org/ licenses/by/4.0/.

## References

  * (1) E. Acerbi, G. Bouchitté, I. Fonseca, Relaxation of convex functionals: the gap problem, _Ann. Inst. Henri Poincaré, Anal. Non Linéaire_ 20, 359–390 (2003).
  * (2) D.R. Adams, L.I. Hedberg, Function spaces and potential theory, Grundlehren der mathematischen Wissenschaften 314. Springer-Verlag, Berlin, 1996.
  * (3) A. Balci, L. Diening, M. Surnachev, New examples on Lavrentiev gap using fractals, _Calc. Var. & PDE_ 59:180 (2020).
  * (4) A. Balci, C. Ortner, J. Storn, Crouzeix-Raviart finite element method for non-autonomous variational problems with Lavrentiev gap, _Numer. Math._ 151, 779–805 (2022).
  * (5) P. Baroni, M. Colombo, G. Mingione, Regularity for general functionals with double phase, _Calc. Var. & PDE_ 57:62 (2018).
  * (6) S. Baasandorj, S.S. Byun, Regularity for Orlicz phase problems, _Memoirs Amer. Math. Soc._ , to appear.
  * (7) L. Beck, M. Bulíček, F. Gmeineder, On a Neumann problem for variational functionals of linear growth, _Ann. Sc. Norm. Super. Pisa Cl. Sci. (V)_ 21, 695–737 (2020).
  * (8) L. Beck, G. Mingione, Lipschitz bounds and nonuniform ellipticity, _Comm. Pure Appl. Math._ 73, 944-1034 (2020).
  * (9) P. Bella, M. Schäffner, Local boundedness and Harnack inequality for solutions of linear nonuniformly elliptic equations, _Comm. Pure Appl. Math._ 74, 453-477 (2021).
  * (10) P. Bella, M. Schäffner, On the regularity of minimizers for scalar integral functionals with $(p,q)$-growth, _Anal. & PDE_ 13, 2241–2257 (2020).
  * (11) P. Bella, M. Schäffner, Lipschitz bounds for integral functionals with $(p,q)$-growth conditions, _Adv. Calc. Var._ 10.1515/acv-2022-0016
  * (12) M. Bildhauer, _Convex variational problems. Linear, nearly linear and anisotropic growth conditions._ Lecture Notes in Mathematics, 1818. Springer-Verlag, Berlin, 2003. x+217.
  * (13) M. Bildhauer, M. Fuchs, Partial regularity for variational integrals with $(s,\mu,q)$-growth, _Calc. Var. & PDE_ 13, 537–560 (2001).
  * (14) M. Bildhauer, M. Fuchs, $C^{1,\alpha}$-solutions to non-autonomous anisotropic variational problems, _Calc. Var. & PDE_ 24, 309–340 (2005).
  * (15) G. Bouchitté, I. Fonseca, J. Malý, The effective bulk energy of the relaxed energy of multiple integrals below the growth exponent, _Proc. R. Soc. Edinb., Sect. A, Math._ 128, 463-479 (1998).
  * (16) M. Bulíček, P. Gwiazda, J. Skrzeczkowski, On a range of exponents for absence of Lavrentiev phenomenon for double phase functionals, _Arch. Ration. Mech. Anal._ 246, 209–240 (2022).
  * (17) S.S. Byun, J. Oh, Regularity results for generalized double phase functionals, _Anal. PDE_ 13, 1269–1300 (2020).
  * (18) M. Colombo, G. Mingione, Regularity for double phase variational problems, _Arch. Ration. Mech. Anal._ 215, 443–496 (2015).
  * (19) C. De Filippis, Quasiconvexity and partial regularity via nonlinear potentials, _J. Math. Pures Appl._ 163, 11–82 (2022).
  * (20) C. De Filippis, G. Mingione, A borderline case of Calderón-Zygmund estimates for nonuniformly elliptic problems, _St. Petersburg Math. J._ 31, 455–477 (2020).
  * (21) C. De Filippis, G. Mingione, Manifold constrained nonuniformly elliptic problems, _J. Geom. Anal._ 30:1661-1723 (2020).
  * (22) C. De Filippis, G. Mingione, On the regularity of minima of non-autonomous functionals, _J. Geom. Anal._ 30:1584-1626 (2020).
  * (23) C. De Filippis, G. Mingione, Lipschitz bounds and nonautonomous integrals, _Arch. Ration. Mech. Anal._ 242, 973-1057 (2021).
  * (24) C. De Filippis, G. Mingione, Nonuniformly elliptic Schauder theory, arXiv:2201.07369v1.
  * (25) C. De Filippis, B. Stroffolini, Singular multiple integrals and nonlinear potentials, J. Funct. Anal. 285, paper 109952 (2023).
  * (26) E. DiBenedetto, $C^{1+\alpha}$-local regularity of weak solutions of degenerate elliptic equations, _Nonlinear Anal._ 7, 827-850 (1983).
  * (27) E. Di Nezza, G. Palatucci, E. Valdinoci, Hitchhiker’s guide to the fractional Sobolev spaces, _Bull. Sci. Math._ 136, 521-573 (2012).
  * (28) L. Esposito, F. Leonetti, G. Mingione, Sharp regularity for functionals with $(p,q)$ growth, _J. Diff. Equ._ 204, 5-55 (2004).
  * (29) I. Fonseca, J. Malý, Relaxation of multiple integrals below the growth exponent, Ann. Inst. H. Poincaré Anal. Non Linéaire 14, 309-338 (1997)
  * (30) I. Fonseca, J. Malý, G. Mingione, Scalar minimizers with fractal singular sets, _Arch. Ration. Mech. Anal._ 172, 295-307 (2004).
  * (31) I. Fonseca, P. Marcellini, Relaxation of multiple integral in subcritical Sobolev spaces, J. Geom. Anal. 7, 57-81 (1997).
  * (32) J. Frehse, G. Seregin, Regularity of solutions to variational problems of the deformation theory of plasticity with logarithmic hardening, Proc. St. Petersburg Math. Soc. V, 127–152, Amer. Math. Soc. Transl. Ser. 2, 193, Amer. Math. Soc., Providence, RI, 1999.
  * (33) M. Fuchs, G. Mingione, Full $C^{1,\alpha}$-regularity for free and constrained local minimizers of elliptic variational integrals with nearly linear growth, _manuscripta math._ 102 (2000), 227–250.
  * (34) M. Fuchs, G. Seregin, _Variational methods for problems from plasticity theory and for generalized Newtonian fluids_ , Lecture Notes in Mathematics, 1749. Springer-Verlag, Berlin, 2000. vi+269 pp.
  * (35) M. Giaquinta, E. Giusti, Differentiability of minima of nondifferentiable functionals, _Invent. Math._ 72, 285-298 (1983).
  * (36) M. Giaquinta, G. Modica, Remarks on the regularity of the minimizers of certain degenerate functionals, _manuscripta math._ 57, 55-99 (1986).
  * (37) E. Giusti, Direct Methods in the calculus of variations, World Scientific Publishing Co., Inc., River Edge (2003).
  * (38) F. Gmeineder, Partial regularity for symmetric quasiconvex functionals on BD, _J. Math. Pures Appl. (IX)_ 145 83–129 (2021).
  * (39) F. Gmeineder, The regularity of minima for the Dirichlet problem on BD, _Arch. Ration. Mech. Anal._ 237, 1099–1171 (2020).
  * (40) F. Gmeineder, J. Kristensen, Partial regularity for BV minimizers, _Arch. Ration. Mech. Anal._ 232, 1429–1473 (2019).
  * (41) F. Gmeineder, J. Kristensen, Sobolev regularity for convex functionals on BD, _Calc. Var. & PDE_ 58:56 (2019).
  * (42) C. Hamburger, Regularity of differential forms minimizing degenerate elliptic functionals, _J. reine angew. Math. (Crelle J.)_ 431, 7-64 (1992).
  * (43) P. Hästö, J. Ok, Maximal regularity for local minimizers of non-autonomous functionals, _J. Eur. Math. Soc._ 24, 1285–1334 (2022).
  * (44) P. Hästö, J. Ok, Regularity theory for non-autonomous partial differential equations without Uhlenbeck structure, Arch. Ration. Mech. Anal. 245, 1401–1436 (2022).
  * (45) A.V. Ivanov, The Dirichlet problem for second order quasilinear nonuniformly elliptic equations, _Trudy Mat. Inst. Steklov._ 116 (1971) 34–54.
  * (46) A.V. Ivanov, Local estimates of the maximum modulus of the first derivatives of the solutions of quasilinear nonuniformly elliptic and nonuniformly parabolic equations and of systems of general form, _Proc. Steklov Inst. Math._ 110, 48–71 (1970).
  * (47) A.V. Ivanov, Quasilinear degenerate and nonuniformly elliptic and parabolic equations of second order, Proc. Steklov Inst. Math. 1984, no. 1 (160), xi+287 pp.
  * (48) N.M. Ivočkina, A.P. Oskolkov, Nonlocal estimates of the first derivatives of solutions of the Dirichlet problem for nonuniformly elliptic quasilinear equations, Zap. Naučn. Sem. Leningrad. Otdel. Mat. Inst. Steklov. (LOMI) 5, 37–109 (1967).
  * (49) T. Kilpeläinen, J. Malý, The Wiener test and potential estimates for quasilinear elliptic equations, Acta Math. 172, 137–161 (1994).
  * (50) L. Koch, Global higher integrability for minimisers of convex functionals with $(p,q)$-growth, _Calc. Var. & PDE_ 60:63 (2021).
  * (51) L. Koch, On global absence of Lavrentiev gap for functionals with $(p,q)$-growth, arXiv:2210.15454
  * (52) J. Kristensen, G. Mingione, The singular set of $\omega$-minima, _Arch. Ration. Mech. Anal._ 177, 93–114 (2005).
  * (53) T. Kuusi, G. Mingione, Gradient regularity for nonlinear parabolic equations, _Ann. Scu. Norm. Sup. Pisa Cl. Sci. (V)_ 12, 755–822 (2013).
  * (54) O.A. Ladyzhenskaya, N.N. Ural’tseva, Local estimates for gradients of solutions of nonuniformly elliptic and parabolic equations, _Comm. Pure Appl. Math._ 23, 677–703 (1970).
  * (55) J. J. Manfredi, Regularity of the gradient for a class of nonlinear possibly degenerate elliptic equations, Ph.D. Thesis, University of Washington, St. Louis.
  * (56) J. J. Manfredi, Regularity for minima of functionals with $p$-growth, _J. Diff. Equ._ 76, 203–212 (1988).
  * (57) P. Marcellini, On the definition and the lower semicontinuity of certain quasiconvex integrals, Ann. Inst. H. Poincaré Anal. Non Linéaire 3, 391–409 (1986).
  * (58) P. Marcellini, The stored-energy for some discontinuous deformations in nonlinear elasticity, _Partial Differential Equations and the Calculus of Variations_ vol. II, Birkhäuser Boston Inc. (1989).
  * (59) P. Marcellini, Regularity of minimizers of integrals of the calculus of variations with non standard growth conditions, Arch. Rat. Mech. Anal. 105, 267–284 (1989).
  * (60) P. Marcellini, Regularity and existence of solutions of elliptic equations with $p,q$-growth conditions, _J. Diff. Equ._ 90, 1–30 (1991).
  * (61) P. Marcellini, Regularity for elliptic equations with general growth conditions, _J. Diff. Equ._ 105, 296–333 (1993).
  * (62) P. Marcellini, G. Papi, Nonlinear elliptic systems with general growth, _J. Diff. Equ._ 221 412–443 (2006).
  * (63) V. Maz’ya, M. Havin, A nonlinear potential theory, _Russ. Math. Surveys_ , 27, 71–148 (1972).
  * (64) J. Serrin, The problem of Dirichlet for quasilinear elliptic differential equations with many independent variables, _Philos. Trans. R. Soc. Lond., Ser. A_ 264, 413-496 (1969).
  * (65) L. Simon, Interior gradient bounds for nonuniformly elliptic equations, _Indiana Univ. Math. J._ 25, 821–855(1976).
  * (66) V. Sverák, X. Yan, Non-Lipschitz minimizers of smooth uniformly convex functionals, Proc. Natl. Acad. Sci. USA 99, 15269–15276 (2002).
  * (67) H. Triebel, _Theory of function spaces. III_. Monographs in Mathematics, 100. Birkhäuser Verlag, Basel, 2006.
  * (68) N. Trudinger, The Dirichlet problem for nonuniformly elliptic equations, Bull. Am. Math. Soc. 73, 410–413 (1967).
  * (69) K. Uhlenbeck, K, Regularity for a class of non-linear elliptic systems, Acta Math. 138, 219–240 (1977).
  * (70) V. V. Zhikov, On Lavrentiev’s Phenomenon, Russian J. Math. Phys. 3, 249–269 (1995).
  * (71) V. V. Zhikov, On some variational problems, _Russian J. Math. Phys._ 5, 105–116 (1997).
  * (72) V. V. Zhikov, S.M. Kozlov, O. A. Oleĭnik, _Homogenization of differential operators and integral functionals._ Springer-Verlag, Berlin, 1994. xii+570 pp. ISBN: 3-540-54809-2

Dipartimento SMFI
Università di Parma
Parco Area delle Scienze 53/a, I-43124, Parma, Italy
Dipartimento SMFI
Università di Parma
Parco Area delle Scienze 53/a, I-43124, Parma, Italy
<EMAIL_ADDRESS>


# CV-VAE: A Compatible Video VAE for Latent Generative Video Models

Sijie Zhao Yong Zhang${}^{~{}\textrm{{\char 0\relax}}}$ Xiaodong Cun Shaoshu
Yang Muyao Niu

Xiaoyu Li Wenbo Hu Ying Shan

Tencent AI Lab

https://github.com/AILab-CVC/CV-VAE

###### Abstract

Spatio-temporal compression of videos, utilizing networks such as Variational
Autoencoders (VAE), plays a crucial role in OpenAI’s SORA and numerous other
video generative models. For instance, many LLM-like video models learn the
distribution of discrete tokens derived from 3D VAEs within the VQVAE
framework, while most diffusion-based video models capture the distribution of
continuous latent extracted by 2D VAEs without quantization. The temporal
compression is simply realized by uniform frame sampling which results in
unsmooth motion between consecutive frames. Currently, there lacks of a
commonly used continuous video (3D) VAE for latent diffusion-based video
models in the research community. Moreover, since current diffusion-based
approaches are often implemented using pre-trained text-to-image (T2I) models,
directly training a video VAE without considering the compatibility with
existing T2I models will result in a latent space gap between them, which will
take huge computational resources for training to bridge the gap even with the
T2I models as initialization. To address this issue, we propose a method for
training a video VAE of latent video models, namely CV-VAE, whose latent space
is compatible with that of a given image VAE, e.g., image VAE of Stable
Diffusion (SD). The compatibility is achieved by the proposed novel latent
space regularization, which involves formulating a regularization loss using
the image VAE. Benefiting from the latent space compatibility, video models
can be trained seamlessly from pre-trained T2I or video models in a truly
spatio-temporally compressed latent space, rather than simply sampling video
frames at equal intervals. To improve the training efficiency, we also design
a novel architecture for the video VAE. With our CV-VAE, existing video models
can generate four times more frames with minimal finetuning. Extensive
experiments are conducted to demonstrate the effectiveness of the proposed
video VAE.

††footnotetext:  ${{}^{~{}\textrm{{\char 0\relax}}}}$Corresponding author

##  1 Introduction

Video generation has gained significant public attention, especially after the
announcement of OpenAI SORA [22]. Current popular video models can be divided
into two categories based on the modeling space, i.e., pixel and latent space.
Imagen Video [14], Make-a-video [27], and Show-1 [39] are representative video
diffusion models that directly learn the distribution of pixels. On the other
hand, Phenaki [31], MAGVIT [38], VideoCrafter [4], AnimateDiff [12], VideoPeot
[16], and SORA, etc, are representative latent generative video models that
are trained in the latent space formed using variational autoencoders (VAEs).
The latter category is more prevalent due to its training efficiency.

Furthermore, latent video generative models can be classified into two groups
according to the type of VAE they utilize: LLM-like and diffusion-based video
models. LLM-like models train a transformer on discrete tokens extracted by a
3D VAE with a quantizer within the VQ-VAE framework [30]. For example,
VideoGPT [37] initially trains a 3D-VQVAE and subsequently an autoregressive
transformer in the latent space. The 3D-VQVAE is inflated from the 2D-VQVAE
[30] used in image generation. TATS [11] and MAGVIT [38] use 3D-VQGAN for
better visual quality by employing discriminators, while Phenaki [31] utilizes
a transformer-based encoder and decoder, namely CViViT.

However, recent latent diffusion-based video models typically exploit 2D VAEs,
rather than 3D VAEs, to generate continuous latents to train a UNet or DiT
[23]. The commonly used 2D VAE is the image VAE [26] from Stable Diffusion, as
training a video model from scratch can be quite challenging. Almost all high-
performing latent video models are trained with the SD image model [26] as
initialization for the inflated UNet or DiT. Examples include Align-your-
latent [3], VideoCrafter1 [4], AnimateDiff [12], SVD [2], Modelscope [32],
LaVie [33], MagicVideo [41], Latte [21], etc. Temporal compression is simply
achieved by uniform frame sampling while ignoring the motion information
between frames (see Fig. 2). Consequently, the trained video models may not
fully understand smooth motion, even when FPS is set as a condition. When
projecting a sampled latent sequence to a video using the decoder of the 2D
VAE, the generated video exhibits a low FPS and lacks visual smoothness.

Figure 1: Temporal compression difference between an image VAE and our video
one.

Figure 2: SVD inference (a) The pretrained SVD with the independently trained
Video VAE. (b) Finetuning SVD based on (a). (c) The pretrained SVD with our
video VAE.

Currently, the research community lacks a commonly used 3D video VAE for
generating continuous latent variables with spatio-temporal compression for
latent video models. Training a high-quality video VAE without considering the
compatibility with existing pretrained image and video models might not be too
difficult. However, even though the trained video VAE exhibits low
reconstruction errors, a gap exists between its learned latent space and the
one used by pretrained models, such as the video VAE of Open-Sora-Plan [17].
This means that bridging the gap requires significant computational resources
and extensive training time, even when using pre-trained models as
initialization. One example is shown in Fig. 2. When training a video VAE
independently without considering compatibility, the sampled latent of SVD [2]
cannot be projected into the pixel space correctly due to the latent space
gap, as shown in Fig. 2(a). After finetuning the SVD model in the new latent
space on 16 A100 for 58K iterations, the quality of the generated video is
still poor (see Fig. 2(b)). In contrast, our video VAE achieves promising
results in the pretrained SVD even without finetuning the UNet as shown in
Fig. 2(c).

In this work, we propose a novel method to train a video VAE to extract
continuous latents for generative video models, which is compatible with
existing pretrained image and video models, e.g. Stable Diffusion [26] and SVD
[2]. We also inflate the SD image VAE to form a video VAE by adding 3D
convolutions to both encoder and decoder of the 2D VAE, which allows us to
train video models efficiently with the pretrained models as initialization in
a truly spatio-temporally compressed latent space, instead of uniform frame
sampling for temporal compression (see Fig. 2). Consequently, the generated
videos will be smoother and have a higher FPS than those produced using a 2D
VAE.

To ensure latent space compatibility between 2D and 3D VAEs, we propose a
latent space regularization to avoid distribution shifts. We examine the
effectiveness of using either the encoder or decoder of the 2D VAE to form
constraints and explore four types of mapping functions to design
regularization. Moreover, to improve video VAE efficiency, we investigate its
architecture and partially integrate 3D convolutions instead of exploiting 3D
convolution in all blocks. The proposed video VAE can be used not only for
training new video models with pretrained ones as initialization but also as a
frame interpolator for existing video models with slight finetuning.

Our main contributions are summarized as follows: (1) We propose a video VAE
that provides a truly spatio-temporally compressed continuous space for
training latent generative video models, which is compatible with existing
image and video models, greatly reducing the expense of training or finetuning
video models. (2) We propose a latent space regularization to avoid
distribution shifts and design an efficient architecture for the video VAE.
(3) Extensive experiments are conducted to demonstrate the effectiveness of
the proposed video VAE.

##  2 Related Work

#### Variational Autoencoder.

Variational Autoencoders (VAEs), introduced by [15], have been widely used in
two-stage generative models. The first stage involves compressing the pixels
into a lower-dimensional latent representation, followed by a second stage
that generates pixels from this latent space. VAEs can be divided into two
groups according to the token, i.e., discrete and continuous latent. The
difference between the two types of VAEs is the quantization. Continuous VAEs
have no quantization, while discrete VAEs learn a codebook for quantization
and use it to convert the continuous latent features to discrete indices,
called VQVAE [30]. When training discrete VAEs, some methods exploit a
discriminator to improve the visual image quality, called VQGAN [9].

In video generation, 2D VAEs are typically inflated into 3D ones by injecting
3D Conv or temporal attention. 3D Convs are for CNN-based VAEs, e.g., 3D-VQVAE
[37], 3D-VAQGAN [11, 38]. Attentions are for transformer-based VAEs, e.g.,
CViViT [31]. Although there are several discrete 3D VAEs for video generation,
there are no commonly used continuous 3D VAEs.

#### Video Generative models.

Video generation has achieved remarkable progress in recent years. The
announcement of Imagen Video [14] and Make-A-Video [27] made researchers see
the hope of purely AI-generated videos. Then, the launch of OpenAI SORA [22]
brought the enthusiasm of researchers in academia and industry to a climax.
Many video generation models [14, 27, 39] directly learn the distribution of
pixels while some others [3, 4, 41, 32, 33, 21, 38, 11, 31, 37, 2, 12] learn
the distribution of tokens in a latent space. The tokens are always extracted
by a variational autoencoder [8]. Latent video generation models can be
categorized into two groups according to whether the token is discrete or
continuous. TATS [11], MAGVIT [38], VideoGPT [37], and Phenaki [31] are
representative models trained with discrete tokens extracted by a 3D VAE
within the VQVAE framework [30]. A codebook is learned jointly with the VAE
for quantization. SVD [2], AnimateDiff [12], VideoCrafter [4], etc., are video
models trained with continuous latent extracted by a 2D VAE without
quantization, rather than a 3D VAE. SD image VAE is the commonly used 2D VAE.
One reason is that video models are difficult to train from scratch and they
are always initialized with the weights of a pretrained T2I model such as
Stable Diffusion UNet [26]. Hence, the corresponding image VAE is used to
extract latents from a video. Since the image VAE can only perform spatial
compression, the temporal compression is realized by uniform frame sampling.
This strategy ignores the motion between key frames.

There lacks a video VAE that is compatible with the pretrained T2I or video
models. Though it is not difficult to train a video VAE (3D VAE) independently
with high reconstruction accuracy; it will result in a latent space gap
between the learned video VAE and existing pre-trained image and video models
that are always used as initialization. The Open-Sora-Plan project [17] offers
a video VAE; however, it is not compatible with existing image or video
models. Large computational resources and a long training time are required to
bridge the gap. In this work, we propose a latent space regularization method
to train a video VAE whose latent space is compatible with pretrained models.

##  3 Method

Figure 3: (a-b): Two different regularization methods; (c) The framework of
CV-VAE with the regularization of the pretrained 2D decoder.

We propose a latent space regularization method for training a video VAE that
is compatible with pre-trained image and video models. We examine multiple
strategies for implementing the regularization, focusing on either the encoder
or the decoder of the image VAE. Additionally, we explore four types of
mapping functions to develop the regularization loss. To enhance the
efficiency of the video VAE, we introduce an architecture that employs
different inflation strategies in distinct blocks, instead of incorporating 3D
convolutions in all blocks.

###  3.1 Latent Space Regularization

We inflate a 2D VAE into a 3D VAE, initializing it with the 2D VAE’s weights.
The 3D VAE is designed to be capable of encoding both image and video (see
details in Sec.3.2). The key of building a compatible video VAE is the latent
space alignment between the video VAE and the image VAE.

#### Notations.

Let $x\in\mathbb{R}^{H\times W\times 3}$ denote an image in RGB space and
$X\in\mathbb{R}^{(T+1)\times H\times W\times 3}$ denote a video with $T+1$
frames. When $T=0$, $X$ degrades into an image and the video VAE will process
it with temporal padding. $z\in\mathbb{R}^{h\times w\times c}$ denotes the
latent tokens extracted by either an image VAE or a video VAE.
$Z\in\mathbb{R}^{(t+1)\times h\times w\times c}$ is latent tokens extracted by
the video VAE. $\rho_{s}=H/h=W/w$ and $\rho_{t}=T/t$ are the spatial and
temporal compression rates. Let $\mathcal{E}_{i}$ and $\mathcal{D}_{i}$ denote
the encoder and decoder of the image VAE, respectively. While
$\mathcal{E}_{v}$ and $\mathcal{D}_{v}$ are for the video VAE. Then, we have
$z=\mathcal{E}_{i}(x)$, $Z=\mathcal{E}_{v}(X)$, and $z=\mathcal{E}_{v}(x)$.
$\tilde{x}=\mathcal{D}_{i}(z)=\mathcal{D}_{i}(\mathcal{E}_{i}(x))$,
$\tilde{X}=\mathcal{D}_{v}(Z)=\mathcal{D}_{v}(\mathcal{E}_{v}(X))$, and
$\tilde{x}=\mathcal{D}_{v}(z)=\mathcal{D}_{v}(\mathcal{E}_{v}(x))$ are the
reconstructed image and video from the latent tokens.

#### Regularization.

We assume the latent of the image VAE follow a distribution, i.e, $z\sim
p^{i}(z)$. The joint distribution of $t+1$ independent frames is
$p^{i}(Z)=\prod_{k}^{t+1}p^{i}(z_{k})$. The latent distribution of the video
VAE can be denoted as $Z\sim p^{v}(Z)$. To achieve the alignment between the
latent spaces of the image and video VAEs, we have to build mappings between
$p^{i}(Z)$ and $p^{v}(Z)$. Since both distributions have no analytic
formulation, distance metric for measuring differences between distributions
is not applicable.

Here, we build the cooperation between the image VAE and the video one to
construct reconstruction loss for space alignment. When exploiting the encoder
of the image VAE for alignment, the latent extracted from the image encoder
should be corrected decoded by the decoder of the video VAE, i.e.,
$\tilde{X}_{i}^{v}=\mathcal{D}_{v}(\mathcal{E}_{i}(\psi(X)))$. The
illustration is shown in Fig. 3(a). For a given input video
$X\in\mathbb{R}^{(T+1)\times H\times W\times 3}$, we use a mapping function
$\psi$ to sample $\psi(X)\in\mathbb{R}^{(T/\rho_{t}+1)\times H\times W\times
3}$. Thus the reconstructed video $\tilde{X}_{i}^{v}$ is the same as the shape
of $X$. Then, the reconstruction loss of using the image encoder can be
defined as

$L_{\text{reg}}^{\text{en}}=||X-\tilde{X}_{i}^{v}||^{2}.$ (1)

When exploiting the decoder of the image VAE, the latent extracted by the
video encoder can be decoded by the decoder of the image VAE, i.e.,
$\tilde{X}_{v}^{i}=\mathcal{D}_{i}(\mathcal{E}_{v}(X))$. The illustration is
shown in Fig. 3(b). For a given input video $X\in\mathbb{R}^{(T+1)\times
H\times W\times 3}$, the reconstructed video is
$\tilde{X}_{i}^{v}\in\mathbb{R}^{(T/\rho_{t}+1)\times H\times W\times 3}$.
Then, the reconstruction loss of using the image decoder can be defined as

$L_{\text{reg}}^{\text{dec}}=||\psi(X)-\tilde{X}_{v}^{i}||^{2}.$ (2)

#### Mapping Functions.

To bridge the dimension gap between $\tilde{X}_{v}^{i}$ or $\tilde{X}_{i}^{v}$
and $X$, we investigate four types of mapping functions $\psi$ as follows. 1)
First frame. We compare only the first frame of the input video and the
reconstructed one. The regularization loss degenerates to measure the
difference between the input and reconstruction of the image. 2) Slice. $\psi$
samples one frame every $\rho_{t}$ frames to form a shorter video. It starts
from the second frame and the first one is reserved. 3) Average. $\psi$
computes the average of every $\rho_{t}$ frames, starting from the second
frame. 4) Random. $\psi$ randomly samples one frame from every $\rho_{t}$
frames, starting from the second frame.

#### Training Objective.

Following the training of the 2D VAE in LDM [26], our basic objective is a
combination of a reconstruction loss [40], an adversarial loss [9], and a KL
regularization [15], i.e.,

$L_{\text{AE}}=\min_{\mathcal{E}_{v},\mathcal{D}_{v}}\max_{D_{v}}~{}~{}L_{\text{rec}}(X,\mathcal{D}_{v}(\mathcal{E}_{v}(X))-L_{\text{adv}}(\mathcal{D}_{v}(\mathcal{E}_{v}(X)))+\log
D_{v}(X)+L_{\text{KL}}(X;\mathcal{E}_{v},\mathcal{D}_{v}),$

where the first term is the reconstruction loss, the second and third are the
adversarial loss, and the last is the KL regularization. $D_{v}$ is the
discriminator that differentiates original videos from reconstructed ones. It
is inflated from the image discriminator in LDM by injecting 3D convolutions.
Then, for latent space alignment, our full training objective is:

$L_{\text{AE}}^{\text{align}}=L_{\text{AE}}+\lambda_{1}L_{\text{reg}}^{\text{dec}}+\lambda_{2}L_{\text{reg}}^{\text{en}},$
(3)

where $\lambda_{1}$ and $\lambda_{2}$ are trade-off parameters. We explore
different settings of $\lambda_{1}$ and $\lambda_{2}$ and find that using the
decoder only achieves the best performance. The framework of CV-VAE is shown
in Fig. 3(c) and evaluations between different regularization methods can be
found in Tab. 4.

###  3.2  Architecture Design of Video VAE

We design the architecture of the video VAE according to the image VAE in LDM
[26]. The detailed architecture is presented in the Appendix A.1. We explain
the key modifications as follows.

#### Model Inflation.

Considering the latent space compatibility and the convergence speed of the
video VAE, we make full use of the pretrained weights of the image VAE for
initialization, instead of training from scratch. We inflate the image VAE
into the video VAE by replacing 2D convolutions with 3D convolutions. 3D
convolutions are used to model the temporal dynamics among frames. To
initialize the 3D convolutions, we copy the weights of the 2D Conv kernel to
the corresponding positions in the 3D Conv kernel and set the remaining
parameters to zero. We set the stride to achieve temporal downsampling and
increase the number of 3D kernels by a factor of $s$ to achieve $s\times$
temporal upsampling. To enable the video VAE to handle both image and video,
given $T+1$ frames as input, we use reflection padding in the temporal
dimension for the first frame. By initializing the video VAE using the above
operations, we can reconstruct images without training, significantly
accelerating the training convergence speed on video datasets.

#### Efficient 3D Architecture.

Expanding 2D Convs to 3D Convs (e.g., $k\times k\to k\times k\times k$)
results in $k\times$ parameters and computational complexity. To improve the
computational efficiency of the model, we adopt a 2D+3D network structure.
Specifically, we retain half of the convolutions in the ResBlock as 2D Convs
and set the other half as 3D Convs. We find that, compared to setting all
Convs to 3D, the number of parameters and the computational complexity are
reduced by roughly 30%, while the reconstruction performance remains nearly
the same. See Sec. 4.2 for experimental comparisons.

#### Temporal Tiling for Arbitrary Video Length

Existing image VAEs employ spatial tiling on large spatial resolution images
to achieve memory-friendly processing, which cannot handle long videos. As a
result, we introduce temporal tiling processing. During encoding, the video
$X$ is divided into $[X_{1},X_{2},...X_{n}]$, where
$X_{i}\in\mathbb{R}^{(1+f\cdot\rho_{t})\times H\times W\times 3}$ and $f$ is a
parameter controlling the size of each block. $X_{i}$ and $X_{i+1}$ have a
one-frame overlap in the temporal dimension. After encoding each $X_{i}$ to
obtain $Z_{i}$, we discard the first frame of $Z_{i}$ when $i\neq 0$ and
concatenate all $Z_{i}$ in the temporal dimension to obtain $Z$. The decoding
process is handled similarly to the encoding process. By combining our method
with 2D tiling, we can encode videos with arbitrary resolution and length.

##  4 Experiments

###  4.1 Experimental Setups

Datasets and Metrics. We evaluate our CV-VAE on the COCO2017 [18] validation
dataset and the Webvid [1] validation dataset which includes 1024 videos. Both
images and videos are resized and cropped to a resolution of $256\times 256$.
Each video is sampled with 33 frames and a frame stride of 3. We evaluate the
reconstruction performance of CV-VAE on images and videos using metrics such
as PSNR, SSIM [34], and LPIPS scores [40]. We employ 3D tiled processing to
encode and decode videos with arbitrary resolution and length within a limited
memory footprint. During inference, we allow a single video block size of
$17\times 576\times 576$. We evaluate the video generation quality of our
model using 2048 randomly sampled videos from UCF101 [28] and MSR-VTT [36].
Videos are resized and cropped to a resolution of $576\times 1024$ to fit the
SVD [2]. We use Frechet Video Distance (FVD) [29], Kernel Video Distance (KVD)
[29], and Perceptual Input Conformity (PIC) [35] metrics to evaluate video
generation quality. For evaluating image generation quality, we use 2048
samples from the COCO2017 validation dataset and employ FID [13], CLIP score
[24], and PIC score metrics.

Training Details. We train our CV-VAE model using image datasets including
LAION-COCO [7] and Unsplash [20], as well as the video dataset Webvid-10M [1].
For image datasets, we employ two resolutions, i.e., $256\times 256$ and
$512\times 512$. In the case of video datasets, we use two settings of frames
and resolutions: $9\times 256\times 256$ and $17\times 192\times 192$. The
batch sizes for these four settings are 8, 2, 1, and 1, with sampling ratios
of 40%, 10%, 25%, and 25%, respectively. We employed the AdamW optimizer [19]
with a learning rate of 1e-4 and cosine learning rate decay. To avoid
numerical overflow, we trained CV-VAE using float32 precision, and the
training was carried out on 16 A100 GPUs for 200K steps. To fine-tune the SVD
on CV-VAE, we utilize in-house data with a frame rate and resolution of
$97\times 576\times 1024$. We employ deepspeed stage 2 [25], gradient
checkpointing [6] techniques, and train with bfloat16 precision. We used a
constant learning rate of 1e-5 with the AdamW [19] optimizer, and only
optimized the last layer of U-Net. The training was carried out on 16 A100
GPUs for 5K steps.

###  4.2  Image and Video Reconstruction

We evaluated the reconstruction quality of various VAE models on image and
video test sets. The comparison group includes: (1) VAE-SD2.1 [26] which is
widely used in the community for image and video generation models. (2) VQGAN
[9] which encoding pixels into discrete latents. We use the f8-8192 version
for comparision. (3) TATS [11]: a 3D VQGAN designed for video generation. (4)
VAE-OSP [17]: a 3D VAE from Open-Sora-Plan which is initialized from VAE-SD2.1
and trained with video data. (5) Our CV-VAE (2D+3D): retains half of the 2D
convolutions to reduce computational overhead. (6) Our CV-VAE (3D): utilizes
only 3D convolutions.

Method | Params | FCR | Comp. | COCO-Val | Webvid-Val
---|---|---|---|---|---
PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
VAE-SD2.1 [26] | 34M + 49M | 1x | - | 26.6 | 0.773 | 0.127 | 28.9 | 0.810 | 0.145
VQGAN [9] | 26M + 38M | 1x | $\times$ | 22.7 | 0.678 | 0.186 | 24.6 | 0.718 | 0.179
TATS [11] | 7M + 16M | 4x | $\times$ | 23.4 | 0.741 | 0.287 | 24.1 | 0.729 | 0.310
VAE-OSP [17] | 94M +135M | 4x | $\times$ | 27.0 | 0.791 | 0.142 | 26.7 | 0.781 | 0.166
Ours(2D+3D) | 68M + 114M | 4x | $\checkmark$ | 27.6 | 0.805 | 0.136 | 28.5 | 0.817 | 0.143
Ours(3D) | 100M + 156M | 4x | $\checkmark$ | 27.7 | 0.805 | 0.135 | 28.6 | 0.819 | 0.145
Table 1: Quantitative evaluation on image and video reconstruction. FCR
represents the frame compression rate, and Comp. indicates compatibility with
existing generative models. Figure 4: Qualitative comparison of image and
video reconstruction. Top: Reconstruction with different Image VAE models
(i.e., VQGAN [9] and VAE-SD2.1 [26] ) on images; Bottom: Reconstruction with
different Video VAE models (i.e., TATS [11] and VAE-OSP [17]) on video frames.

As illustrated in Tab. 1, we present the parameter count (Params), Frame
Compression Ratio (FCR), and compatibility with existing diffusion models
(Comp.) for various VAE models. Thanks to the latent constraint, our model is
compatible with current diffusion models, compresses videos by 4$\times$ in
the temporal dimension, and achieves top-tier image and video reconstruction
quality. This enables the generation of longer videos under roughly the same
computational resources.

We also conducted a qualitative comparison of the reconstruction results for
different VAE models, as shown in Fig. 4. In the top row, we reconstructed
images with a resolution of $512\times 512$ and compared them with Image VAE
models. All three models compressed the images to a latent size of $64\times
64$. Our results were close to those of VAE-SD2.1, while VQGAN had the worst
performance. In the bottom row, we reconstructed videos with a resolution of
$33\times 512\times 512$ and compared them with Video VAE models. All three
models compressed the videos to a latent size of $9\times 64\times 64$.
Comparing the decoded videos at the same frames, our model achieved the best
results. Check Appendx A.2 and A.3 for more reconstruction results.

###  4.3 Compatibility with Existing Models

Figure 5: Text-to-image generation comparison. In each pair, the left is
generated by the SD2.1 [26] with the image VAE while the right is generated by
the SD2.1 with our video VAE.

#### Image Models

We tested the compatibility of our CV-VAE by integrating it into the
pretrained SD2.1 [26], replacing the original 2D VAE without any finetuning.
We evaluated it on the COCO-Val [18] dataset and compared the results with the
SD2.1 model using PID, CLIP score, and PIC metrics. The data (see Tab. 2)
suggest that both models perform similarly in text-to-image generation.

We also visualized the text-to-image generation results of both models in Fig.
5. In each pair, the left side depicts the results of SD2.1, while the right
side shows the results generated by our CV-VAE, which replaced the original
VAE, using the same random seed and prompt. The results show that both models
generate images with almost identical content and texture, with only slight
differences in color. This further validates the feasibility of building a
compatible VAE via latent constraint.

SVD |  [width=]4figures/svd_xt_frames_512/video1/frame_024 |  [width=]4figures/svd_xt_frames_512/video2/frame_024 |  [width=]4figures/svd_xt_frames_512/video3/frame_024
---|---|---|---
SVD $+$ CV-VAE |  [width=]16figures/output_layer_frames_512/video1/frame_096 |  [width=]16figures/output_layer_frames_512/video2/frame_096 |  [width=]16figures/output_layer_frames_512/video3/frame_096
SVD |  [width=]4figures/svd_xt_frames_512/video4/frame_024 |  [width=]4figures/svd_xt_frames_512/video5/frame_024 |  [width=]4figures/svd_xt_frames_512/video6/frame_024
SVD $+$ CV-VAE |  [width=]16figures/output_layer_frames_512/video4/frame_096 |  [width=]16figures/output_layer_frames_512/video5/frame_096 |  [width=]16figures/output_layer_frames_512/video6/frame_096
Figure 6: Comparison between the image VAE and our video VAE on image-to-video generation of SVD [2]. ‘SVD’ means using the image VAE. ‘SVD $+$ CV-VAE’ means using our video VAE and tuning the output layer of SVD. Click to play the video clips with Adobe or Foxit PDF Reader. | Trainable | COCO2017-Val
---|---|---
FID($\downarrow$) |  CLIP($\downarrow$) |  PIC($\uparrow$)
SD2.1 [26] | $\times$ | 57.3 | 0.312 | 0.354
SD2.1+CV-VAE | $\times$ | 57.6 | 0.311 | 0.360
Table 2: Quantitative results of text-to-image generation.

#### Video Models

The primary objective of CV-VAE is to train a model that can compress both
time and space, while also being compatible with the existing 2D VAE. In this
section, we validate the compatibility of CV-VAE with existing video
generation models. We integrate CV-VAE into SVD [2], replacing the original
VAE, and decoded the generated video latents. CV-VAE offers the flexibility to
decode either in image mode (CV-VAE-I) or video mode (CV-VAE-V); the former
decodes n frames of latent into n frames of video, while the latter decodes n
frames of latent into $1+(n-1)\times 4$ frames of video. We tested the video
generation quality of both models. Furthermore, we fine-tuning the SVD for
better alignment.

Method | Trainable | FCR | Frames | UCF-101 | MSR-VTT
---|---|---|---|---|---
|  |  |  |  FVD($\downarrow$) |  KVD($\downarrow$) |  PIC($\uparrow$) |  FVD($\downarrow$) |  KVD($\downarrow$) |  PIC($\uparrow$)
SVD [2] | $\times$ | 1x | 25 | 402 | 8.20 | 0.791 | 310 | 1.30 | 0.588
SVD+CV-VAE-I | $\times$ | 1x | 25 | 419 | 6.73 | 0.763 | 262 | 1.67 | 0.609
SVD+CV-VAE-V | $\times$ | 4x | 97 | 762 | 15.7 | 0.791 | 319 | 3.31 | 0.696
SVD+CV-VAE-V | Output layer | 4x | 97 | 681 | 13.1 | 0.858 | 295 | 2.26 | 0.734
Table 3: Evaluation results of image-to-video generation. FCR denotes the frame compression rate. Constraint | COCO-Val | Webvid-Val
---|---|---
|  PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
2D Enc. | 26.0 | 0.759 | 0.205 | 26.0 | 0.748 | 0.222
2D Dec. | 27.5 | 0.801 | 0.151 | 28.0 | 0.803 | 0.158
2D Dnc. + Dec. | 27.9 | 0.808 | 0.150 | 27.6 | 0.795 | 0.176
Table 4: Comparison of different regularization types. Mapping Function | COCO-Val | Webvid-Val
---|---|---
PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$) |  PNSR($\uparrow$) |  SSIM($\uparrow$) |  LPIPS($\downarrow$)
1st Frame | 27.3 | 0.797 | 0.156 | 26.6 | 0.771 | 0.191
Average | 27.5 | 0.801 | 0.151 | 27.9 | 0.801 | 0.172
Slice | 27.5 | 0.802 | 0.152 | 27.7 | 0.799 | 0.168
Random | 27.6 | 0.803 | 0.138 | 28.4 | 0.811 | 0.153
Table 5: Comparison of different mapping functions.

As shown in Tab. 3, incorporating CV-VAE-I into a frozen SVD immediately
yields video generation quality comparable to the original VAE. Using CV-VAE
in video mode can also decode videos generated by SVD, and further
improvements in video decoding quality can be achieved by fine-tuning only the
output layer (approximately 12k parameters). One of the reasons for the
noticeable gap in test metrics between SVD+CV-VAE-I and SVD+CV-VAE-V is that
they use different numbers of frames, making a direct comparison challenging.

In Fig. 6, we also display the comparison results with SVD [2]. The top row
shows the generated results by SVD, and the bottom row shows the generated
results after inserting CV-VAE into SVD and fine-tuning the output layer. We
use the first frame as a condition and generate with the same random seed. The
U-Net generates 25 frames of latent, which are decoded by CV-VAE into a
97-frame video. As can be seen, compared to the original SVD, our results
exhibit smoother motion. It is worth noting that both models have the same
computational complexity during the diffusion process, which means that our
model is more scalable.

###  4.4 Ablation Study

#### Influence of Regularization Type

We evaluated the impact of three types of latent regularization, which are:
(1) 2D Enc. , i.e., $\lambda_{1}=0$ and $\lambda_{2}=1$ in Eq. 3; (2) 2D Dec.
, i.e., $\lambda_{1}=1$ and $\lambda_{2}=0$ in Eq. 3; (3) 2D Enc. + Dec. ,
i.e., $\lambda_{1}=1$ and $\lambda_{2}=1$ in Eq. 3.

Tab. 4 shows the impact of various latent regularization methods. Using the 2D
decoder for latent regularization results in better reconstruction for both
image and video test sets compared to the 2D encoder. This is likely because
the gradient backpropagation through the 2D decoder provides better guidance
for the 3D VAE’s learning, while the frozen 2D encoder doesn’t propagate
gradients. The ‘2D Enc. + Dec.’ method performs slightly better on image test
sets but worse on video datasets compared to ‘2D Enc.’ Since our main goal is
video reconstruction and for simplicity, we use the 2D decoder for
regularization.

#### Influence of Mapping Functions

The 2D decoder decodes $n$ frames of latents into $n$ frames of video, while
the 3D decoder decodes the same $n$ frames of latents into $1+(n-1)\times 4$
frames of video. Therefore, we need to mapping the input video to $n$ frames
to calculate the regularization loss in Eq. 2. We evaluated four mapping
functions mentioned in Sec. 3.1.

As shown in Tab. 5, the four methods have similar effects on image
reconstruction, with the main differences being in video reconstruction. The
‘1st Frame’ approach yields the worst video reconstruction results due to the
lack of regularization and guidance for subsequent frames. The ‘Slice’ method
results in poor reconstruction quality for the three unsampled middle frames.
The ‘Average’ method is inferior to ‘Random’ in video reconstruction,
primarily because calculating the mean for multiple consecutive frames leads
to motion blur in the target.

##  5 Conclusion and Limitations

We propose a novel method to train a video VAE that is compatible with
existing image and video models trained with SD image VAE. The video VAE
provides a truly spatio-temporally compressed latent space for latent
generative video models, as opposed to uniform frame sampling. Due to the
latent space compatibility, a new video model can be trained efficiently with
the pretrained image or video models as initialization. Besides, existing
video models such as SVD can generate smoother videos with four times more
frame using our video VAE by slightly fine-tuning a few parameters. Extensive
experiments are performed to demonstrate the effectiveness of the proposed
VAE.

#### Limitations.

The performance of the proposed video VAE relies on the channel dimension of
the latent space. A higher dimension may yield better reconstruction accuracy.
Since we pursue the latent space compatibility with existing image and video
models trained with SD image VAE, the channel dimension of our video VAE is
limited to be the same as the image VAE. This can be improved if an image VAE
with a higher channel dimension becomes available, e.g., the VAE of SD3 [10].

## References

  * Bain et al. [2021] M. Bain, A. Nagrani, G. Varol, and A. Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _IEEE International Conference on Computer Vision_ , 2021.
  * Blattmann et al. [2023a] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _arXiv preprint arXiv:2311.15127_ , 2023a.
  * Blattmann et al. [2023b] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22563–22575, 2023b.
  * Chen et al. [2023] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q. Chen, X. Wang, et al.  Videocrafter1: Open diffusion models for high-quality video generation.  _arXiv preprint arXiv:2310.19512_ , 2023.
  * Chen et al. [2024] H. Chen, Y. Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y. Shan.  Videocrafter2: Overcoming data limitations for high-quality video diffusion models.  _arXiv preprint arXiv:2401.09047_ , 2024.
  * Chen et al. [2016] T. Chen, B. Xu, C. Zhang, and C. Guestrin.  Training deep nets with sublinear memory cost.  _arXiv preprint arXiv:1604.06174_ , 2016.
  * Christoph et al. [2022] S. Christoph, K. Andreas, V. Richard, C. Theo, and B. Romain.  Laion-coco: 600m synthetic captions from laion2b-en, 2022.  URL https://laion.ai/blog/laion-coco/.
  * Doersch [2016] C. Doersch.  Tutorial on variational autoencoders.  _arXiv preprint arXiv:1606.05908_ , 2016.
  * Esser et al. [2021] P. Esser, R. Rombach, and B. Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Esser et al. [2024] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al.  Scaling rectified flow transformers for high-resolution image synthesis.  _arXiv preprint arXiv:2403.03206_ , 2024.
  * Ge et al. [2022] S. Ge, T. Hayes, H. Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh.  Long video generation with time-agnostic vqgan and time-sensitive transformer.  In _European Conference on Computer Vision_ , pages 102–118, 2022\.
  * Guo et al. [2023] Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ , 2023.
  * Heusel et al. [2017] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2022] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kondratyuk et al. [2023] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, R. Hornung, H. Adam, H. Akbari, Y. Alon, V. Birodkar, et al.  Videopoet: A large language model for zero-shot video generation.  _arXiv preprint arXiv:2312.14125_ , 2023.
  * Lab and etc. [2024] P.-Y. Lab and T. A. etc.  Open-sora-plan, 2024.  URL https://doi.org/10.5281/zenodo.10948109.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Loshchilov and Hutter [2017] I. Loshchilov and F. Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Luke Chesser [2023] A. Z. Luke Chesser, Timothy Carbone.  Unsplash.  https://github.com/unsplash/datasets, 2023.
  * Ma et al. [2024] X. Ma, Y. Wang, G. Jia, X. Chen, Z. Liu, Y.-F. Li, C. Chen, and Y. Qiao.  Latte: Latent diffusion transformer for video generation.  _arXiv preprint arXiv:2401.03048_ , 2024.
  * OpenAI [2024] OpenAI.  Openai sora.  Accessed May 15, 2024 [Online], 2024.  URL https://openai.com/index/video-generation-models-as-world-simulators/.
  * Peebles and Xie [2023] W. Peebles and S. Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763, 2021.
  * Rajbhandari et al. [2020] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16, 2020.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Singer et al. [2022] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Soomro et al. [2012] K. Soomro, A. R. Zamir, and M. Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * Unterthiner et al. [2018] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly.  Towards accurate generative models of video: A new metric & challenges.  _arXiv preprint arXiv:1812.01717_ , 2018.
  * Van Den Oord et al. [2017] A. Van Den Oord, O. Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Villegas et al. [2022] R. Villegas, M. Babaeizadeh, P.-J. Kindermans, H. Moraldo, H. Zhang, M. T. Saffar, S. Castro, J. Kunze, and D. Erhan.  Phenaki: Variable length video generation from open domain textual descriptions.  In _International Conference on Learning Representations_ , 2022.
  * Wang et al. [2023a] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang.  Modelscope text-to-video technical report.  _arXiv preprint arXiv:2308.06571_ , 2023a.
  * Wang et al. [2023b] Y. Wang, X. Chen, X. Ma, S. Zhou, Z. Huang, Y. Wang, C. Yang, Y. He, J. Yu, P. Yang, et al.  Lavie: High-quality video generation with cascaded latent diffusion models.  _arXiv preprint arXiv:2309.15103_ , 2023b.
  * Wang et al. [2004] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Xing et al. [2023] J. Xing, M. Xia, Y. Zhang, H. Chen, X. Wang, T.-T. Wong, and Y. Shan.  Dynamicrafter: Animating open-domain images with video diffusion priors.  _arXiv preprint arXiv:2310.12190_ , 2023.
  * Xu et al. [2016] J. Xu, T. Mei, T. Yao, and Y. Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5288–5296, 2016.
  * Yan et al. [2021] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas.  Videogpt: Video generation using vq-vae and transformers.  _arXiv preprint arXiv:2104.10157_ , 2021.
  * Yu et al. [2023] L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y. Hao, I. Essa, et al.  Magvit: Masked generative video transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10459–10469, 2023.
  * Zhang et al. [2023] D. J. Zhang, J. Z. Wu, J.-W. Liu, R. Zhao, L. Ran, Y. Gu, D. Gao, and M. Z. Shou.  Show-1: Marrying pixel and latent diffusion models for text-to-video generation.  _arXiv preprint arXiv:2309.15818_ , 2023.
  * Zhang et al. [2018] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 586–595, 2018.
  * Zhou et al. [2022] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv preprint arXiv:2211.11018_ , 2022.

##  Appendix A Appendix

###  A.1 CV-VAE Model Architecture

Figure 7: Architecture of CV-VAE.

As illustrated in Fig. 7, we introduce the structure of the CV-VAE. The
architecture of CV-VAE is primarily derived from the VAE in Stable Diffusion
[26], with several notable differences: (1) Some or all 2D convolutions within
the network are transformed into 3D convolutions, while retaining their
weights. (2) Temporal downsampling is executed in the encoder through the use
of strides. (3) Temporal upsampling is accomplished by increasing the output
channel number of 3D convolutions by a specific factor. (4) A discriminator,
comprising 3D convolutions, is utilized. The main differences are marked in
red text in Fig. 7.

###  A.2 Qualitative Examples of Image Reconstruction

Figure 8: Our CV-VAE is capable of encoding and reconstructing images with
high fidelity.

In Fig. 8, we showcase additional image reconstruction results using CV-VAE.
we use the version of ‘2D + 3D’. These images are sourced from the COCO2017
[18] dataset with a resolution of $512\times 512$. The reconstructed image
precisely shares the same colors and textures as the original, demonstrating
the high fidelity of our CV-VAE in encoding and reconstructing images.
Interestingly, in Fig. 5, slight color differences can be observed between the
images decoded by the Image VAE and CV-VAE, given the same latent generated by
the Image Diffusion Model. This suggests that there is still a minor
discrepancy between the latent spaces of the video VAE trained with latent
regularization and the Diffusion Model. This gap can be bridged with minimal
additional training.

###  A.3 Qualitative Examples of Video Reconstruction

Figure 9: Reconstruction results of consecutive frames using CV-VAE.

As shown in Fig. 9, we present the reconstruction results of 4 consecutive
frames from a video clip ($33\times 576\times 1024$) using CA-VAE. The
reconstructed video frames maintain consistency in color, structure, and
motion with the ground truth. According to CA-VAE, these continuous frames are
condensed into a single latent frame, signifying that even a single latent
frame encapsulates motion information.

###  A.4 Compatibility with Existing Text-to-video Model

Figure 10: Comparison between the image VAE and our video VAE on text-to-video
generation of VC2 [5]. We fine-tuned the last layer of U-Net in VC2 to adapt
it to CV-VAE. VC2 generates videos with a resolution of $16\times 320\times
512$, while the ‘VC2 + CV-VAE’ produces videos of $61\times 320\times 512$
resolution under the same computation. The missing frames in the VC2 results
are marked in gray.

We tested the compatibility of CV-VAE with existing text-to-video diffusion
models, such as Videocrafter2 [5], which also employs a 2D VAE from the SD as
its first-stage model. We adopted a strategy similar to the training of ‘SVD +
CV-VAE’ in Sec. 4.3, by fine-tuning the last layer of the U-Net in VC2 to
adapt it to CV-VAE. We finetuned the model using in-house data at a resolution
of $61\times 320\times 512$, which is equivalent to a latent size of $16\times
40\times 64$.

As shown in Fig. 10, compared to the original VC2, the ‘VC2 + CV-VAE generates
videos approximately four times longer, resulting in smoother motion. This
further validates the feasibility of obtaining a compatible video VAE through
latent regularization, thereby avoiding the massive computational power
required to train a video diffusion model from scratch.

##  Appendix B Society Impacts

The CV-VAE can be seamlessly integrated into existing diffusion models,
replacing the original 2D VAE for image or video generation, which may result
in potential societal implications. While it proves beneficial in fields such
as entertainment and advertising, by providing more realistic and immersive
content, it also raises ethical and safety concerns. The ease of generating
high-quality synthetic images and videos could lead to a surge in the
production of harmful or misleading content, such as deepfakes, potentially
exacerbating issues of misinformation and privacy invasion. We condemn the
misuse of generative AI that harms individuals or spreads misinformation.