# Energy storage in lead-free Ba(Zr, Ti)O${}_{\textrm{3}}$ relaxor
ferroelectrics:
Large densities and efficiencies and their origins

Zhijun Jiang  MOE Key Laboratory for Nonequilibrium Synthesis and Modulation
of Condensed Matter, Shaanxi Province Key Laboratory of Advanced Functional
Materials and Mesoscopic Physics, School of Physics, Xi’an Jiaotong
University, Xi’an 710049, China  Physics Department and Institute for
Nanoscience and Engineering, University of Arkansas, Fayetteville, Arkansas
72701, USA  Sergey Prosandeev  Physics Department and Institute for
Nanoscience and Engineering, University of Arkansas, Fayetteville, Arkansas
72701, USA  L. Bellaiche  Physics Department and Institute for Nanoscience and
Engineering, University of Arkansas, Fayetteville, Arkansas 72701, USA

###### Abstract

An atomistic first-principles-based effective Hamiltonian is used to
investigate energy storage in Ba(Zr0.5Ti0.5)O3 relaxor ferroelectrics, both in
their bulk and epitaxial films’ forms, for electric fields applied along
different crystallographic directions. We find that the energy density
linearly increases with temperature for electric fields applied along the
pseudocubic [001], [110] and [111] directions in Ba(Zr0.5Ti0.5)O3 bulk. For
films at room temperature, the energy density adopts different behaviors
(i.e., increase versus decrease) with strain depending on the direction of the
applied electric fields. We also predicted ultrahigh energy densities
(basically larger than 100 J/cm3) with an ideal efficiency of 100% in all
these Ba(Zr0.5Ti0.5)O3 systems. In addition, a phenomenological model is used
to reveal the origin of all the aforementioned features, and should be
applicable to other relaxor ferroelectrics.

##  I Introduction

Dielectric capacitors are particularly promising for high-power energy storage
applications because of their ultrafast charging/discharging rates and high
reliability Chu2006 ; Li2015 ; Peng2015 ; Prateek2016 ; Palneedi2018 ;
Yang2019 ; Pan2019 . However, dielectric capacitors have relatively low energy
densities and efficiencies, which is the main bottleneck towards applications
in electronics and electric power systems. Intensive efforts have been devoted
to find novel materials with higher energy density and efficiency. Recently, a
special class of ferroelectrics with a frequency-dependent dielectric
response-versus-temperature and several characteristic temperatures Cross1994
; Bokov2006 ; Shvartsman2012 ; Akbarzadeh2012 ; Prosandeev2015 , namely the
relaxor ferroelectrics, have been attracting much attention because of their
ultrahigh energy densities and efficiencies Instan2017 ; Kim2020 ; Pan2021 .
One can for instance cite Ba(ZrxTi1-x)O3 thin films with a recoverable energy
density of 158 J/cm3 and an efficiency of 72.8% Instan2017 ,
0.68Pb(Mg1/3Nb2/3)O3-0.32PbTiO3 (PMN-PT) films with 133 J/cm${}^{\textrm{3}}$
and 75% Kim2020 , and Sm-doped $y$BFO-(1$-$$y$)BTO (Sm-BFBT) with 152
J/cm${}^{\textrm{3}}$ and a marked enhancement of efficiency above 90% at an
electric field of 3.5 MV/cm Pan2021 . It is also worth noting that relaxor
ferroelectrics in the range between the Burns temperature $T_{\textrm{b}}$ and
temperature $T_{\textrm{m}}$ at which the dielectric constant exhibits a peak
can be defined as superparaelectric relaxor ferroelectrics Pan2021 ; Cross1987
, which is highly promising for energy storage applications. As a matter of
fact, the superparaelectric state in relaxor ferroelectrics can exhibit large
polarizability at high electric fields with nonlinear $P$-$E$ loop behavior
while maintaining very small hysteresis Glazounov1995 .

Despite these impressive progresses on the use of relaxor ferroelectrics for
energy storage, several questions remain unaddressed, to the best of our
knowledge. For instance, can ab-initio-based methods reproduce the
experimental finding about ultrahigh energy density in the lead-free relaxor
Ba(ZrxTi1-x)O3 system, and, if yes, can they provide new insight into its
origin? The effect of the direction of the applied electric field on energy
storage density and efficiency is also presently unknown in this material, to
the best of our knowledge. Similarly, we are not aware that the consequence of
the epitaxial strain on energy-storage-related properties has ever been
investigated, and thus revealed and explained, in films made of
Ba(ZrxTi1-x)O3.

The aim of this article is to answer all these questions, by using an
atomistic first-principles-based effective Hamiltonian and analyzing its
results via a simple and straightforward phenomenological model. In
particular, we demonstrate, and explain why, ultrahigh energy density and
efficiency can be achieved in Ba(Zr0.5Ti0.5)O3 ferroelectric relaxors, both in
their bulk and epitaxial films’ forms. The effects of the direction of the
applied field as well as epitaxial strain are also revealed and understood.

This article is organized as follows. Section II provides details about the
atomistic effective Hamiltonian method used here. Sections III A and III B
report and explain energy-storage results in Ba(Zr0.5Ti0.5)O3 bulk and films,
respectively. Finally, Section IV concludes this work.

##  II Methods

Here, the first-principles-based effective Hamiltonian ($H_{\textrm{eff}}$)
approach that has been developed and used in Refs. Akbarzadeh2012 ;
Prosandeev2013 ; Prosandeev2013_film ; Jiang2017  is employed to investigate
bulk and epitaxial films made of Ba(Zr0.5Ti0.5)O3 (BZT) solid solutions. This
$H_{\textrm{eff}}$ successfully reproduced (i) the existence of temperatures
characteristic of relaxor ferroelectrics Akbarzadeh2012  (such as the Burns
temperature $T_{\textrm{b}}$ that typically marks the existence of dynamical
polar nanoregions (PNRs) Akbarzadeh2012 ; Burns1983 , $T_{\textrm{b}}$
$\simeq$ 450 K, the so-called $T^{*}$ temperature at which static PNRs
typically appear Akbarzadeh2012 ; Dkhil2009 ; Svitelskiy2005 , $T^{*}$
$\simeq$ 240 K, and $T_{\textrm{m}}$ $\simeq$ 130 K, at which the dielectric
response adopts a peak Cross1994 ; Akbarzadeh2012 ); (ii) polar nanoregions
Akbarzadeh2012 ; Prosandeev2013 ; and (iii) the unusual dielectric relaxation
Wang2016 —which is consistent with experimental findings for BZT systems
Maiti2008 . The total internal energy $E_{\textrm{int}}$ of the
$H_{\textrm{eff}}$ consists of two main terms:
$E_{\textrm{int}}(\\{\mathrm{\mathbf{u}}_{i}\\},\thinspace\\{\mathbf{v}_{i}\\},\thinspace\eta_{H},\thinspace\\{\sigma_{j}\\})=E_{\textrm{ave}}(\\{\mathrm{\mathbf{u}}_{i}\\},\thinspace\\{\mathbf{v}_{i}\\},\thinspace\eta_{H})+E_{\textrm{$\textrm{loc}$}}(\\{\mathrm{\mathbf{u}}_{i}\\},\thinspace\\{\mathbf{v}_{i}\\},\thinspace\\{\sigma_{j}\\})$,
where $\\{\mathrm{\mathbf{u}}_{i}\\}$ is the local soft mode in unit cell $i$
(which is proportional to the local electric dipole moment centered on Zr or
Ti atoms), $\\{\mathbf{v}_{i}\\}$ are Ba-centered local displacements related
to the inhomogeneous strain inside each cell, $\eta_{H}$ represents the
homogeneous strain tensor, and $\\{\sigma_{j}\\}$ characterizes the $B$
sublattice atomic configuration in the BZT solid solutions. Actually,
$\sigma_{j}$ $=$ $+$1 or $-$1 corresponds to the distribution of Zr or Ti ion
located at the $j$ site of the $B$ sublattice, respectively. The first
energetic term of $E_{\textrm{ave}}$ is composed of five energetic parts: (i)
the local soft mode self-energy; (ii) the long-range dipole-dipole
interaction; (iii) the short-range interactions between local soft modes; (iv)
the elastic energy; and (v) the interaction between the local soft modes and
strains Zhong1995 . The second term, $E_{\textrm{loc}}$, represents how the
distribution of $B$ sites (Zr and Ti ions) affects the energetics involving
the local soft modes $\mathbf{u}_{i}$ and the local strain variables, which
depends on the $\\{\sigma_{j}\\}$ atomic configuration distribution
Akbarzadeh2012 ; Prosandeev2013 . In order to mimic the effect under an
applied dc electric field, an energy given by minus the dot product between
polarization and applied electric field needs to be added to
$E_{\textrm{int}}$. Note that we numerically find that the simulated electric
field is larger than the corresponding experimental one by a factor of 100 in
BZT when comparing the calculated polarization $P$ with the experimental one
for a Ba(Zr0.5Ti0.5)O3 thin film at room temperature Instan2017 . Such
discrepancy is typical for atomistic effective Hamiltonian simulations Xu2017
; Jiang2018  and direct density functional theory (DFT) calculations Lu2019 ;
Chen2019 ; Jiang2020 ; Jiang2021 , and is likely due to the fact that
structural defects are not considered in these simulations. To address such
discrepancy, we presently divide our theoretical field by a factor of 100 in
the results to be reported and discussed below. Note also that we chose here a
(renormalized) maximum applied electric field of $E_{\textrm{max}}$ $=$
$3.0\times 10^{8}$ V/m, which has been experimentally achieved in
Ba(ZrxTi1-x)O3 thin films Instan2017 .

Moreover, we employ the effective Hamiltonian scheme within Monte Carlo (MC)
simulations on $12\times 12\times 12$ supercells (8,640 atoms) with periodic
boundary conditions and the distribution of Zr and Ti ions is chosen randomly
over the $B$ sublattice. In the case of the films grown along the [001]
pseudocubic direction, epitaxial strains are associated with the freezing of
some components of the homogeneous strain tensors, namely (in Voigt notation)
$\eta_{6}$ $=$ 0 and $\eta_{1}$ $=$ $\eta_{2}$ $=$ ($a_{\textrm{sub}}$$-$
$a_{\textrm{eq}}$)$/$$a_{\textrm{sub}}$, where $a_{\textrm{sub}}$ and
$a_{\textrm{eq}}$ are the lattice constants of the substrate and BZT bulk at
130 K, respectively Prosandeev2013_film . We also limited the simulations for
strains ranging between $-$3% and $+$3%, which is a physically reasonable
range. Note that the films are periodic along the three Cartesian direction
and only the strain is thus considered here when modeling epitaxial films
(i.e., no surface or thickness effect are taken into account).

##  III Results and discussion

###  III.1 Energy storage in BZT bulk

Figures 1(a)-1(c) show the $P$-$E$ curves at a selected room temperature of
300 K for electric field applied along the pseudocubic [001], [110] and [111]
directions, respectively, in BZT bulk. For all these directions, the resulting
polarization aligns along the field. As depicted in Figs. 1(a)-1(c), the
charging and discharging processes of the $P$-$E$ curves are completely
reversible, which implies that the energy efficiency is 100% (the charging and
discharging cycles correspond to electric field increasing from zero to
$E_{\textrm{max}}$ $=$ $3.0\times 10^{8}$ V/m and then decreasing back to zero
field, respectively). Note that the ideal 100% efficiency has also been
predicted in epitaxial AlN/ScN superlattices Jiang2021  as a result of a
field-induced second-order transition towards a ferroelectric state. In the
case of BZT, measurements also found a large value for this efficiency, namely
of 82.46$\pm$1.0% at an electric field $\simeq$3.0$\times$108 V/m Instan2017
(the deviation from the ideal value of 100% may also arise from the presence
of defects in the grown sample).

Figure 1: (a)-(c) $P$-$E$ hysteresis curves at 300 K for electric field
applied along the pseudocubic [001], [110] and [111] directions, respectively,
in BZT bulk. The insets show the dipolar configurations in a given ($y$, $z$)
or ($x$, $y$) plane at 300 K for the magnitude of the field equal to
$\simeq$$0.1\times 10^{8}$ V/m with three different field’s applied
directions. (d)-(f) Electric field versus polarization at 300 K for electric
field applied along the pseudocubic [001], [110] and [111] directions,
respectively. The green areas represent the energy densities and the solid red
lines represent the fit of the MC data by the Landau model.

Moreover, the insets of Figs. 1(a)-1(c) display the local dipole
configurations in a given ($y$, $z$) or ($x$, $y$) plane at 300 K for these
three different applied field directions with a magnitude of
$\simeq$$0.1\times 10^{8}$ V/m. Recalling that the $H_{\textrm{eff}}$
predicted that, under zero field and for temperatures below the Burns
temperature ($T_{\textrm{b}}$ $\simeq$ 450 K) Akbarzadeh2012 , BZT possesses
different polar nanoregions for which the dipoles align along different
$\left\langle 111\right\rangle$ pseudocubic directions (hence resulting in an
overall vanishing polarization), these insets demonstrate that the application
of electric field along a given direction forces dipoles to rotate towards the
field’s direction (hence giving rise to a finite polarization along that
latter direction).

Furthermore, Figs. 1(d)-1(f) display the electric field applied along these
[001], [110] and [111] pseudocubic directions as a function of polarization
for temperature at 300 K, which allows us to extract the energy density since
it is simply the green area shown in these figures.

Doing that for all considered temperatures (up to 700 K) therefore allows us
to compute the energy densities as a function of temperature for electric
field applied along the [001], [110] and [111] pseudocubic directions, which
are shown in Fig. 2(a). It is interesting to realize that (1) the energy
densities linearly increase with temperature for the three different
considered field’s directions; and (2) we predict ultrahigh energy densities
and values varying between 147 and 155 J/cm3, which agrees very well with
experimental reports in Ba(ZrxTi1-x)O3 thin films Instan2017 .

Figure 2: (a) Energy density obtained from MC data as a function of
temperature for electric field applied along the [001], [110] and [111]
directions, with a maximal electric field being equal to $3.0\times 10^{8}$
V/m, in BZT bulk. (b)-(d) Total and decomposed energy densities obtained from
Eq. (3) as a function of temperature at $E_{\textrm{max}}$ $=$ $3.0\times
10^{8}$ V/m for electric field applied along the [001], [110] and [111]
directions, respectively. Stars also display the MC data of the total energy
densities again in Panels (b)-(d) for comparison.

In order to understand the origin of these energy density features, let us
take advantage of the simple Landau-type free energy model developed in Ref.
Jiang2021 :

$F=\frac{1}{2}aP^{2}+\frac{1}{4}bP^{4}-EP,$ (1)

where $a$ and $b$ are quadratic and quartic coefficients, respectively.

At equilibrium, one must have $\frac{\partial F}{\partial P}=0$, which thus
leads to:

$E=aP+bP^{3}.$ (2)

Interestingly, the electric field versus polarization ($E$-$P$) MC data for
all considered temperatures can indeed be nicely fitted by Eq. (2), which
shows that such equation is valid but also allows to extract the $a$ and $b$
parameters for each temperature and considered field’s direction. These latter
$a$ and $b$ coefficients are shown in Figs. 3(a)-3(c) as a function of
temperature for fields up to the maximum field $E_{\textrm{max}}$ $=$
$3.0\times 10^{8}$ V/m applied along the [001], [110] and [111] pseudocubic
directions, respectively. These $a$ and $b$ coefficients are important for
energy storage since they are involved in the expression of the energy
density, according to the Landau model Jiang2021 :

$U=\int_{0}^{P_{\textrm{max}}}(aP+bP^{3})dP=\frac{1}{2}aP_{\textrm{max}}^{2}+\frac{1}{4}bP_{\textrm{max}}^{4},$
(3)

where $P_{\textrm{max}}$ is the polarization at $E_{\textrm{max}}$. Note that,
practically, one can determine $P_{\textrm{max}}$ via two different ways, that
are directly from the MC data or via Eq. (2) at the field of
$E_{\textrm{max}}$ since $a$ and $b$ parameters are now known. As shown in
Fig. 3(d), these two methods give nearly identical results.

Figure 3: (a)-(c) Temperature dependence of fitting parameters $a$ and $b$
(see text) for electric field applied along the [001], [110] and [111]
directions, respectively, with a maximal applied electric field of
$E_{\textrm{max}}$ $=$ $3.0\times 10^{8}$ V/m, in BZT bulk. (d)
$P_{\textrm{max}}$ obtained from MC data (filled symbols) and Eq. (2) (open
symbols) as a function of temperature at $E_{\textrm{max}}$ $=$ $3.0\times
10^{8}$ V/m for electric field applied along the [001], [110] and [111]
directions, in BZT bulk.

Equation (3) therefore tells us that only three quantities completely govern
the behaviors and values of the energy density, namely $a$, $b$ and
$P_{\textrm{max}}$. It is thus necessary to comment on their behaviors with
temperature and field’s direction that are reported in Fig. 3. The fitting
parameter $a$ linearly increases with temperature for the three different
field’s directions—as expected from usual Landau theory of ferroelectrics
stating that this coefficient should be equal to $a_{0}(T-T_{c})$, where
$a_{0}$ is a positive constant and $T_{c}$ is a critical temperature (that can
be negative for relaxor ferroelectrics) Chandra2007 . It varies from from
0.139$\times$108 to 0.174$\times$108 V m/C for the [001] direction, from
0.177$\times$108 to 0.216$\times$108 V m/C for [110], and from
0.203$\times$108 to 0.247$\times$108 V m/C for [111], respectively, for
temperatures ranging between 10 K and 700 K. In contrast, the $b$ parameter
only slightly linearly decreases with temperature (its value concomitantly
ranges from 0.455$\times$108 to 0.447$\times$108 V m5/C3 for [001], from
0.454$\times$108 to 0.445$\times$108 V m5/C3 for [110], and from
0.441$\times$108 to 0.428$\times$108 V m5/C3 for [111], respectively).
Finally, $P_{\textrm{max}}$ basically only very slightly linearly decreases
with temperature from 1.820 to 1.816 C/m${}^{\textrm{2}}$ for fields applied
along [001], from 1.809 to 1.805 C/m${}^{\textrm{2}}$ for [110], and from
1.817 to 1.813 C/m${}^{\textrm{2}}$ for [111], respectively, with temperatures
varying from 10 K to 700 K. Its rather large value indicates that BZT is
easily polarizable. The parameter that is the most sensitive to both
temperature and field’s direction is therefore the $a$ coefficient. Its
temperature behavior (linear increase and always positive values) basically
indicates that increasing the temperature makes BZT going further away from a
ferroelectric state at zero field for any direction of the field. Its
dependency on field’s direction at any temperature (smaller positive values
$a$ for the [001] direction and larger values for [111]) reveals that it is
easier to induce a ferroelectric state when applying an electric field along
[001] than [110] and then [111]—likely because inducing a ferroelectric state
with a polarization along [111] requires the polar nanoregions existing at
zero field and having electric dipoles along [$\bar{1}$$\bar{1}$$\bar{1}$] to
completely revert their polarization rather than simply rotate towards an
intermediate direction, such as [001].

The behaviors of $a$, $b$ and $P_{\textrm{max}}$ allow us to understand the
results of the energy density in Fig. 2(a) since Eq. (3) indicates that such
energy density can be decomposed in two terms, that are
$\frac{1}{2}aP_{\textrm{max}}^{2}$ and $\frac{1}{4}bP_{\textrm{max}}^{4}$, and
which are shown in Figs. 2(b)-2(d) for the three different field’s direction
with $E_{\textrm{max}}$ $=$ $3.0\times 10^{8}$ V/m, along with the (total)
energy densities directly obtained from the MC data. One can clearly see that,
for any considered temperature, Eq. (3) and the MC energy densities provide
nearly identical results. Since $b$ and $P_{\textrm{max}}$ are in first
approximation independent of both the temperature and the field’s direction,
the dependencies of the total energy density on temperature and
crystallographic direction of the electric field basically arise from the
aforementioned corresponding dependencies of the $a$ parameter. Note that the
contribution of $\frac{1}{2}aP_{\textrm{max}}^{2}$
($\frac{1}{4}bP_{\textrm{max}}^{4}$) to the total energy density at 300 K is
17% (83%), 21% (79%), and 23% (77%) for electric field applied along the
[001], [110] and [111] directions, respectively. These numbers, as well as
Fig. 2, Fig. 3 and Eq. (3), therefore tell us that having a large energy
density accompanied by a large efficiency can be accomplished by (1) having
large positive $a$ and $b$ coefficients, which characterize systems that can
not be energetically too close to a ferroelectric state and that will undergo
a second-order field-induced transition to a ferroelectric state at large
fields; while (2) having a large $P_{\textrm{max}}$ at feasible electric
fields, which indicates that the system is easily polarizable at these fields
and which thus also implies that $a$ and $b$ can not be too large. Note that
conditions (1) and (2) are relevant to analyze and understand the ultrahigh
energy storage in initially non-polar AlN/ScN superlattices Jiang2021  and
superparaelectric relaxor ferroelectrics Pan2021 .

###  III.2 Energy storage in (001) BZT films

Let us now present the energy storage results for (001) BZT films as a
function of epitaxial strain for the selected temperature of 300 K. Figures
4(a)-4(c) respectively show the $P$-$E$ curves at 300 K and zero strain for
field applied along (i) the pseudocubic [001] that results in a polarization
lying along that out-of-plane direction; (ii) the [110] direction that yields
a polarization aligned along that in-plane direction; and (iii) along the
[111] direction that induces a polarization being along $[uuv]$ directions,
that is $P_{x}$$=$ $P_{y}$ $\neq$ $P_{z}$, due to the freezing of $\eta_{1}$
$=$ $\eta_{2}$ strain components while $\eta_{3}$ can relax. Note that the
direction of these polarization in the three cases is also consistent with the
insets of Figs. 4(d)-4(f) showing the dipole configurations in a given ($y$,
$z$) or ($x$, $y$) plane at zero strain and 300 K. Note also that,
consequently, we show three types of $P$-$E$ data for fields applied along the
[111] direction in Fig. 4(c), that is for the in-plane component of the
polarization $P_{\textrm{in}}$ (which is along the [110] direction), the out-
of-plane component of the polarization $P_{\textrm{out}}$ (that is along
[001]) and the total polarization $P_{\textrm{tot}}$ (that is given by
$P_{\textrm{tot}}$ $=$ $\sqrt{P_{x}^{2}+P_{y}^{2}+P_{z}^{2}}$).

Figure 4: Same as Fig. 1 but for (001) BZT films at 0% strain.

Moreover, Figs. 4(d)-4(f) show the corresponding $E$-$P$ data at zero strain
and 300 K for fields applied along these three different directions up to
$E_{\textrm{max}}$ $=$ $3.0\times 10^{8}$ V/m, which, once again, allows the
energy density to be extracted via the computation of areas similar to the
green one of Figs. 1(d)-1(f). Such types of calculations are then performed
for all considered strains at 300 K, which yields the results for energy
density reported in Fig. 5 for the different directions of the field. Note
that we continue to distinguish between in-plane versus out-of-plane
components of the polarization in case of a field applied along [111], with
this distinction resulting in the wording of [111]${}_{\textrm{in}}$ versus
[111]${}_{\textrm{out}}$ in Fig. 4(f) and Fig. 5 as well as in the text.

One can first realize that for all considered strains, the energy density is
still large in magnitude in the films (typically larger than 100 J/cm3), while
being smaller than that of BZT bulk at room temperature. For instance, the
energy densities of bulk BZT at 300 K are 148.7 J/cm3 when the field is
applied along [001], 150.6 J/cm3 for [110], and 153.3 J/cm3 for [111],
respectively. This slight decrease when going from bulk to films originates
from the fact that, for each considered field and at a fixed temperature, the
in-plane lattice constants are constant in the epitaxial films while they can
vary when changing the fields in the bulk—which can thus give rise to larger
polarizations in bulk than in films. Other striking features of Fig. 5, are
that the energy densities resulting from the application of a field along
[001] as well as the one associated with the out-of-plane component of
polarization under a [111] field, that is [111]${}_{\textrm{out}}$, both
linearly increase with strain ranging between $-$3% and $+$3%. In contrast,
the energy densities when the field is applied along [110] and the other one
linked with in-plane component of polarization, namely
[111]${}_{\textrm{in}}$, both linearly decrease with such strain. Furthermore,
the largest energy density is found for our maximal considered tensile strain
(that is $+$3%) in case of a field applied along [001].

Figure 5: Energy density obtained from MC data versus strain at 300 K and
$E_{\textrm{max}}$ $=$ $3.0\times 10^{8}$ V/m for fields applied along the
[001], [110] and [111] directions, respectively, in (001) BZT films. The
distinction between [111]${}_{\textrm{in}}$ versus [111]${}_{\textrm{out}}$ is
explained in the text, in case of fields applied along [111].

To understand the energy density results in Fig. 5 for (001) BZT films, we
also decided to use Eq. (2) (which is once again found to fit well the MC
data) and Eq. (3). Consequently, Fig. 6 shows the $a$ and $b$ fitting
parameters while Fig. 7 displays the resulting $P_{\textrm{max}}$, for these
films. Figure 6(a) and Figure 7 indicate that, when the field is applied along
the [001] direction, the (i) fitting parameter $a$ is positive and linearly
increases when the strain increases from $-$3% to $+$3% ($a$ varies from
0.02$\times$108 to 0.35$\times$108 V m/C); (ii) the $b$ coefficient is
basically a constant with strain (it varies from 0.70$\times$108 to
0.69$\times$108 V m5/C3); and (iii) $P_{\textrm{max}}$ linearly decreases with
strain from 1.62 to 1.53 C/m${}^{\textrm{2}}$. In contrast, when the field is
along [110], Fig. 6(b) and Fig. 7 reveal that (iv) $a$ linearly decreases with
strain (from 0.39$\times$108 to $-$0.07$\times$108 V m/C, therefore becoming
slightly negative at large tensile strains); (v) $P_{\textrm{max}}$ linearly
increases with strain (from 1.34 to 1.45 C/m${}^{\textrm{2}}$); (vi) while $b$
continues to be basically constant with strain (it only changes from
1.03$\times$108 to 1.02$\times$108 V m5/C3). Items (i)-(vi) can be simply
understood by realizing that increasing strain from compressive to tensile in
epitaxial (001) films is known to progressively disfavor a ferroelectric state
with an out-of-plane polarization in favor of a ferroelectric state with an
in-plane polarization Dupe2011 ; Yang2012 ; Chen2015 . For the same reasons
and as shown in Fig. 6(c), Fig. 6(d) and Fig. 7, the $a$ parameter and
$P_{\textrm{max}}$ associated with the out-of-plane (respectively, in-plane)
components for fields applied along [111] have the same behavior with strain
as those for the field lying along [001] (respectively, [110]). Note also that
the $b$ parameter continues to be basically independent of strain in the case
of fields applied [111], as well—which reveals that BZT can adopt second-order
phase transition when under strain and/or field Prosandeev2013 .

Figure 6: Panels (a) and (b): Same as Figs. 3(a) and 3(b) but as a function of
strain at 300 K in BZT films. Panels (c) and (d) show the in-plane and out-of-
plane fitting parameters $a$ and $b$ when field is applied along the [111]
direction, respectively.  Figure 7: Same as Fig. 3(d) but as a function of
strain at 300 K in BZT films.

Let us now pay attention to the two terms of Eq. (3) that sump up to be the
total energy density. They are shown in Fig. 8, using the fitting parameters
($a$ and $b$) of Fig. 6 and $P_{\textrm{max}}$ of Fig. 7. Due to the
aforementioned behavior of $a$ that grows faster than $P_{\textrm{max}}$
decreases with strain, the first contribution,
$\frac{1}{2}aP_{\textrm{max}}^{2}$, increases with strain (from 2.7 for $-$3%
to 41.1 J/cm${}^{\textrm{3}}$ for $+$3%), when the field is applied along the
[001] direction—as displayed in Fig. 8(a). In contrast, the second
contribution, $\frac{1}{4}bP_{\textrm{max}}^{4}$, decreases with strain from
119.8 to 93.6 J/cm${}^{\textrm{3}}$) when the field is also along this [001]
direction, as a result of the concomitant decrease of $P_{\textrm{max}}$ while
$b$ is basically constant with strain. Note that the change of value in
$\frac{1}{4}bP_{\textrm{max}}^{4}$ is smaller than
$\frac{1}{2}aP_{\textrm{max}}^{2}$ for this field’s direction, once again
because of the fast increase of $a$ with strain reflecting the desire of the
system to be energetically far from a polarized state with a polarization
along [001] when enhancing the strain from compressive to tensile values.
Consequently, the total energy density increases with strain.

The behaviors are opposite when the field is applied along [110] because BZT
films become much closer in energy to adopt a ferroelectric state with an in-
plane [110] polarization direction as the strain is enhanced. Consequently,
$a$ strongly decreases with strain while $P_{\textrm{max}}$ is enhanced but at
a smaller extent. As a result, $\frac{1}{2}aP_{\textrm{max}}^{2}$ decreases
with strain (values varying between 34.9 and $-$7.1 J/cm3) faster than
$\frac{1}{4}bP_{\textrm{max}}^{4}$ increases (from 83.5 to 112.4
J/cm${}^{\textrm{3}}$) for field applied along the [110] direction—as shown in
Fig. 8(b). The resulting total energy density thus decreases with strain.

Regarding the energy density for fields applied along the [111] direction,
Fig. 8(c) [respectively, Fig. 8(d)] that is related to the in-plane
(respectively, out-of-plane) component of the polarization shows that the
behaviors of total and decomposed energy densities are very similar to Fig.
8(b) that corresponds the field applied along [110] (respectively to Fig. 8(a)
that corresponds the field applied along [001]) for the same energetic
reasons, i.e., going from compressive strain to tensile strain favors the
formation of in-plane polarization while disfavoring ferroelectric states with
out-of-plane polarization.

Figure 8: Panels (a)-(d): Total and decomposed energy densities versus strain
for field applied along [001] [Panel (a)], [110] [Panel (b)] and [111] [Panels
(c) and (d) for in-plane versus out-of-plane components of the polarization],
at a maximal applied electric field equal to $3.0\times 10^{8}$ V/m and 300 K
in BZT films.

##  IV Summary

In summary, an atomistic effective Hamiltonian scheme combined with Monte-
Carlo simulations was used to investigate the energy storage in bulk and
epitaxial (001) films made of BZT. We find that these BZT systems can exhibit
ultrahigh energy densities and an ideal efficiency of 100%. These energy
storage results are then interpreted via a simple phenomenological model that
reproduces these MC data. More precisely, energy density can be decomposed in
two terms: the first term being the product of the fitting $a$ parameter and
$P_{\textrm{max}}^{2}$, and the second term being the product between the $b$
parameter and $P_{\textrm{max}}^{4}$. The behavior of $a$, $b$ and
$P_{\textrm{max}}$ lead to a competition between these two terms (that can be
understood in terms of energetics) that eventually causes the temperature-
dependency, field’s direction dependency and strain-dependency of the total
energy density. The proposed phenomenological model can be easily employed for
nonlinear dielectrics with large energy density. We thus hope that the present
article deepens the fields of energy storage in relaxor ferroelectrics and
other nonlinear dielectrics.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of China
(Grant No. 11804138), Shandong Provincial Natural Science Foundation (Grant
No. ZR2019QA008), China Postdoctoral Science Foundation (Grants No.
2020T130120 and No. 2018M641905), and “Young Talent Support Plan” of Xi’an
Jiaotong University (Grant No. WL6J004). S.P and L.B. acknowledge the Office
of Naval Research for the support under Grants No. N00014-17-1-2818 and No.
N00014-21-1-2086. L.B. also acknowledges ARO Grant No. W911NF-21-1-0113 and
DARPA Grant No. HR0011-15-2-0038 (MATRIX program). The HPC Platform of Xi’an
Jiaotong University is also acknowledged.

## References

  * (1) B. Chu, X. Zhou, K. Ren, B. Neese, M. Lin, Q. Wang, F. Bauer, and Q. M. Zhang, Science 313, 334 (2006).
  * (2) Q. Li, L. Chen, M. R. Gadinski, S. Zhang, G. Zhang, H. U. Li, E. Iagodkine, A. Haque, L.-Q. Chen, T. N. Jackson, and Q. Wang, Nature (London) 523, 576 (2015).
  * (3) B. Peng, Q. Zhang, X. Li, T. Sun, H. Fan, S. Ke, M. Ye, Y. Wang, W. Lu, H. Niu, J. F. Scott, X. Zeng, and H. Huang, Adv. Electron. Mater. 1, 1500052 (2015).
  * (4) Prateek, V. K. Thakur, and R. K. Gupta, Chem. Rev. 116, 4260 (2016).
  * (5) H. Palneedi, M. Peddigari, G.-T. Hwang, D.-Y. Jeong, and J. Ryu, Adv. Funct. Mater. 28, 1803665 (2018).
  * (6) L. Yang, X. Kong, F. Li, H. Hao, Z. Cheng, H. Liu, J.-F. Li, and S. Zhang, Prog. Mater. Sci. 102, 72 (2019).
  * (7) H. Pan, F. Li, Y. Liu, Q. Zhang, M. Wang, S. Lan, Y. Zheng, J. Ma, L. Gu, Y. Shen, P. Yu, S. Zhang, L.-Q. Chen, Y.-H. Lin, and C.-W. Nan, Science 365, 578 (2019).
  * (8) L. E. Cross, Ferroelectrics 151, 305 (1994).
  * (9) A. A. Bokov and Z.-G. Ye, J. Mater. Sci. 41, 31 (2006).
  * (10) V. V. Shvartsman and D. C. Lupascu, J. Am. Ceram. Soc. 95, 1 (2012).
  * (11) A. R. Akbarzadeh, S. Prosandeev, E. J. Walter, A. Al-Barakaty, and L. Bellaiche, Phys. Rev. Lett. 108, 257601 (2012).
  * (12) A. Al-Barakaty, S. Prosandeev, D. Wang, B. Dkhil, and L. Bellaiche, Phys. Rev. B 91, 214117 (2015).
  * (13) A. A. Instan, S. P. Pavunny, M. K. Bhattarai, and R. S. Katiyar, Appl. Phys. Lett. 111, 142903 (2017).
  * (14) J. Kim, S. Saremi, M. Acharya, G. Velarde, E. Parsonnet, P. Donahue, A. Qualls, D. Garcia, and L. W. Martin, Science 369, 81 (2020).
  * (15) H. Pan, S. Lan, S. Xu, Q. Zhang, H. Yao, Y. Liu, F. Meng, E.-J. Guo, L. Gu, D. Yi, X. R. Wang, H. Huang, J. L. MacManus-Driscoll, L.-Q. Chen, K.-J. Jin, C.-W. Nan, and Y.-H. Lin, Science 374, 100 (2021).
  * (16) L. E. Cross, Ferroelectrics 76, 241 (1987).
  * (17) A. E. Glazounov, A. J. Bell, and A. K. Tagantsev, J. Phys.: Condens. Matter 7, 4145 (1995).
  * (18) S. Prosandeev, D. Wang, A. R. Akbarzadeh, B. Dkhil, and L. Bellaiche, Phys. Rev. Lett. 110, 207601 (2013).
  * (19) S. Prosandeev, D. Wang, and L. Bellaiche, Phys. Rev. Lett. 111, 247602 (2013).
  * (20) Z. Jiang, S. Prokhorenko, S. Prosandeev, Y. Nahas, D. Wang, J. Íñiguez, E. Defay, and L. Bellaiche, Phys. Rev. B 96, 014114 (2017).
  * (21) G. Burns and F. H. Dacol, Phys. Rev. B 28, 2527 (1983).
  * (22) B. Dkhil, P. Gemeiner, A. Al-Barakaty, L. Bellaiche, E. Dul’kin, E. Mojaev, and M. Roth, Phys. Rev. B 80, 064103 (2009).
  * (23) O. Svitelskiy, D. La-Orauttapong, J. Toulouse, W. Chen, and Z.-G. Ye, Phys. Rev. B 72, 172106 (2005).
  * (24) D. Wang, A. A. Bokov, Z.-G. Ye, J. Hlinka, and L. Bellaiche, Nat. Commun. 7, 11014 (2016).
  * (25) T. Maiti, R. Gu, and A. S. Bhalla, J. Am. Ceram. Soc. 91, 1769 (2008).
  * (26) W. Zhong, D. Vanderbilt and K. M. Rabe, Phys. Rev. B 52, 6301 (1995).
  * (27) B. Xu, J. Íñiguez, and L. Bellaiche, Nat. Commun. 8, 15682 (2017).
  * (28) Z. Jiang, Y. Nahas, S. Prokhorenko, S. Prosandeev, D. Wang, J. Íñiguez, and L. Bellaiche, Phys. Rev. B 97, 104110 (2018).
  * (29) J. Lu, G. Chen, W. Luo, J. Íñiguez, L. Bellaiche, and H. Xiang, Phys. Rev. Lett. 122, 227601 (2019).
  * (30) L. Chen, C. Xu, H. Tian, H. Xiang, J. Íñiguez, Y. Yang, and L. Bellaiche, Phys. Rev. Lett. 122, 247701 (2019).
  * (31) Z. Jiang, C. Paillard, H. Xiang, and L. Bellaiche, Phys. Rev. Lett. 125, 017401 (2020).
  * (32) Z. Jiang, B. Xu, H. Xiang, and L. Bellaiche, Phys. Rev. Mater. 5, L072401 (2021).
  * (33) P. Chandra and P. B. Littlewood, A Landau Primer for Ferroelectrics, in Physics of Ferroelectrics: A Modern Perspective (Springer, Berlin, Heidelberg, 2007), pp. 69–116.
  * (34) B. Dupé, S. Prosandeev, G. Geneste, B. Dkhil, and L. Bellaiche, Phys. Rev. Lett. 106, 237601 (2011).
  * (35) Y. Yang, W. Ren, M. Stengel, X. H. Yan, and L. Bellaiche, Phys. Rev. Lett. 109, 057602 (2012).
  * (36) L. Chen, Y. Yang, Z. Gui, D. Sando, M. Bibes, X. K. Meng, and L. Bellaiche, Phys. Rev. Lett. 115, 267602 (2015).


# Uncertainty-Aware Deep Video Compression with Ensembles

Wufei Ma, , Jiahao Li, Bin Li, , Yan Lu  W. Ma is with Johns Hopkins
University, E-mail<EMAIL_ADDRESS>This work was done when W. Ma was a full-
time intern with Microsoft Research Asia. J. Li, B. Li, and Y. Lu are with
Microsoft Research Asia, Beijing, China, E-mail: li.jiahao, libin,
<EMAIL_ADDRESS>
###### Abstract

Deep learning-based video compression is a challenging task, and many previous
state-of-the-art learning-based video codecs use optical flows to exploit the
temporal correlation between successive frames and then compress the residual
error. Although these two-stage models are end-to-end optimized, the epistemic
uncertainty in the motion estimation and the aleatoric uncertainty from the
quantization operation lead to errors in the intermediate representations and
introduce artifacts in the reconstructed frames. This inherent flaw limits the
potential for higher bit rate savings. To address this issue, we propose an
uncertainty-aware video compression model that can effectively capture the
predictive uncertainty with deep ensembles. Additionally, we introduce an
ensemble-aware loss to encourage the diversity among ensemble members and
investigate the benefits of incorporating adversarial training in the video
compression task. Experimental results on 1080p sequences show that our model
can effectively save bits by more than 20% compared to DVC Pro.

###### Index Terms:

Deep video compression, uncertainty, prediction, motion estimation

##  I Introduction

Video contents are reported to account for 82% percent of all consumer
Internet traffic by 2021, and they are proliferating with an increasing demand
for high-resolution videos (e.g., 4K movies) and live streaming services [1].
Therefore, we must improve the video compression performance to transmit video
with a higher quality given limited Internet bandwidth. In recent years, there
has been a surge of deep learning-based video compression models [2, 3, 4, 5]
and some of them have achieved comparable or even better performance than
previous traditional video codecs, such as x264 and x265 [6].

Although previous deep learning-based video codecs have achieved improved
performance on many challenging datasets, most state-of-the-art models
estimate deterministic predictions for intermediate representations, such as
optical flows and residuals. These models fail to represent the aleatoric
uncertainty inherent in the model inputs or the epistemic uncertainty in the
model parameters and would blindly assume the predictions to be accurate,
which is not always the case [7, 8]. In terms of video compression, such
models produce deterministic motion vectors (or optical flows) and residuals
for each pixel location, ignoring the fact that optical flows may not be
estimated accurately in occluded regions and around object boundaries, and the
quantization operation before lossless entropy coding also introduces
additional noises to the inputs of the decoders. Underlying errors in such
overconfident intermediate predictions are propagated to later stages of the
P-frame model and even to subsequent frames for models built on temporal
correlation, leading to suboptimal performance of the compression system.

Predictive uncertainty is crucial for us to understand how confident the model
is about the predictions, especially for out-of-distribution data. However,
most neural networks do not offer such information and tend to produce
overconfident predictions [9, 10]. Bayesian neural networks [11, 12] are
widely used to quantify predictive uncertainty but lack practicality due to
significantly increased computation complexity and do not scale well to high-
dimensional data. [13] proposed Monte Carlo dropout that performs test-time
dropout. It is simple to implement but unsuitable for deep learning-based
compression, since it requires multiple decoding-time inferences and yields
nondeterministic outputs.

In terms of deep learning-based video compression, two non-Bayesian approaches
are considered to represent the predictive uncertainty: (1) modeling the
uncertainty explicitly by regressing the empirical variance of the model
outputs [14]; and (2) using ensembles for predictive uncertainty estimation
[10]. Scale-space flow [4] took the first approach and proposed to regress a
scale field besides the standard 2D flow field, representing the variance
associated with each predicted MV (motion vector). Gaussian blurring is then
applied to the reference frame, and the scale parameter is used to control the
size of the Gaussian kernel. Although this approach has been shown to be
effective, regressed scales are unreliable for out-of-distribution data and
are often misinterpreted as predictive uncertainty [9].

In this work, we consider the second approach and represent the underlying
uncertainty with deep ensembles. Instead of producing a deterministic
prediction, ensemble methods perform model combination and reflect the
uncertainty of out-of-distribution data. Our ensemble-based decoding module
generates an ensemble of intermediate outputs, such as motion vectors and
residuals, and implicitly represents the predictive uncertainty with the
variance of the Gaussian mixture prediction. This uncertainty is then
propagated to later stages, and all modules in our framework are optimized in
an end-to-end fashion. Moreover, unlike previous works on whole model-level
ensembles, our approach ensembles the partial intermediate layers of the
decoding module and achieves improved performance with limited overhead.

To further improve the performance of our uncertainty-aware video compression
model, we propose an ensemble-aware loss to encourage diversity between
different branches and incorporate an adversarial training strategy, fast
gradient sign method (FGSM) [15], to effectively learn a smooth latent
representation. Our experiments show that our model can achieve a bitrate
saving of more than 20% on 1080p sequences compared to DVC Pro [3].
Visualizations of the predictive uncertainty captured by our model support our
claims and demonstrate the effectiveness of our approach.

The contributions of this work are summarized as follows:

  * •

We identify the underlying uncertainty of intermediate representations as a
key limitation of residual-based video compression models and propose an
ensemble-based decoder to effectively capture the predictive uncertainty.

  * •

We design a novel ensemble-aware loss to encourage the diversity between
ensemble members and better capture the predictive uncertainty. We also show
that fast gradient sign method can benefit deep learning-based video
compression by learning a smooth intermediate representation.

  * •

Experiments show that our model outperforms previous state-of-the-art models
such as DVC Pro [3] and scale-space flow [4], and our approach can be widely
applied to optical flow-based video codecs with negligible complexity
increase.

##  II Related Work

Video compression. Previous learning-based video compression methods can be
categorized into two groups: (i) one-stage models, such as methods based on 3D
autoencoders [16, 17]; and (ii) two-stage models, which are adopted by most
previous state-of-the-art methods, consist of predicted frame generation and
residual coding. [18] proposed an end-to-end trainable video codec, DVC, that
utilizes an optical-flow network [19] for motion compensation and then
compresses the residuals. DVC Pro [3] improves the compression performance by
introducing refinement modules and auto-regressive entropy models. [20]
proposed to learn robust spatio-temporal representations from coding
information and to reveal double compression. In order to obtain better motion
vectors for motion compensation, [21] proposed GPU-based hierarchical motion
estimation and [4] proposed scale-space flow to blur intermediate
reconstructions when motion vectors are not estimated well. [22] designed a
cross-resolution synthesis module to pursue better compression efficiency.
Moreover, [23] exploited the temporal masking effect for better visual
qualities.

Model uncertainty. Predictive uncertainty can be grouped into aleatoric
uncertainty and epistemic uncertainty [7]. Aleatoric uncertainty captures the
noises inherent in the observations and cannot be explained away with more
data, while epistemic uncertainty accounts for uncertainty in the model
structure or parameters and can be reduced with more training data. Bayesian
neural networks [11, 12] is a widely used approach for modeling predictive
uncertainty that extends the traditional neural networks by learning a
posterior distribution of model parameters from the observed data. Various
non-Bayesian approaches have also been proposed, such as utilizing the
probabilities of softmax distributions [24] and Specialists+1 Ensemble for
representing the predictive uncertainty for adversarial samples [25]. [13]
proposed Monte Carlo dropout by performing multiple inferences with dropout at
test time. [10] proposed to use an ensemble of neural networks for quantifying
predictive uncertainty.

Deep Ensembles. The neural networks community has been investigating ensembles
of deep networks since the early 1990s [26, 27, 28]. [29] proved the bias-
variance trade-off for ensemble models, which suggested the importance of the
diversity among ensemble members. [30] investigated several training
strategies to train an ensemble and proposed ensemble-aware oracle loss to
encourage diversity. GoogLeNet [31], one of the best-performing models on
ILSVRC 2014, is an ensemble of CNNs. [10] proposed to estimate predictive
uncertainty by training multiple stand-alone neural networks. [32, 33] showed
that deep ensembles could learn different modes of function with ensemble
members that only differ in initialization weights.

##  III Uncertainty-Aware Deep Video Compression

This section presents our main contributions. First, we introduce the
theoretical background and the motivation of our proposed approach in Section
III-A. Then we introduce the ensemble-based decoding module to decode multiple
candidates of motion vectors and residuals in Section III-B. In order to
encourage diversity among the ensemble members and to improve the overall
performance, we propose an ensemble-aware loss for ensemble-based decoders in
Section III-C. Finally, we introduce an adversarial training strategy that we
find beneficial for the learning-based video compression task in Section
III-D.

(a)

(b)

Figure 1: (a) A low latency predictive coding-based video compression
framework. (b) We follow the predictive coding-based video compression and
propose ensemble-based decoders.

###  III-A Uncertainties in Deep Video Compression

The predictive coding-based model is a popular framework for video compression
and is widely used by most previous state-of-the-art models [3, 34, 35]. Let
the current frame be $x_{t}$ and the reconstructed previous frame from the
buffer be $\hat{x}_{t-1}$. We estimate a motion vector (MV) map $f_{t}$ with a
motion estimation network. The optical flow is then sent to a motion auto-
encoder for transform coding, yielding quantized bits $\hat{a}_{t}$ and the
reconstructed optical flow $\hat{f}_{t}$. Bilinear warping is used for motion
compensation (MC) and an MC prediction $\tilde{x}_{t}$ with residual
$r_{t}=x_{t}-\tilde{x}_{t}$ is obtained. The residual $r_{t}$ is then
compressed with a residual encoder and decoder, outputting quantized residual
bitstream $\hat{b}_{t}$ and the decoded residual $\hat{r}_{t}$. The
reconstructed current frame is the sum of the MC prediction and the decoded
residual written as

$\displaystyle\hat{x}_{t}=\mathrm{BilinearWarp}(\hat{x}_{t-1},\hat{f}_{t})+\hat{r}_{t}.$
(1)

Aleatoric uncertainty. Although at encoding time we have complete information
necessary to decode $\hat{f}_{t}$, for lossy compression at certain bit rates,
we quantize the bitstream that is passed to the decoder and inevitably
introduces aleatoric uncertainty at decoding time. Since the aleatoric
uncertainty cannot be reduced with more training data, a well-trained codec
cannot mitigate the quantization noise or fully recover the estimated MV
$f_{t}$.

Consider the MV auto-encoder in the predictive coding-based framework above.
The lossy compression of the motion vectors can be summarized as

$\displaystyle a_{t}$ $\displaystyle=\text{MVEncoder}(f_{t})$
$\displaystyle\hat{a}_{t}$ $\displaystyle=q(a_{t})=a_{t}+\eta$
$\displaystyle\hat{f}_{t}$ $\displaystyle=\text{MVDecoder}(\hat{a}_{t}).$ (2)

If the MV decoder is implemented with a linear model parameterized by $w$, the
impact of the quantization noise $\eta$ on the decoded MV is given by

$\displaystyle
w^{\top}\hat{a}_{t}=w^{\top}(a_{t}+\eta)=w^{\top}a_{t}+w^{\top}\eta.$ (3)

Since $\eta$ is introduced by the quantization operation, we have
$\lVert\eta\rVert_{\infty}\leq 1/2=\varepsilon$ and it follows that the upper
bound of the effects from the quantization operation is given by

$\displaystyle\lVert w^{\top}\eta\rVert_{1}\leq\varepsilon\left\lVert
w^{\top}\text{sign}\left(\frac{\partial w^{\top}a_{t}}{\partial
a_{t}}\right)\right\rVert_{1}=\frac{1}{2}\lVert
w^{\top}\text{sign}(w)\rVert_{1}.$ (4)

While in practice, the MV decoder is usually implemented with a stack of
convolution layers and nonlinear activation layers, such as leaky ReLUs, the
transformation of the MV decoder may be too linear to reject the quantization
noise [15].

We conduct preliminary experiments to visualize the aleatoric uncertainty
introduced by the quantization operation. We add a small perturbation
$\eta_{0}$ to the quantized bitstream and obtain
$\hat{a}_{t}^{\prime}=\hat{a}_{t}+\eta_{0}$. The perturbation $\eta_{0}$ is
only added to positions where the quantization gap is at least $0.1$ and
corresponds to only $20\%$ of the size of the perturbation gap. We model the
aleatoric uncertainty with the L1 norm between two optical flows decoded from
bitstreams that differs in a small perturbation. Results on the first two
frames of the BasketballDrill sequence are shown in Fig. 2(d). As we can see,
the aleatoric uncertainty is not uniform across the whole image. Instead,
there is more aleatoric uncertainty around the object boundaries and regions
where the motion is large. While, by definition, such aleatoric uncertainty
cannot be reduced away, blindly assuming the optical flows to be accurate
would lead to larger intermediate residual errors and cost more bits in the
residual coding.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 2: A preliminary experiment on the underlying uncertainty of the
optical flows. (a) The current frame $x_{t}$ to be compressed. (b) The
estimated MV $f_{t}$. (c) The decoded MV $\hat{f}_{t}$. (d) Aleatoric
uncertainty measured as the L2 distance between two optical flows with and
without a small perturbation on the bitstream. (e) Epistemic uncertainty
measured by motion vectors that cannot be estimated well. (f) The predictive
uncertainty represented by the ensemble-based decoder.

Epistemic uncertainty. Due to limited observed data during training, epistemic
uncertainty accounts for the uncertainty in the model parameters as well as
the estimated motion vectors we use to exploit temporal correlation. Motion
vectors near the object boundaries and occluded regions tend not to be
estimated well, and warping erroneous motion vectors would propagate errors to
the residual coding. We may roughly visualize such uncertainty by optimizing a
motion estimation network with regards to the mean squared errors (MSE)
between the current frame $x_{t}$ and the warped frame

$\displaystyle\mathcal{L}=\text{MSE}(x_{t},\text{BilinearWarp}(\hat{x}_{t-1},f_{t}))$
(5)

We depict the results in Fig. 2(e). Since the motion estimation is optimized
to minimize the MSE, regions where the MSE is large are likely to have a
larger epistemic uncertainty and the corresponding motion vectors cannot be
estimated well given the limited training data.

Ideally, we could save MV bits by not encoding MVs that are not estimated well
and save residual bits by not warping MVs which would not help to reduce
residuals. Unfortunately, this is often difficult to implement as an end-to-
end optimized deep neural network. In the next section, we will show that with
the help of ensemble-based decoders, the model could learn to exploit
available information in the bitstream and handle those predictions with
larger uncertainty.

###  III-B Ensemble-Based Decoder

Our proposed ensemble-based decoder decodes multiple groups of motion vectors
(MV) for motion compensation and multiple groups of residuals for the final
reconstruction. The ensemble-based MV decoder and residual decoder are
depicted in Fig. 1. Take the ensemble-based MV decoder as an example. The MV
decoder backbone first decodes a high-dimensional MV feature representation
from the quantized MV bitstream $\hat{a}_{t}$. Then $h$ groups of MVs, denoted
by $\\{\hat{f}_{t}^{m}\mid m=1,\dots,h\\}$, are decoded from the MV feature
with respective MV decoder branches. We obtain $h$ warped frames
$\\{\tilde{x}_{t}^{m}\mid m=1,\dots,h\\}$ by bilinearly warping each
$\hat{f}_{t}^{m}$ on the reference frame $\hat{x}_{t-1}$. The $h$ warped
frames are then concatenated for motion compensation and retained for the
final reconstruction.

Many previous ensemble-based models train an ensemble of stand-alone neural
networks [31, 25, 36]. While they can outperform the model without ensemble-
based decoders by a wide margin, the number of parameters is greatly
increased, as well as the inference complexity. [30] proposed to share
backbone parameters with TreeNets, but the models achieve the best performance
when very few layers are shared. In our ensemble-based decoder structure, each
decoder branch shares most of the convolution layers, making each decoder
branch lightweight. This design effectively improves the overall performance
with negligible complexity increase (see Section IV-D).

The ensemble of decoded MVs can be represented by an equally weighted Gaussian
mixture model given by

$\displaystyle\hat{f}_{t}\sim\frac{1}{h}\sum_{m=1}^{h}\mathcal{N}(f\mid\hat{f}_{t}^{m},\Sigma_{t}^{m}),\;\Sigma_{t}^{m}=\begin{bmatrix}\sigma_{t,x}^{m}&0\\\
0&\sigma_{t,y}^{m}\end{bmatrix}$ (6)

where $\sigma_{t,x}^{m}$ and $\sigma_{t,y}^{m}$ are the variance in $x$ and
$y$ directions respectively. The mean and variance of the Gaussian mixture
model are respectively

$\displaystyle\mathbb{E}[\hat{f}_{t}]$
$\displaystyle=\mu_{\hat{f}_{t}}=\frac{1}{h}\sum_{m=1}^{h}\hat{f}_{t}^{m}$ (7)
$\displaystyle\sigma_{\hat{f}_{t,x}}^{2}$
$\displaystyle=\frac{1}{h}\sum_{m=1}^{h}\left(\left(\sigma_{t,x}^{m}\right)^{2}+\left(\hat{f}_{t,x}^{m}\right)^{2}\right)-\mu_{\hat{f}_{t}}^{2}.$
(8)

How can ensemble-based decoders capture predictive uncertainty? Our
uncertainty-aware model is end-to-end optimized with the rate-distortion loss,
but each branch in the ensemble-based decoder is initialized with random
weights. With the help of the ensemble-aware loss (Section III-C), the
functions learned by the decoder branches are diverse in the parameter space
but similar in the function space for the training samples. Importantly, for
out-of-distribution data in the testing samples, different decoder branches
would yield highly varied predictions. We represent this predictive
uncertainty as the variance between an ensemble of intermediate
representations, and such uncertainty can be propagated between modules (see
Fig. 1). After being end-to-end optimized with rate-distortion optimization,
each module in our model “sees” the predictive uncertainty and learns to
process the representation accordingly.

Why is predictive uncertainty crucial for learning-based video compression?
Models designed for other vision tasks, such as image recognition or
segmentation, often consist of a stack of convolution layers with nonlinear
activation functions. Uncertainty in such high-dimensional representations can
be easily coded in the magnitude of the values, and noises can be corrected by
high-dimensional nonlinear mappings. However, in learning-based video
compression, the models are built on 2D optical flows and quantized bitstream.
Errors in the optical flows and the quantization noises in the bitstream lead
to artifacts in the reconstructed frames. Although we cannot ignore “bad” MVs,
we can alleviate the influence of such MVs by refining the warped frames with
the learned uncertainty information. Similarly, we could relieve the artifacts
introduced by the quantization noise by processing an ensemble of decoded
residuals from the parallel decoder branches.

Visualization of the predictive uncertainty. In order to investigate the
predictive uncertainty learned by the ensemble-based decoders and to confirm
that the benefit of ensemble-based decoders is not due to extra model
complexity or additional non-linearity (from bilinear warping), we conduct
preliminary experiments. Empirically, we visualize the predictive uncertainty
represented by this ensemble model with the variance of the Gaussian mixture
model by setting $(\sigma_{t,x}^{m})^{2}=(\sigma_{t,y}^{m})^{2}=1$, which
gives

$\displaystyle\sigma_{\hat{f}_{t,x}}^{2}=\frac{1}{h}\sum_{m=1}^{h}\left(\hat{f}_{t,x}^{m}\right)^{2}-\left(\frac{1}{h}\sum_{m=1}^{h}\hat{f}_{t,x}^{m}\right)^{2}+1.$
(9)

The predictive uncertainty for the first two frames in the BasketballDrill
sequence is depicted in Fig. 2. Results from more video sequences are shown in
Fig. 5. We can see that the predictive uncertainty estimated by the ensemble-
based decoders can properly capture both the aleatoric uncertainty and the
epistemic uncertainty shown in Fig. 2 — the basketball and human body parts
have large aleatoric uncertainty due to rapid motion and the object boundaries
have large epistemic uncertainty.

Relation to scale-space flow. [4] estimated a scale field $\hat{\sigma}_{t}$
besides the 2-dimensional optical flow $(\hat{f}_{t,x},\hat{f}_{t,y})$. We may
represent the decoded MV with a multivariate Gaussian distribution given by

$\displaystyle\hat{f}_{t}\sim\mathcal{N}((\hat{f}_{t,x},\hat{f}_{t,y})^{\top},\Sigma),\;\Sigma=\begin{bmatrix}\hat{\sigma}_{t}&0\\\
0&\hat{\sigma}_{t}\end{bmatrix}$ (10)

and the scale-space warp gives a weighted mean of the warped value obtained
from $\hat{f}_{t}$. As we can see, the MV prediction from our proposed
ensemble-based decoder (Eq. 6) can represent a more diverse distribution than
the multivariate Gaussian distribution from the scale-space flow (Eq. 10). On
the other hand, the regressed variance $\hat{\sigma}_{t}$ can be unreliable
for out-of-distribution data and is often mis-interpreted as the predictive
uncertainty [9]. Instead, ensemble models produce diverse results by learning
different modes of the function, rather than interpolating around a given mean
in the output space [32, 37]. Given out-of-distribution inputs, each decoder
branch would perform very differently and our ensemble-based decoder can
capture the predictive uncertainty from the Gaussian mixture representation.

###  III-C Ensemble-Aware Training

Intuitively, diversity is a key factor for ensemble models. Ensemble members
similar in the parameter space are unlikely to provide any more useful
information than their non-ensemble counterparts. [29] proved the bias-
variance trade-off in ensemble, $E=\bar{E}-\bar{A}$, which suggested that the
inherent variance is the key for the ensemble models to be effective and we
should encourage the diversity among the ensemble members.

In the previous literature, multiple approaches are considered, including
random initialization, bagging, and boosting. Randomly initializing the model
parameters is a simple but effective approach to induce randomness and is
quite suitable for deep ensembles [30]. Bagging trains ensemble members on
independently drawn examples with bootstrap sampling but could harm the model
performance since each model may see only 63% of the available data [30] and
would perform poorly when there is a high correlation inherent in the data
[38]. Boosting generates the ensemble models sequentially and can be very time
consuming for training deep ensembles.

To induce diversity in different branches of our ensemble-based decoding
module, we choose to randomly initialize the network parameters, and initial
experiments show the efficacy of our approach. To further encourage the
diversity among the ensemble members, we propose an ensemble-aware loss that
can be applied to any deep ensemble model and induce additional randomness.

Consider a deep ensemble model with $h$ ensemble members and the task is to
regress an image $x$. Let the $h$ predictions from the $h$ ensemble members be
$\hat{x}^{m}$ for $m=1,\dots,h$. For each 2D location $(i,j)$, let $p$ be the
decoder with the $k$-th smallest loss. The ensemble-aware loss is given by

$\displaystyle\mathcal{L}_{\text{ensemble-
aware}}(x,\hat{x}^{1},\dots,\hat{x}^{h})=$ (11)
$\displaystyle\sum_{m=1}^{h}\frac{1}{H\times W}\sum_{1\leq i,j\leq
H,W}\min(\lVert\hat{x}_{i,j}^{m}-x_{i,j}\rVert_{2}^{2},\lVert\hat{x}_{i,j}^{p}-x_{i,j}\rVert_{2}^{2})$

where $i,j$ traverses all locations in the 2D lattice. For each 2D location
$(i,j)$, the gradient derived from the ensemble-aware loss with respect to the
$m$-th ensemble decoder is equivalent to the ensemble member with the $k$-th
smallest MSE. As demonstrated in Algorithm 1, this loss function can be
implemented by clipping the gradients with respect to each ensemble member.

Algorithm 1 Training with ensemble-aware loss.

1:Given reconstructed frame $\hat{x}^{1},\dots,\hat{x}^{h}$.

2:Compute MSE loss for each reconstruction
$\mathcal{L}_{\text{MSE}}^{m}\in\mathbb{R}^{H\times W}$ for $m=1\dots h$.

3:Concatenate $\mathcal{L}_{\text{MSE}}^{m}$ for $m=1\dots h$ and obtain a
loss matrix $\mathcal{L}_{\text{MSE}}\in\mathbb{R}^{H\times W\times h}$.

4:for position $(i,j)$ in 2D lattice do

5: Let $p$ ($1\leq p\leq h$) be the decoder with the $k$-th smallest loss in
$\mathcal{L}_{\text{MSE}}^{m}(i,j)$ for $m=1\dots h$.

6: for $m=1\dots h$ do

7:
$\mathcal{L}_{\text{MSE}}^{m}(i,j):=min(\mathcal{L}_{\text{MSE}}^{m}(i,j),\mathcal{L}_{\text{MSE}}^{p}(i,j))$.

8: end for

9:end for

10:By back propagating $\mathcal{L}^{m}_{\text{MSE}}$, the gradients w.r.t.
each ensemble decoder is properly clipped.

The advantage of our ensemble-aware loss is two-fold. On the one hand, this
ensemble-aware loss can effectively encourage diversity among the ensemble
members. With the standard loss, each decoder is forced to perfectly
reconstruct every frame regardless of the outputs from other decoders. This
would push all decoders to a comparable representation space and perform
similarly. Instead, with the ensemble-aware loss, we clip the gradients with
respect to decoders with large MSE losses, allowing disagreement between
ensemble members. Hence each decoder only aims to perfectly reconstruct a
subset of frames, allowing them to explore a more diverse parameter space and
as a whole, better capture the predictive uncertainty. On the other hand, each
ensemble member is supervised by all the training samples. The oracle set loss
proposed in [30] assigned exclusive training samples to each ensemble member
and significantly harmed the performance of individual ensemble members since
each branch only sees a small portion of all training data. Instead, our
ensemble-aware loss can effectively encourage diversity among ensemble
members, and at the same time, guarantee reliable performance for each
ensemble member.

###  III-D Adversarial Training with FGSM

Adversarial examples [39] are training samples with small but non-random
perturbations that are misclassified by neural networks with high confidence.
[15] proposed the fast gradient sign method (FGSM) that applies linear but
intentionally worst-case perturbation to the training samples, as given by

$\displaystyle\eta=\epsilon\cdot\text{sign}(\nabla xJ(\theta,x,y))$ (12)

where $J(\theta,x,y)$ is the cost function, and $\epsilon$ controls the norm
of the perturbation. This adversarial training strategy has been shown to
boost the image classification performance and improve the model’s robustness
to adversarial examples. [10] interpreted FGSM as an efficient solution to
smooth the predictive distributions by increasing the likelihood of the target
around an $\epsilon$-neighborhood of the observed training samples.

We find adversarial training with FGSM closely related to learned lossy
compression and an effective approach to improve the performance of learned
video codecs. In transform coding, we want the latent representation to be as
smooth as possible, since after quantization, all latent representations in
the $\epsilon$-neighborhood,
$\\{\hat{a}+\eta\mid\lVert\eta\rVert_{\infty}<\epsilon\\}$, correspond to the
same decoded output. Learning a smooth latent representation would help to
make the output more robust to quantization noise. Although this could be a
natural result of an end-to-end optimized video codec, the experimental
results show that FGSM can effectively improve the rate-distortion
performance. Algorithm 2 summarizes our training pipeline with ensemble-aware
training and FGSM.

Algorithm 2 Overview of our training pipeline.

1:Compute $\mathcal{L}_{\text{MSE}}$ given reconstructed frame
$\hat{x}^{1},\dots,\hat{x}^{h}$ and the ground truth frame $x$.

2:Compute FGSM perturbations $\eta$ based on the gradients w.r.t. $x$:
$\eta=\epsilon\times\text{sign}(\nabla_{x}\mathcal{L}_{\text{MSE}})$.

3:Add FGSM perturbations to the ground truth frame $x:=x+\eta$, and train
network with the ensemble-aware training in Algorithm 1.

##  IV Experiments

###  IV-A Experimental Setup

Model architecture. Our base model architecture follows the design in [3], and
we use auto-regressive and hierarchical priors for both the motion vector and
residual compression. In order to optimize the model in an end-to-end manner,
we need to relax the bits estimation since quantizing the latent bits would
make the gradients zero almost everywhere. Following [40], we substitute the
quantization operation with additive uniform noise during training and perform
actual quantization during inference.

Training datasets. Our model is trained on 64,612 video sequences from the
training part in Vimeo-90K settuplet dataset [41]. Each video clip has seven
frames with a resolution of $448\times 256$. We randomly crop the video
sequences into $256\times 256$ pixels during training. Given two successive
frames from a random sequence, we treat the first frame as the reference
frame, and our model is trained to minimize the rate-distortion cost of
encoding and decoding the second frame.

Implementation of ensemble-based Decoders. As depicted in Fig. 1, ensemble-
based decoders consist of a shared feature backbone and multiple parallel
decoder branches. The decoder branches are lightweight and include two
convolution layers with one leaky ReLU in between. For the ensemble-based MV
decoder, the decoder backbone first decodes MV feature representation from the
MV bitstream $\hat{a}_{t}$, and then $h$ MVs are decoded with respective MV
decoder branches. From the $h$ decoded MVs and the previous decoded frame, we
obtain $h$ motion compensation (MC) predictions with bilinear warping. The $h$
MC predictions are concatenated with the previous decoded frame and sent to
the Prediction Refine Net, from which we get $h$ refined MC predictions. For
the ensemble-based residual decoder, the decoder backbone first decodes the
residual feature representation from the residual bitstream $\hat{b}_{t}$, and
then $h$ residuals are decoded with the respective residual decoder branches.
From the $h$ decoded residuals and the $h$ refined MC predictions, we obtain
$h$ reconstructions. Finally, the $h$ reconstructions are concatenated with
$h$ refined MC predictions and sent to the Reconstruction Refine Net, from
which we get one refined reconstruction as the final decoded frame. All
modules in our model, including decoder backbone, decoder branch, and refine
nets, are implemented with neural networks and optimized in an end-to-end
manner.

Training details. In our experiments, we adopt the progressive training
strategy and warm up the inter-coding module for 150,000 steps with the
ensemble-aware motion compensation loss in Eq. 11. Then the model is end-to-
end optimized with the rate-distortion loss given by

$\displaystyle\mathcal{L}_{RD}=\left(R_{\text{mv}}(\hat{a}_{t})+R_{\text{res}}(\hat{b}_{t})\right)+\lambda\cdot
D(x_{t},\hat{x}_{t})$ (13)

where $R_{\text{mv}}(\hat{a}_{t})$ and $R_{\text{res}}(\hat{b}_{t})$ represent
the numbers of bits used to encode the motion vectors and the residual,
$D(F_{t},\hat{F}_{t})$ measures the distortion in mean squared error or multi-
scale structural similarity [42], and $\lambda$ is the hyperparameter
controlling the trade-off. Four models are trained with different quality
rates by setting $\lambda=256,\;512,\;1024,\;2048$. We use the AdamW optimizer
[43] with an initial learning rate of $1\times 10^{-4}$, which is then
decreased to $1\times 10^{-5}$ after $1.2\times 10^{6}$ steps. Each model is
trained on one NVIDIA V100 GPU. For the main results reported in Section IV-B
and Section IV-D, we used $h=4$ ensemble decoders, with $k=1$ in ensemble-
aware loss and $\epsilon=4/255$ in FGSM.

###  IV-B Quantitative Results

Testing datasets. To show the effectiveness of our proposed uncertainty-aware
model, we test our model on the first 100 frames from video sequences in HEVC
[44] with GoP size 10, and the first 120 frames from sequences in UVG [45],
MCL-JCV [46] with GoP size 12. To balance the trade-off between complexity and
performance, we use ensemble-based decoders with $h=4$ members.

Baseline models. DVC [18] is the pioneer model in deep video compression area.
It adopts the residual coding-based framework which is the most common
framework in traditional coding standards. DVC Pro [3] is the enhanced model
of DVC and was also the state-of-the-art model when we developed our
algorithm. Thus, considering the significant influence of DVC and DVC Pro, we
chose these two models as the benchmarks and tested our algorithms based on
DVC Pro. In order to build the best learning-based video codec, we adopt the
state-of-the-art image compression model [47] for intra-frame coding. To
fairly compare performance and to demonstrate the effectiveness of our
approach, we test DVC [18] and DVC Pro [3] trained with identical experimental
settings and the same intra-frame model, denoted as DVC (cheng2020) and DVC
Pro (cheng2020).

Moreover, we also compare our methods with other state-of-the-art methods in
Table I to further exhibit the advantage of our approach. HU_ECCV20 [35] and
LU_ECCV20 [34] were parallel works based on DVC and addressed the resolution
issues and content domains. NeRV_NeurIPS21 [5] proposed a general neural
representation for videos and achieved good performance for video compression
by transmitting the model weights for each video.

We calculate the BD-rate [48] of different learning-based video compression
models using x.265 as the anchor model, and the results on HEVC, UVG, MCL-JCV
are reported in Table I. We also plot the RD curves of different codec models
in Fig. 3.

Quantiative results. From the results reported in Table I and Fig. 3, we could
see that our proposed model can effectively save bits compared to our strong
baseline and outperform previous state-of-the-art learning-based video codecs
by a wide margin in all testing datasets.

TABLE I: Comparisons between different learning-based video compression models
measured in BD rate. The anchor model is x265. veryslow preset is used for
both x264 and x265. A negative number means bitrate saving, and a positive
number means bitrate increase.
MODEL | HEVC B | HEVC C | HEVC D | HEVC E | UVG | MCL-JCV
---|---|---|---|---|---|---
x264 | 35.0% | 19.9% | 15.5% | 50.0% | 32.7% | 30.3%
x265 | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0%
DVC (public) | 26.7% | 41.5% | 31.1% | 17.8% | 21.9% | 16.0%
DVC Pro (public) | -0.4% | 11.5% | 4.5% | -3.8% | -12.6% | -5.3%
DVC (cheng2020) | 7.9% | 15.1% | 7.2% | 21.1% | 17.2% | 13.3%
DVC Pro (cheng2020) | -9.0% | 7.2% | -6.9% | 17.2% | -7.9% | -4.1%
HU_ECCV20 | 2.4% | 13.0% | 10.8% | -8.6% | -5.4% | -12.6%
LU_ECCV20 | 5.0% | 8.4% | 3.6% | 11.7% | 8.8% | 8.4%
Agustsson_CVPR20 |  |  |  |  | -14.3% | -16.9%
NeRV_NeurIPS21 |  |  |  |  | -19.1% | -2.7%
Ours | -22.3% | -6.0% | -19.0% | -24.3% | -25.5% | -18.2%

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Rate-distortion comparisons between our model and x264, x265, DVC
[18] , DVC Pro [3], Hu_ECCV20 [35], LU_ECCV20 [34], Agustsson_CVPR20 [4], and
NeRV [5] on different datasets. veryslow preset is used for both x264 and
x265. Best viewed in color.

###  IV-C Qualitative Results

In Fig. 2, we visualize the aleatoric and epistemic uncertainty in the first
two frames of the BasketballDrill sequence, as well as the predictive
uncertainty represented by the ensemble-based MV decoder. As we can see, the
predictive uncertainty is more significant in regions where the motion cannot
be accurately estimated or is too complicated to encode. Our uncertainty-aware
model learns to effectively represent the underlying uncertainty with an
ensemble of decoded MVs, and this uncertainty is retained until the final
reconstruction.

Predictive uncertainty. To demonstrate that the decoded representation from
each member is indeed diversified to effectively capture the underlying
uncertainty, we visualize the predictive uncertainty by modeling the ensemble
predictions with a Gaussian mixture model and representing the predictive
uncertainty with the variance term (see Eq. 9). For each 2D location, the
variance would be larger if the predictions from the ensemble-based decoder
are quite diversified, and the variance would be smaller if the predictions
are consistent. We visualize the predictive uncertainty on three video
sequences, BasketballDrive, RaceHorses, and Kimono1 in Fig. 5. As we could
see, the decoded representations from ensemble members are indeed diversified
and larger predictive uncertainty corresponds to locations with large
aleatoric uncertainty due to rapid motion and large epistemic uncertainty near
object boundaries.

###  IV-D Ablation Study

Effectiveness of various proposed modules. We evaluate the effectiveness of
ensemble-based decoders, ensemble-aware loss, and adversarial training with
FGSM by running ablation experiments on the first 30 frames of all sequences
in the HEVC dataset. We adopt the short training strategy for fast
experimentation. The RD curves are presented in Fig. 4(a) and the BD-rates are
reported in Table II. “ED-MV” and “ED-Res” represent ensemble-based decoders
for MV and residual, and “EA-L” refers to training with ensemble-aware loss.
Specifically, “Ours” is the baseline model augmented with ED-MV, ED-Res, and
EA-L. The results show the efficacy of various proposed modules.

TABLE II: Ablation study on the effectiveness of each module. The performance is measured in BD rates using our baseline model as the anchor. ED-MV: ensemble-based MV decoder, ED-Res: ensemble-based residual decoder, EA-L: ensemble-aware loss, FGSM: adversarial training with FGSM. Specifically, “Ours” is the baseline model augmented with ED-MV, ED-Res, and EA-L. All models have $h=4$ considering the trade-off between performance and complexity. Setting | HEVC B | HEVC C | HEVC D | HEVC E
---|---|---|---|---
Ours - ED-MV - ED-Res - EA-L | 0.0 | 0.0 | 0.0 | 0.0
Ours - ED-Res - EA-L | -5.8 | -3.1 | -4.0 | -3.7
Ours - ED-Res | -7.0 | -4.0 | -6.7 | -4.8
Ours | -8.7 | -6.4 | -8.5 | -6.7
Ours + FGSM | -12.7 | -7.9 | -11.2 | -11.4

Ablation study on the number of ensemble members. We investigate the model’s
performance with different numbers of members in the ensemble-based decoders
on HEVC sequences. As shown in Fig. 4(b), we train eight models with
$h=1,\dots,8$ members where $h=1$ is the baseline without ensembling. We see
that the ensemble-based decoder module is effective even with only two
ensemble members, and the performance is improved with more ensemble members.
Quantitative results are reported in Table III.

TABLE III: Ablation study on the number of ensemble members in the ensemble-
based decoders. Performance is measured in BD rates using our baseline model
as the anchor. We train eight models with $h=1,\dots,8$ using the fast
training strategy.
Setting | HEVC B | HEVC C | HEVC D | HEVC E
---|---|---|---|---
$h=1$ | 0.0 | 0.0 | 0.0 | 0.0
$h=2$ | -7.8 | -4.8 | -8.3 | -5.7
$h=3$ | -8.2 | -5.0 | -8.2 | -6.1
$h=4$ | -8.7 | -6.4 | -8.5 | -6.7
$h=5$ | -8.9 | -6.1 | -9.5 | -7.6
$h=6$ | -9.4 | -6.9 | -9.7 | -8.2
$h=7$ | -10.5 | -7.2 | -9.9 | -8.7
$h=8$ | -10.4 | -6.8 | -9.8 | -8.2

Ablation study on the ensemble-aware loss. We introduced a novel ensemble-
aware loss Eq. 11 in Section III-C with a hyperparameter $k$, where the $k$
ensemble members with the smallest MSE are untouched, and the gradients of the
other members are clipped. Following the same experimental setting in the
ablation study, we experiment with our model with $h=8$ ensemble members in
the decoder for $k=2,4,6,8$. The bits savings on HEVC B compared to the model
without ensemble-based decoders are $-8.7\%$ $-10.4\%$, $-7.5\%$, and
$-4.2\%$, respectively.

Complexity analysis. Most previous deep ensembles train multiple stand-alone
models [31, 25, 36] or share very few shallow layers [30] for the model to be
effective. With an ensemble of 6 models, the inference complexity (in MACs)
and model size (in the number of parameters) easily increase by $500\%$. In
our ensemble-based decoder, the ensemble members share the backbone features,
and we achieve superior results with a limited complexity increase. For one
extra ensemble member in the MV and residual decoders, the complexity
increases by $6\%$ and only $1\%$ in the model size. For our largest model
with $h=8$, there is only a $48\%$ increase in complexity and $10\%$ in model
size.

Choice of model designs. Considering the trade-off between model performance
and computational complexity, we chose $h=4$ and $k=1$ in our main experiments
for an desirable performance with negligible complexity costs. For the
implementation of FGSM, we followed a previous implementation of FGSM on
ImageNet [49] and chose $\varepsilon=4/255$.

(a)

(b)

Figure 4: (a) Effectiveness of various proposed modules. (b) Ablation study on
the number of members in ensemble-based decoders.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Visualization of the predictive uncertainty represented by our
proposed ensemble-based decoder on the first two frames in the
BasketballDrive, RaceHorses, and Kimono1 sequence. The detailed calculations
are presented in Eq. 9.

##  V Discussion

In this paper, we studied the aleatoric and epistemic uncertainty in deep
learning-based video compression and proposed to utilize an ensemble of
intermediate predictions to represent the predictive uncertainty at decoding
time. With ensemble-based decoders, our model can adequately model the
uncertainties in the decoded MVs or residuals and effectively refine the
motion compensation predictions and the reconstructed frames with the
predictive uncertainty.

We investigated the performance of our uncertainty-aware decoding module and
proposed a novel ensemble-aware loss to boost the diversity among the parallel
ensemble branches in a single model. We also proposed to incorporate
adversarial training for learning-based video codecs. Experimental results
show the effectiveness of our approach.

Compared with one-stage learning-based video compression models, such as those
based on 3D autoencoders [16, 17], two-stage motion compensation-based models
can decode high-quality frames with low latency. However, intermediate
predictions in these two-stage pipelines are not always accurate, and
erroneous predictions could severely harm the performance of later stages,
especially for out-of-distribution data. Therefore, it is critical to
represent the predictive uncertainty, and our proposed ensemble-based decoder
is a simple but very effective approach to capture such uncertainty. Future
directions could involve modules on the encoder side to model and propagate
the uncertainty to the decoders for an end-to-end uncertainty awareness.

## References

  * [1] Cisco, “Cisco annual internet report (2018-2023) white paper,” 2020. [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html
  * [2] O. Rippel, S. Nair, C. Lew, S. Branson, A. G. Anderson, and L. Bourdev, “Learned video compression,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 3454–3463.
  * [3] G. Lu, X. Zhang, W. Ouyang, L. Chen, Z. Gao, and D. Xu, “An end-to-end learning framework for video compression,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2020.
  * [4] E. Agustsson, D. Minnen, N. Johnston, J. Balle, S. J. Hwang, and G. Toderici, “Scale-space flow for end-to-end optimized video compression,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [5] H. Chen, B. He, H. Wang, Y. Ren, S.-N. Lim, and A. Shrivastava, “Nerv: Neural representations for videos s,” in _NeurIPS_ , 2021.
  * [6] S. Tomar, “Converting video formats with ffmpeg,” _Linux Journal_ , vol. 2006, no. 146, p. 10, 2006.
  * [7] A. Der Kiureghian and O. Ditlevsen, “Aleatory or epistemic? does it matter?” _Structural safety_ , vol. 31, no. 2, pp. 105–112, 2009.
  * [8] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?” _arXiv preprint arXiv:1703.04977_ , 2017.
  * [9] Y. Gal, “Uncertainty in deep learning,” Ph.D. dissertation, University of Cambridge, 2016.
  * [10] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\.
  * [11] D. J. MacKay, “A practical bayesian framework for backpropagation networks,” _Neural computation_ , vol. 4, no. 3, pp. 448–472, 1992.
  * [12] G. E. Hinton and R. Neal, “Bayesian learning for neural networks,” 1995.
  * [13] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [14] D. Nix and A. Weigend, “Estimating the mean and variance of the target probability distribution,” in _Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN’94)_ , vol. 1, 1994, pp. 55–60 vol.1.
  * [15] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [16] J. Pessoa, H. Aidos, P. Tomás, and M. A. Figueiredo, “End-to-end learning of video compression using spatio-temporal autoencoders,” in _2020 IEEE Workshop on Signal Processing Systems (SiPS)_. IEEE, 2020, pp. 1–6.
  * [17] A. Habibian, T. v. Rozendaal, J. M. Tomczak, and T. S. Cohen, “Video compression with rate-distortion autoencoders,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7033–7042.
  * [18] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao, “Dvc: An end-to-end deep video compression framework,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 006–11 015.
  * [19] A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [20] P. He, H. Li, H. Wang, S. Wang, X. Jiang, and R. Zhang, “Frame-wise detection of double hevc compression by learning deep spatio-temporal representations in compression domain,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 3179–3192, 2021.
  * [21] F. Luo, S. Wang, S. Wang, X. Zhang, S. Ma, and W. Gao, “Gpu-based hierarchical motion estimation for high efficiency video coding,” _IEEE Transactions on Multimedia_ , vol. 21, no. 4, pp. 851–862, 2019.
  * [22] M. Lu, T. Chen, Z. Dai, D. Wang, D. Ding, and Z. Ma, “Decoder-side cross resolution synthesis for video compression enhancement,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2022.
  * [23] S. Wang, X. Zhang, X. Liu, J. Zhang, S. Ma, and W. Gao, “Utility-driven adaptive preprocessing for screen content video compression,” _IEEE Transactions on Multimedia_ , vol. 19, no. 3, pp. 660–667, 2017.
  * [24] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _arXiv preprint arXiv:1610.02136_ , 2016.
  * [25] M. Abbasi and C. Gagné, “Robustness to adversarial examples through an ensemble of specialists,” _arXiv preprint arXiv:1702.06856_ , 2017.
  * [26] L. Hansen and P. Salamon, “Neural network ensembles,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 12, no. 10, pp. 993–1001, 1990.
  * [27] D. H. Wolpert, “Stacked generalization,” _Neural Networks_ , vol. 5, no. 2, pp. 241–259, 1992.
  * [28] M. Perrone and L. Cooper, “When networks disagree: Ensemble methods for hybrid neural networks,” _Neural networks for speech and image processing_ , 08 1993\.
  * [29] A. Krogh and J. Vedelsby, “Neural network ensembles, cross validation, and active learning,” in _Advances in Neural Information Processing Systems_ , G. Tesauro, D. Touretzky, and T. Leen, Eds., vol. 7. MIT Press, 1995.
  * [30] S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra, “Why m heads are better than one: Training a diverse ensemble of deep networks,” 2015.
  * [31] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1–9.
  * [32] T. Garipov, P. Izmailov, D. Podoprikhin, D. P. Vetrov, and A. G. Wilson, “Loss surfaces, mode connectivity, and fast ensembling of dnns,” in _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018\.
  * [33] S. Fort, H. Hu, and B. Lakshminarayanan, “Deep ensembles: A loss landscape perspective,” 2020.
  * [34] G. Lu, C. Cai, X. Zhang, L. Chen, W. Ouyang, D. Xu, and Z. Gao, “Content adaptive and error propagation aware deep video compression,” in _European Conference on Computer Vision_. Springer, 2020, pp. 456–472.
  * [35] Z. Hu, Z. Chen, D. Xu, G. Lu, W. Ouyang, and S. Gu, “Improving deep video compression by resolution-adaptive flow coding,” in _European Conference on Computer Vision_. Springer, 2020, pp. 193–209.
  * [36] Y. Wang, D. Liu, S. Ma, F. Wu, and W. Gao, “Ensemble learning-based rate-distortion optimization for end-to-end image compression,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2020.
  * [37] S. Fort, H. Hu, and B. Lakshminarayanan, “Deep ensembles: A loss landscape perspective,” _arXiv preprint arXiv:1912.02757_ , 2019.
  * [38] P. Bartlett, Y. Freund, W. S. Lee, and R. E. Schapire, “Boosting the margin: A new explanation for the effectiveness of voting methods,” _The annals of statistics_ , vol. 26, no. 5, pp. 1651–1686, 1998.
  * [39] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [40] J. Ballé, V. Laparra, and E. P. Simoncelli, “End-to-end optimization of nonlinear transform codes for perceptual quality,” in _2016 Picture Coding Symposium (PCS)_. IEEE, 2016, pp. 1–5.
  * [41] T. Xue, B. Chen, J. Wu, D. Wei, and W. T. Freeman, “Video enhancement with task-oriented flow,” _International Journal of Computer Vision_ , vol. 127, no. 8, pp. 1106–1125, 2019.
  * [42] Z. Wang, E. P. Simoncelli, and A. C. Bovik, “Multiscale structural similarity for image quality assessment,” in _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, vol. 2. Ieee, 2003, pp. 1398–1402.
  * [43] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [44] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, “Overview of the high efficiency video coding (hevc) standard,” _IEEE Transactions on circuits and systems for video technology_ , vol. 22, no. 12, pp. 1649–1668, 2012\.
  * [45] A. Mercat, M. Viitanen, and J. Vanne, “Uvg dataset: 50/120fps 4k sequences for video codec analysis and development,” in _Proceedings of the 11th ACM Multimedia Systems Conference_ , 2020, pp. 297–302.
  * [46] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C.-C. J. Kuo, “Mcl-jcv: a jnd-based h. 264/avc video quality assessment dataset,” in _2016 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2016, pp. 1509–1513.
  * [47] Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, “Learned image compression with discretized gaussian mixture likelihoods and attention modules,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [48] G. Bjøntegaard, “Calculation of average psnr differences between rd-curves,” 2001.
  * [49] E. Wong, L. Rice, and J. Z. Kolter, “Fast is better than free: Revisiting adversarial training,” _arXiv preprint arXiv:2001.03994_ , 2020.

Wufei Ma is a Ph.D. student in Computer Science at Johns Hopkins University.
He obtained his B.S. degree with summa cum laude honor in Computer Science and
Mathematics from Rensselaer Polytechnic Institute in 2020. His research
primarily focuses on representation learning and 3D vision.

Jiahao Li received the B.S. degree in computer science and technology from the
Harbin Institute of Technology in 2014, and the Ph.D. degree from Peking
University in 2019. He is currently a Senior Researcher with the Media
Computing Group, Microsoft Research Asia. His research interests include
neural video compression and other video tasks, like video backbone design and
video representation learning. He has more than thirty published papers,
standard proposals, and patents in the related area.

Bin Li received the B.S. and Ph.D. degrees in electronic engineering from the
University of Science and Technology of China (USTC), Hefei, Anhui, China, in
2008 and 2013, respectively. He joined Microsoft Research Asia (MSRA),
Beijing, China, in 2013 and now he is a Principal Researcher. He has authored
or co-authored over 50 papers. He holds over 30 granted or pending U.S.
patents in the area of image and video coding. He has more than 40 technical
proposals that have been adopted by Joint Collaborative Team on Video Coding.
His current research interests include video coding, processing, transmission,
and communication. Dr. Li received the best paper award for the International
Conference on Mobile and Ubiquitous Multimedia from Association for Computing
Machinery in 2011. He received the Top 10% Paper Award of 2014 IEEE
International Conference on Image Processing. He received the best paper award
of IEEE Visual Communications and Image Processing 2017.

Yan Lu received his Ph.D. degree in computer science from Harbin Institute of
Technology, China. He joined Microsoft Research Asia in 2004, where he is now
a Partner Research Manager and manages research on media computing and
communication. He and his team have transferred many key technologies and
research prototypes to Microsoft products. From 2001 to 2004, he was a team
lead of video coding group in the JDL Lab, Institute of Computing Technology,
China. From 1999 to 2000, he was with the City University of Hong Kong as a
research assistant. Yan Lu has broad research interests in the fields of real-
time communication, computer vision, video analytics, audio enhancement,
virtualization, and mobile-cloud computing. He holds 30+ granted US patents
and has published 100+ papers in refereed journals and conference proceedings.