100

# Supplementary Material: Ab initio surface chemistry with chemical accuracy

Hong-Zhou Ye<EMAIL_ADDRESS>Department of Chemistry, Columbia University,
New York, New York 10027, USA  Timothy C. Berkelbach
<EMAIL_ADDRESS>Department of Chemistry, Columbia University, New
York, New York 10027, USA

Note: figures and equations appearing in the main text will be referred to as
“Fig. Mxxx” and “Eq. Mxxx” in this Supplementary Material document.

###### Contents

  1. S1 Overall computational details
    1. S1.1 Geometries
    2. S1.2 PBE$+U$, PBE0, HF, and MP2
    3. S1.3 LNO-CCSD(T)
  2. S2 Water on the $\alpha$-Al2O3(0001) surface
    1. S2.1 Slab size, surface size, basis size, and Brillouin zone sampling
    2. S2.2 Infinite vacuum correction
  3. S3 Water on the rutile TiO2(110) surface
    1. S3.1 Slab size, surface size, basis size, and Brillouin zone sampling

##  S1 Overall computational details

###  S1.1 Geometries

All geometries are optimized at the DFT@PBE level of theory using the Quantum
Espresso (QE) code 1, 2 with an energy convergence threshold of $0.2$ meV/cell
and a force convergence threshold of $0.05$ eV/Å. The projector augmented wave
pseudopotentials for PBE as obtained from the QE official site are used for
all elements (Table S1). For both surface systems (Al2O3 and TiO2), bulk
geometries are first optimized, from which clean surfaces are constructed and
optimized. The molecularly and dissociatively adsorbed water-surface
geometries are then constructed and optimized, followed by climbing image
nudged elastic band (CI-NEB) calculations to find the transition-state (TS)
geometries. The D3 dispersion correction by Grimme and co-workers 3 is used
for the DFT@PBE geometry optimization of both clean surfaces and surfaces with
adsorbates but not for bulk.

Table S1: The pseudopotential file (as obtained from the Quantum Espresso official site) and valence electrons for each element used by the plane-wave DFT calculations in this work. Element | PP file | Valence electrons
---|---|---
Al | `Al.pbe-n-kjpaw_psl.1.0.0.UPF` | $3s^{2}3p^{1}$
Ti | `Ti.pbe-spn-kjpaw_psl.1.0.0.UPF` | $3s^{2}3p^{6}3d^{2}$
O | `O.pbe-n-kjpaw_psl.1.0.0.UPF` | $2s^{2}2p^{4}$
H | `H.pbe-kjpaw_psl.1.0.0.UPF` | $1s^{1}$

###  S1.2 PBE$+U$, PBE0, HF, and MP2

Single point calculations are performed at the DFT@PBE-optimized geometries
from above using different methods to obtain the reaction energies and kinetic
barriers reported in the main text. These numbers are carefully converged to
accuracy within $1$ kcal/mol with respect to the basis set size, the surface
size, the slab size, and Brillouin zone sampling, as detailed in Sections S2
and S3 for the two surface systems, respectively. The D3 dispersion correction
is included in all reported DFT energies (PBE, PBE0, and PBE$+U$).

The PBE$+U$ calculations are performed using QE with the simplified version of
Cococcioni and de Gironcoli 4 (`lda_plus_u_kind = 0`) and the atomic projector
(`U_projection_type = ’atomic’`). All other calculations, including the
wavefunction calculations and the hybrid DFT calculations with the PBE0
functional, are performed using the PySCF code 5, 6, 7 with translational
symmetry-adapted Gaussian basis functions. The Goedecker-Teter-Hutter (GTH)
pseudopotentials optimized for HF 8 or PBE 9, 10 (for PBE0 only) are employed,
with the same number of valence electrons as listed in Table S1. The
correlation-consistent Gaussian basis set series optimized for the GTH
pseudopotentials from Ref. 11, GTH-cc-pV$X$Z with $X=$ D, T, Q (henceforth
denoted by $X$Z), are used for all elements except for Ti, for which we
performed similar basis optimization as in Ref. 11 to generate the
corresponding GTH-cc-pV$X$Z series (see supporting data file `gth-cc-
pvxz.dat`). We verified at DFT@PBE level that the reaction energetics
calculated from PySCF agree within $1$ kcal/mol with those from QE when using
a TZ basis set. The four-center electron-repulsion integrals are handled using
the fast periodic density fitting algorithm from Ref. 12, 13, 14. The fitting
basis sets are optimized for the GTH-cc-pV$X$Z series of these elements by
minimizing the error of HF and MP2 energies of molecular monohydrides (see
supporting data file `gth-cc-pvxz-fit.dat`). We verified that the error in the
calculated reaction energetics (from both mean-field and correlated
calculations) introduced by these optimized fitting basis sets are less than
$0.1$ kcal/mol by comparing to a large even-tempered fitting basis set
generated by PySCF.

The finite-size error of the HF exchange energy arising from the integrable
divergence at $\bm{G}=\bm{0}$ is handled using the probe charge approach from
Ref. 15, 16, 17, which shifts the energy of all occupied orbitals by a
Madelung constant. This renders the HF total energy an asymptotic $1/N_{k}$
convergence to the thermodynamic limit (TDL), with $N_{k}$ the number of
$k$-points used to sample the Brillouin zone. These “finite-size error-
corrected” orbital energies are used to obtain both the MP2 correlation energy
and the (T) part of the CCSD(T) correlation energy. The CCSD calculations are
performed using the uncorrected orbital energies.

###  S1.3 LNO-CCSD(T)

LNO-CCSD(T) calculations are performed using an in-house implementation in a
developer version of PySCF based on the molecular LNO-CCSD(T) theory by Kállay
and co-workers 18, 19, 20. In PySCF, most methods implemented for molecular
calculations are automatically applicable to solids with $\Gamma$-point
Brillouin zone sampling by replacing the preceding molecular mean-field
calculations (HF in our case) with the corresponding periodic version. For the
general case where the Brillouin zone is sampled by an evenly spaced,
$\Gamma$-included $k$-point mesh, we first transform the $k$-point HF orbitals
in the original simulation cell into the corresponding $\Gamma$-point HF
orbitals in the BvK supercell and then perform LNO-CCSD(T) calculations
therein.

For completeness, we briefly review the LNO-CCSD(T) theory here and guide the
reader to the original publications 18, 19, 20 for details. All systems
studied in this work are closed-shell and described by a spin-restricted HF
reference determinant $\ket{\Phi_{0}}$, with canonical HF orbitals $\psi_{p}$,
orbital energies $\epsilon_{p}$, and total energy $E_{\mathrm{HF}}$. We use
$i,j,k$ for $N_{\mathrm{o}}$ occupied orbitals, $a,b,c$ for $N_{\mathrm{v}}$
virtual orbitals, and $p,q,r,s$ for $N$ unspecified molecular orbitals. In
this basis, the electronic Hamiltonian is

$H=\sum_{pq,\sigma}^{N}h_{pq}a_{p\sigma}^{\dagger}a_{q\sigma}+\frac{1}{2}\sum_{pqrs,\sigma\sigma^{\prime}}^{N}V_{prqs}a_{p\sigma}^{\dagger}a_{q\sigma^{\prime}}^{\dagger}a_{s\sigma^{\prime}}a_{r\sigma}$
(S1)

with $V_{pqrs}=(pq|rs)$ in $(11|22)$ notation.

We choose the generalized Pipek-Mezey method 21 with a `minao` projector to
localize the HF occupied orbitals

$\phi_{I}=\sum_{i}^{N_{\mathrm{o}}}U_{iI}\psi_{i}$ (S2)

Representative localized occupied orbitals are shown in Fig. M1C for water-
Al2O3 and M3B for water-TiO2. For each localized occupied orbital, $\phi_{I}$,
one constructs a local active space $\mathcal{P}_{I}$ by augmenting $\phi_{I}$
with selected LNOs from second-order Møller-Plesset perturbation theory 22
(MP2). Specifically, one computes the occupied-occupied and the virtual-
virtual blocks of the MP2 density matrix

$\displaystyle D^{I}_{ij}$
$\displaystyle=\sum_{ab}t^{(1)}_{iaIb}\big{[}2t^{(1)}_{jaIb}-t^{(1)}_{Iajb}\big{]}$
(S3) $\displaystyle D^{I}_{ab}$
$\displaystyle=\sum_{jc}t^{(1)}_{Iajc}\big{[}2t^{(1)}_{Ibjc}-t^{(1)}_{jbIc}\big{]}$
(S4)

where

$t^{(1)}_{Iajb}=\frac{(Ia|jb)}{\tilde{\epsilon}_{I}+\epsilon_{j}-\epsilon_{a}-\epsilon_{b}}$
(S5)

is an approximate MP2 amplitude with
$\tilde{\epsilon}_{I}=\braket{\phi_{I}}{f}{\phi_{I}}$, and $f$ is the Fock
operator. Diagonalizing the virtual-virtual block

$D^{I}_{ab}=\sum_{c}\xi_{c}^{I}X_{ac}^{I}X_{bc}^{I},$ (S6)

gives the virtual LNOs associated with $\phi_{I}$, i.e.,
$\phi_{b}=\sum_{c}X_{cb}^{I}\psi_{c}$. For the occupied LNOs, we follow Ref.
18 and diagonalize

$\tilde{D}^{I}_{ij}=\sum_{kl}Q_{ik}^{I}D_{kl}^{I}Q_{lj}^{I}=\sum_{k}^{N_{\textrm{o}}-1}\xi_{k}^{I}X_{ik}^{I}X_{jk}^{I}$
(S7)

where $Q_{ij}^{I}=\delta_{ij}-U_{iI}U_{jI}$ projects out $\phi_{I}$ from
$D_{ij}^{I}$ to prevent it from mixing with other occupied orbitals, giving
the occupied LNOs $\phi_{j}=\sum_{k}X_{kj}^{I}\psi_{k}$. The eigenvalues
$\xi^{I}_{p}$, which are between 2 and 0, quantify the importance of a given
LNO to the electron correlation of localized orbital $\phi_{I}$. In practice,
we construct the local active space $\mathcal{P}_{I}$ by keeping those LNOs
satisfying

$\xi_{i}\geq\lambda_{\textrm{o}},\quad{}\xi_{a}\geq\lambda_{\textrm{v}},$ (S8)

for some user-selected thresholds $\lambda_{\textrm{o}}$ and
$\lambda_{\textrm{v}}$, producing $n=n_{\mathrm{o}}+n_{\mathrm{v}}$ orbitals.
Typically, $n$ is much less than $N$ and, for gapped systems such as the
surface systems studied in this work, does not increase with system size for a
targeted level of accuracy. In Fig. M1C and M3B, we also show the density of
the virtual LNOs generated for the representative localized occupied orbitals
with $\lambda_{v}=10^{-5}$, $10^{-6}$, and $10^{-7}$.

A local Hamiltonian is then constructed by projecting $H$ into
$\mathcal{P}_{I}$

$H_{I}=\sum_{pq\in\mathcal{P}_{I},\sigma}^{n}f^{I}_{pq}a_{p\sigma}^{\dagger}a_{q\sigma}+\frac{1}{2}\sum_{pqrs\in\mathcal{P}_{I},\sigma\sigma^{\prime}}^{n}V_{prqs}a_{p\sigma}^{\dagger}a_{q\sigma^{\prime}}^{\dagger}a_{s\sigma^{\prime}}a_{r\sigma^{\prime}}$
(S9)

where

$f_{pq}^{I}=h_{pq}+\sum_{j\notin\mathcal{P}_{I}}^{N_{\mathrm{o}}-n_{\mathrm{o}}}\left(2V_{pqjj}-V_{pjjq}\right),$
(S10)

which includes a frozen core contribution. Solving the CCSD amplitude
equations with $H_{I}$ gives the local CCSD amplitudes $t_{ia}$ and $t_{iajb}$
in $\mathcal{P}_{I}$, from which the local contribution of the CCSD(T)
correlation energy $E^{(I)}$ can be evaluated by

$E^{(I)}_{\mathrm{CCSD(T)}}=\sum_{kl\in\mathcal{P}_{I}}U_{kI}\left[G^{(I),\mathrm{CCSD}}_{kl}+G^{(I),\mathrm{(T)}}_{kl}\right]U_{lI}$
(S11)

where

$G^{(I),\mathrm{CCSD}}_{kl}=\sum_{iab\in\mathcal{P}_{I}}(t_{iakb}+t_{ia}t_{kb})(2V_{ialb}-V_{ibla})$
(S12)

and

$\begin{split}G^{(I),\mathrm{(T)}}_{kl}&=-\frac{1}{3}\sum_{a\geq b\geq
c\in\mathcal{P}_{I}}f_{abc}\sum_{ij\in\mathcal{P}_{I}}\bigg{\\{}\\\
&w_{ijk}^{abc}\left(+8v_{ijl}^{abc}-5v_{ikj^{\prime}}^{abc}-2v_{jil}^{abc}+2v_{jli}^{abc}+2v_{lij}^{abc}-5v_{lji}^{abc}\right)+\\\
&w_{ikj}^{abc}\left(-5v_{ijl}^{abc}+8v_{ilj}^{abc}+2v_{jil}^{abc}-2v_{jli}^{abc}-5v_{lij}^{abc}+2v_{lji}^{abc}\right)+\\\
&w_{kij}^{abc}\left(+2v_{ijl}^{abc}-5v_{ilj}^{abc}-5v_{jil}^{abc}+2v_{jli}^{abc}+8v_{lij}^{abc}-2v_{lji}^{abc}\right)\bigg{\\}}\end{split}$
(S13)

with

$f_{abc}=\left\\{\begin{split}\frac{1}{6},&\quad{}a=c,\\\
\frac{1}{2},&\quad{}a=b~{}\mathrm{or}~{}b=c,\\\
1,&\quad{}\mathrm{otherwise},\\\ \end{split}\right.$ (S14)
$w_{ijk}^{abc}=W_{ijk}^{abc}/\sqrt{-D_{ijk}^{abc}},\quad{}v_{ijk}^{abc}=V_{ijk}^{abc}/\sqrt{-D_{ijk}^{abc}}$
(S15)
$W_{ijk}^{abc}=P_{ijk}^{abc}\bigg{(}\sum_{d}V_{bdai}t_{kj}^{cd}-\sum_{l}V_{ckjl}t_{il}^{ab}\bigg{)}$
(S16)
$V_{ijk}^{abc}=W_{ijk}^{abc}+\frac{1}{2}P_{ijk}^{abc}(V_{aibj}t_{k}^{c})$
(S17)
$D_{ijk}^{abc}=\epsilon_{ii}+\epsilon_{jj}+\epsilon_{kk}-\epsilon_{aa}-\epsilon_{bb}-\epsilon_{cc}$
(S18)

where $P_{ijk}^{abc}(\cdot)$ generates 6-fold permutation

$P_{ijk}^{abc}(X_{ijk}^{abc})=X_{ijk}^{abc}+X_{ikj}^{acb}+X_{jik}^{bac}+X_{jki}^{bca}+X_{kij}^{cab}+X_{kji}^{cba}$
(S19)

The remaining correlation contribution from orbitals $\notin\mathcal{P}_{I}$
can be included at $O(N^{5})$ cost by invoking a composite correction at MP2
level, leading to the final expression for the LNO-CCSD(T) total correlation
energy

$E_{\mathrm{c}}=\sum_{I}^{N_{\mathrm{o}}}\left(E^{(I)}_{\mathrm{CCSD(T)}}-E^{(I)}_{\mathrm{MP2}}\right)+E_{\mathrm{MP2,full}}$
(S20)

where $E^{(I)}_{\mathrm{MP2}}$ is the MP2 correlation energy evaluated in
$\mathcal{P}_{I}$ with $H_{I}$ and $E_{\mathrm{MP2,full}}$ is the MP2
correlation energy for the entire system. The latter can also be decomposed
into contributions from each localized orbital

$E_{\mathrm{MP2,full}}=\sum_{I}^{N_{\mathrm{o}}}\sum_{jab}\tilde{t}_{Iajb}^{(1)}(2V_{Iajb}-V_{Ibja})=\sum_{I}^{N_{\mathrm{o}}}E^{(I)}_{\mathrm{MP2,full}}$
(S21)

Note that the summation over $jab$ in Eq. S21 is not restricted to
$\mathcal{P}_{I}$ and $\tilde{t}_{Iajb}$ is the exact MP2 amplitude for the
entire system with one occupied index being transformed into $\phi_{I}$, which
is not the same as the approximate local MP2 amplitudes defined in Eq. S5.
Combining Eqs. S20 and S21, we obtain the final expression of the LNO-CCSD(T)
correlation energy

$E_{\mathrm{c}}=\sum_{I}^{N_{\mathrm{o}}}E^{(I)}_{\mathrm{CCSD(T)}}-E^{(I)}_{\mathrm{MP2}}+E^{(I)}_{\mathrm{MP2,full}}=\sum_{I}^{N_{\mathrm{o}}}E^{(I)}.$
(S22)

For the purpose of calculating reaction energetics as in this work, the local
nature of a chemical reaction allows us to further reduce the computational
cost by invoking a strategy resembling the spirit of many fragment embedding
methods. Specifically, we can choose to use CCSD(T) to evaluate $E^{(I)}$ for
only a subset of $M_{\mathrm{o}}\leq N_{\mathrm{o}}$ localized occupied
orbitals, with the contributions from the remaining
$N_{\mathrm{o}}-M_{\mathrm{o}}$ localized occupied orbitals being accounted
for by the MP2 composite correction, leading to a modified energy expression
from Eq. S20

$E_{\mathrm{c}}=\sum_{I}^{M_{\mathrm{o}}}(E^{(I)}_{\mathrm{CCSD(T)}}-E^{(I)}_{\mathrm{MP2}})+E_{\mathrm{MP2,full}}.$
(S23)

The final reaction energetics calculated from Eq. S23 thus need to be
converged with respect to both the LNO truncation parameters defined in Eq. S8
and the embedded cluster size $M_{\mathrm{o}}$. We detail the convergence of
reaction energetics for the two systems studied in this work in Sections S2.1
and S3.1, respectively.

##  S2 Water on the $\alpha$-Al2O3(0001) surface

The final numbers for the reaction energy $\Delta E$ and barrier $\Delta
E^{\ddagger}$ for the water dissociation on $\alpha$-Al2O3(0001) surface
reported in Fig. M2A are tabulated in Table S2. These numbers are obtained by
combining the energetics from calculations with a finite vacuum size of $23$
$\mathrm{\AA}$ in the direction perpendicular to the slab and a correction to
account for an infinite vacuum size ($\Delta_{\infty}$). As detailed in the
rest of this section, these numbers have been converged to accuracy within $1$
kcal/mol. The convergence with respect to basis size, slab/surface size, and
Brillouin zone sampling, which gives rise to the numbers listed in column
“$23\mathrm{\AA}$ vacuum” in Table S2, is detailed in Section S2.1. The
correction to account for an infinite vacuum, which gives rise to the numbers
in column “$\Delta_{\infty}$”, is discussed in Section S2.2.

Table S2: Final numbers (highlighted in grey) for the reaction energy $\Delta E$ and barrier $\Delta E^{\ddagger}$ of a single water dissociating on the $\alpha$-Al2O3(0001) surface calculated using different methods as reported in Fig. M2A. These numbers are obtained by summing over two contributions, detailed in Sections S2.1 and S2.2. All numbers are in kcal/mol. Method | $\Delta E$ | $\Delta E^{\ddagger}$
---|---|---
23 $\mathrm{\AA}$ vacuum | $\Delta_{\infty}$ | Final | 23 $\mathrm{\AA}$ vacuum | $\Delta_{\infty}$ | Final
DFT@PBE | $-9.3$ | $+1.4$ | $-7.9$ | $4.6$ | $+0.5$ | $5.0$
DFT@PBE0 | $-9.3$ |  | $-7.9$ | $6.6$ |  | $7.1$
MP2 | $-9.1$ | $+1.8$ | $-7.3$ | $7.0$ | $+0.7$ | $7.7$
LNO-CCSD(T) | $-8.9$ |  | $-7.1$ | $8.2$ |  | $8.9$

###  S2.1 Slab size, surface size, basis size, and Brillouin zone sampling

We follow Ref. 23 and other previous DFT studies (e.g., Refs. 24, 25, 26, 27)
of the same system for the notation of slab size (in terms of the number of
atomic layers) and surface size. We will use a notation $(n,m)$ for a slab of
$n$ layers, with the top $m$ layers being allowed to relax during geometry
optimization. A vacuum of $23$ $\mathrm{\AA}$ perpendicular to the slab plane
is used throughout this section. The determination of a correction to account
for the infinite vacuum limit is detailed in Section S2.2.

DFT@PBE. By first fixing the slab size to be $(12,8)$, we determined from
Table S3 the following converged parameters for the plane wave DFT@PBE
calculations:

  * •

a kinetic energy cutoff of $50$ Hartree or $1350$ eV,

  * •

a surface size of $2\times 2$,

  * •

Brillouin zone sampling at the $\Gamma$ point.

By fixing these parameters and scanning over different slab sizes, we
determined from Table S4 a converged slab size of $(12,8)$.

Table S3: DFT@PBE dissociation energy $\Delta E_{\textrm{diss}}$ and the barrier height $\Delta E^{\ddagger}$ (both in kcal/mol) of a single adsorbed water on the $\alpha$-Al2O3(0001) surface evaluated for a $(12,8)$ slab model with different kinetic energy cutoff (in Hartree), surface size, and Brillouin zone sampling. The parameters that converge both energies within $1$ kcal/mol are highlighted in grey. K.E. cutoff | surface |  $k$-point mesh | $\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$
---|---|---|---|---
$50$ | $2\times{}2$ | $1\times{}1\times{}1$ | $-9.3$ | $4.6$
$100$ | $2\times{}2$ | $1\times{}1\times{}1$ | $-9.3$ | $4.6$
$100$ | $2\times{}2$ | $2\times{}2\times{}1$ | $-9.3$ | $4.6$
$50$ | $3\times{}3$ | $1\times{}1\times{}1$ | $-9.0$ | $4.6$
Table S4: Same reaction energetics as in Table S3 for different choices of slab size. All other parameters are fixed at their optimum values determined in Table S3. The slab size that converge both energies within $1$ kcal/mol is highlighted in grey. Slab size | $\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$
---|---|---
(12,5) | $-9.0$ | $4.4$
(12,7) | $-9.2$ | $4.6$
(12,8) | $-9.3$ | $4.6$
(15,10) | $-9.4$ | $4.6$

DFT@PBE0 and MP2. By fixing the slab size to be $(9,5)$, we determined from
Table S5 that a TZ Gaussian basis set is sufficient for converged energetics
for both mean-field (HF and DFT@PBE) and correlated (MP2) calculations. The
same table also confirms that the DFT@PBE energetics from PySCF with the TZ
Gaussian basis set agrees with the plane wave DFT@PBE numbers from Quantum
Espresso (the remaining difference, which is less than $1$ kcal/mol, is mostly
caused by the use of different pseudopotentials). By fixing the basis set and
scanning over different slab sizes, we verified with data in Table S6 that the
$(12,8)$ slab, which converges the DFT@PBE energetics as concluded from Table
S4, is also sufficient to converge the MP2 energetics. Finally, we also expect
these parameters (TZ basis set and $(12,8)$ slab) to be sufficient for
DFT@PBE0 calculations, as the DFT@PBE0 energy is a mixture of the DFT@PBE
energy and the HF exchange energy.

Table S5: Reaction energetics (in kcal/mol) from HF, MP2, and PBE calculated using PySCF with the GTH-cc-pV$X$Z basis sets for a $(9,5)$-$2\times 2$ surface model with $\Gamma$-point Brillouin zone sampling. A TZ basis set (highlighted in grey) is sufficient to converge reaction energetics to within $1$ kcal/mol for both mean-field (HF, PBE, and hence also PBE0) and correlated (MP2) calculations. The basis-set converged DFT@PBE reaction energetics also agree with those from plane wave/Quantum Espresso very well (the small difference is likely caused by the difference in pseudopotentials).  Basis set | HF | MP2 | PBE
---|---|---|---
$\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$ | $\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$ | $\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$
DZ/PySCF | $-13.6$ | $12.6$ | $-8.4$ | $7.2$ | $-7.6$ | $5.0$
TZ/PySCF | $-13.6$ | $13.5$ | $-8.7$ | $7.2$ | $-8.7$ | $5.3$
QZ/PySCF | $-13.6$ | $13.5$ | $-8.7$ | $7.2$ | $-8.7$ | $5.3$
PW/QE |  |  |  |  | $-9.0$ | $4.7$
Table S6: Reaction energetics (in kcal/mol) from MP2 calculated for different slab models using a TZ basis set and for a $2\times 2$ surface model with $\Gamma$-point Brillouin zone sampling. As in Table S4, the same $(12,8)$ slab model is sufficient to converge the MP2 energy. Slab | $\Delta E_{\textrm{diss}}$ | $\Delta E^{\ddagger}$
---|---|---
(12,5) | $-9.1$ | $7.1$
(12,7) | $-9.0$ | $7.2$
(12,8) | $-9.1$ | $7.0$
(15,10) | $-9.1$ | $7.4$

LNO-CCSD(T). As detailed in Section S1.3, we use Eq. S23 to evaluate the LNO-
CCSD(T) reaction energetics, which must be converged with respect to both the
LNO truncation parameter, $\lambda_{\mathrm{o}}$ and $\lambda_{\mathrm{v}}$,
defined in Eq. S8, and the “embedded” cluster size, $M_{\mathrm{o}}$, defined
in Eq. S23. For the LNO truncation parameters, we follow Ref. 19 and scan
$\lambda_{\mathrm{v}}$ while keeping
$\lambda_{\mathrm{o}}=10\lambda_{\mathrm{v}}$. For the embedded cluster size
$M_{\mathrm{o}}$, we note that the localized occupied orbitals (LOs) for this
system fall into two classes: $4$ LOs on the water molecule and $48$ LOs for
every layer of oxygen atoms (there are no LOs localized on aluminum due to the
electron negativity difference between aluminum and oxygen). This results in a
natural way of systematically increasing $M_{\mathrm{o}}$:

  1. 1.

LOs on water molecules only, $M_{\mathrm{o}}=4$,

  2. 2.

plus LOs on the first oxygen layer, $M_{\mathrm{o}}=4+48=52$,

  3. 3.

plus LOs on the second oxygen layer, $M_{\mathrm{o}}=4+48+48=100$,

  4. 4.

plus LOs on the third oxygen layer, $M_{\mathrm{o}}=4+48+48+48=148$,

  5. 5.

plus LOs on the third oxygen layer,
$M_{\mathrm{o}}=4+48+48+48+48=196=N_{\mathrm{o}}$.

The convergence with respect to $M_{\mathrm{o}}$ for different choices of
$\lambda_{\mathrm{v}}$ is shown in Fig. S1 for a (12,8)-$2\times 2$ surface
model using a TZ basis set, from which we see that including LOs up to the
third oxygen layer with $M_{\mathrm{o}}=148$ is sufficient to converge the
reaction energetics to accuracy within $1$ kcal/mol. The
$M_{\mathrm{o}}$-converged values from different choices of
$\lambda_{\mathrm{v}}$ (marked as stars in Fig. S1) are then plotted in Fig.
M1E to investigate the convergence with respect to the LNO truncation, where
we also see a fast convergence to the desired accuracy of $1$ kcal/mol, as
already discussed in the main text. The convergence with respect to surface
size, slab thickness, and basis size is presented in Fig. M1G and M1H, from
which we determine that the (12,8)-$2\times 2$ model and the TZ basis set are
sufficient for converging both the reaction energy and the barrier height to
chemical accuracy.

Figure S1: Convergence of the LNO-CCSD(T) reaction energy (left) and barrier
(right) for water-Al2O3 with respect to the embedded cluster size
$M_{\mathrm{o}}$. A (12,8)-$2\times 2$ surface model and a TZ basis set are
employed. Data of different color correspond to different virtual LNO
truncation parameter $\lambda_{\mathrm{v}}$ (the occupied truncation parameter
is chosen to be $\lambda_{\mathrm{o}}=10\lambda_{\mathrm{v}}$ in all cases).

###  S2.2 Infinite vacuum correction

DFT@PBE and DFT@PBE0. We use the dipole correction at DFT@PBE level as
implemented in Quantum Espresso to account for the finite-vacuum effect. We
found a correction of $+1.4$ kcal/mol for the reaction energy and $+0.5$
kcal/mol for the barrier, respectively, as tabulated in Table S2. These
corrections are also applied to obtain the final DFT@PBE0 energetics in Table
S2.

MP2 and LNO-CCSD(T). For MP2, we explicitly increase the vacuum size up to
$200\%$ from the value of $23$ $\mathrm{\AA}$ used above and extrapolated to
the infinite vacuum limit, as shown in Fig. S2. Two extrapolation schemes are
used:

  1. 1.

a linear extrapolation (linear in $1/z_{\mathrm{vac}}$) using the two data
points with largest vacuum size

$E(z_{\mathrm{vac}})=\frac{A}{z_{\mathrm{vac}}}+E(z_{\mathrm{vac}}=\infty)$
(S24)

  2. 2.

a non-linear extrapolation using all data points

$E(z_{\mathrm{vac}})=\frac{A^{\prime}}{z_{\mathrm{vac}}+B}+E(z_{\mathrm{vac}}=\infty)$
(S25)

which agree very well with each other. We thus choose the linear extrapolation
scheme (S24) to obtain the final corrections in Table S2, i.e., a correction
of $+1.8$ kcal/mol for the reaction energy and $+0.7$ kcal/mol for the
barrier. In the spirit of the MP2 composite correction, these corrections are
also applied to obtain the final LNO-CCSD(T) energetics in Table S2.

Figure S2: Convergence of the MP2 reaction energy (left) and barrier (right)
for water-Al2O3 as a function of the reciprocal vacuum size. Extrapolations
based on a linear model (S24) and a non-linear model (S25) are shown in green
and orange, respectively, which agree very well with each other within $1$
kcal/mol.

##  S3 Water on the rutile TiO2(110) surface

The final numbers for the reaction energy $\Delta E$ and barrier $\Delta
E^{\ddagger}$ for the water dissociation on rutile TiO2(110) surface reported
in Fig. M3D are tabulated in Table S7. These numbers are obtained by using a
slab model with a vacuum size of $18$ $\mathrm{\AA}$ perpendicular to the
slab. The error introduced by using this finite vacuum to the final reaction
energetics was estimated at DFT@PBE level to be less than $0.2$ kcal/mol and
hence ignored in all other calculations. The convergence of the numbers
reported in Table S7 to accuracy within $1$ kcal/mol with all other parameters
is detailed as follows.

Table S7: Final numbers (highlighted in grey) for the reaction energy $\Delta E$ and barrier $\Delta E^{\ddagger}$ of a single water dissociating on the rutile TiO2(110) surface calculated using different methods as reported in Fig. M3b.  Method | $\Delta E$ | $\Delta E^{\ddagger}$
---|---|---
DFT@PBE | $0.6$ | $6.3$
DFT@PBE+$U=1$ eV | $0.4$ | $5.8$
DFT@PBE+$U=3$ eV | $-0.2$ | $4.9$
DFT@PBE+$U=5$ eV | $-0.9$ | $3.7$
DFT@PBE+$U=8$ eV | $-2.4$ | $1.5$
MP2 | $1.6$ | $9.6$
LNO-CCSD(T) | $2.2$ | $7.8$

###  S3.1 Slab size, surface size, basis size, and Brillouin zone sampling

We follow Ref. 28 and other DFT studies on the same system for the notation of
slab size and surface size. Like in water-Al2O3, we use a notation $(n,m)$ to
denote a slab of $n$ layers, with the top $m$ layers being allowed to relax
during geometry optimization

DFT@PBE and PBE$+U$. Our preliminary tests suggest that a kinetic energy
cutoff of $1350$ eV and a $2\times 2\times 1$ $k$-point mesh properly converge
the basis set size and the Brillouin zone. By fixing these parameters, we
determined a converged surface size of $1\times 3$ and a converged slab model
of $(7,4)$ from Fig. S3. The same parameters are used for PBE$+U$.

Figure S3: Convergence of the reaction energy (blue) and barrier (red) for
water dissociation on TiO2 surface with respect to the surface size (left) and
the slab size (right). A $(4,2)$ slab model is used for calculations shown on
the left, while a $1\times 2$ surface model is used for calculations shown on
the right. All calculations were done with the Brillouin zone sampled by a
$2\times 2\times 1$ mesh.

MP2. Table S8 suggests that a TZ basis set is sufficient to converge reaction
energetics evaluated at both mean-field (HF) and correlated (MP2) level. Table
S9 suggests that the HF part of the MP2 energy requires sampling the Brillouin
zone with a $2\times 2\times 1$ mesh, while the correlation part converges
faster and requires only $\Gamma$-point Brillouin sampling. Ideally, we thus
would like to use the following energy expression for the total MP2 energy

$E_{\mathrm{MP2}}(\mathrm{TZ}/2\times 2\times 1)\approx
E_{\mathrm{MP2,c}}(\mathrm{TZ}/1\times 1\times
1)+E_{\mathrm{HF}}(\mathrm{TZ}/2\times 2\times 1)$ (S26)

However, a TZ/$2\times\times 1$ HF calculation for the large $(7.4)$ slab
model we want to study is infeasible with the available computational
resources. We thus designed the following strategy

$\begin{split}E_{\mathrm{MP2}}(\mathrm{TZ}/2\times 2\times 1)&\approx
E_{\mathrm{MP2,c}}(\mathrm{TZ}/1\times 1\times
1)+E_{\mathrm{HF}}(\mathrm{DZ}/2\times 2\times 1)+\\\
&\quad{}E_{\mathrm{HF}}(\mathrm{TZ}/1\times 1\times
1)-E_{\mathrm{HF}}(\mathrm{DZ}/1\times 1\times 1)\end{split}$ (S27)

where the desired TZ/$2\times 2\times 1$ HF energy is approximated in a
composite-correction way.

Table S8: Reaction energy $\Delta E$ and barrier $\Delta E^{\ddagger}$ of water dissociation on TiO2 for a $(4,2)$ slab model with $1\times 2$ surface calculated using MP2 with $\Gamma$-point Brillouin zone sampling and different Gaussian basis sets. Contributions from the HF and the MP2 correlation energy components to the MP2 energetics are also shown. As highlighted in gray, both the HF energy and the MP2 correlation energy contributions to the reaction energetics converge to accuracy within $1$ kcal/mol with a TZ basis set. BZ sampling | $\Delta E$ | $\Delta E^{\ddagger}$
---|---|---
HF | MP2,c | MP2 | HF | MP2,c | MP2
DZ | $0.5$ | $6.1$ | $6.6$ | $10.8$ | $0.4$ | $11.2$
TZ | $-1.1$ | $7.0$ | $5.9$ | $9.9$ | $-0.2$ | $9.7$
QZ | $-1.4$ | $7.3$ | $5.9$ | $9.7$ | $0.4$ | $10.1$
Table S9: Same reaction energetics as in Table S9 but calculated using MP2 with a DZ basis set and different Brillouin zone sampling. Contributions from the HF and the MP2 correlation energy components to the MP2 energetics are also shown. As highlighted in gray, the HF energy converges to accuracy within $1$ kcal/mol with $2\times 2\times 1$ Brillouin zone sampling, while the MP2 correlation energy converges even faster with $\Gamma$-point Brillouin zone sampling. BZ sampling | $\Delta E$ | $\Delta E^{\ddagger}$
---|---|---
HF | MP2,c | MP2 | HF | MP2,c | MP2
$1\times 1\times 1$ | $0.5$ | $6.1$ | $6.6$ | $10.8$ | $0.4$ | $11.2$
$2\times 2\times 1$ | $-1.5$ | $6.2$ | $4.7$ | $10.7$ | $-0.1$ | $10.6$
$3\times 3\times 1$ | $-1.7$ |  |  | $10.7$ |  |

LNO-CCSD(T). For the $(7.4)$ slab model, we calculate the LNO-CCSD(T)
correlation energy using Eq. S23 with a MP2 composite correction using the MP2
energy calculated by Eq. S27. The convergence of the LNO-CCSD(T) reaction
energetics with respect to the cluster size $M_{\mathrm{o}}$ is shown in Fig.
S4, where the choice of $M_{\mathrm{o}}$ follows that in water-Al2O3, i.e.,
the first $4$ LOs are localized on water, followed by $72$ LOs in each atomic
layer. Figure S4 suggests that for all choices of the LNO truncation
parameters, $M_{\mathrm{o}}=364$, i.e., including LOs in up to the fifth
atomic layer is sufficient for a converged results. The converged energetics
from using different LNO truncation parameters (marked as stars in Fig. S4)
are then plotted in Fig. M3C to investigate the convergence with respect to
the LNO truncation, where we also see a fast convergence to the desired
accuracy of $1$ kcal/mol, as already discussed in the main text.

Figure S4: Convergence of the LNO-CCSD(T) reaction energy (left) and barrier
(right) for water-TiO2 with respect to the embedded cluster size
$M_{\mathrm{o}}$. Data of different color correspond to different virtual LNO
truncation parameter $\lambda_{\mathrm{v}}$ (the occupied truncation parameter
is chosen to be $\lambda_{\mathrm{o}}=10\lambda_{\mathrm{v}}$ in all cases
except for $\lambda_{\mathrm{v}}=5\mathrm{e}-7$, where
$\lambda_{\mathrm{o}}=1\mathrm{e}-5$).

## References

  * Giannozzi et al. 2009 Giannozzi, P.; Baroni, S.; Bonini, N.; Calandra, M.; Car, R.; Cavazzoni, C.; Ceresoli, D.; Chiarotti, G. L.; Cococcioni, M.; Dabo, I.; Corso, A. D.; de Gironcoli, S.; Fabris, S.; Fratesi, G.; Gebauer, R.; Gerstmann, U.; Gougoussis, C.; Kokalj, A.; Lazzeri, M.; Martin-Samos, L.; Marzari, N.; Mauri, F.; Mazzarello, R.; Paolini, S.; Pasquarello, A.; Paulatto, L.; Sbraccia, C.; Scandolo, S.; Sclauzero, G.; Seitsonen, A. P.; Smogunov, A.; Umari, P.; Wentzcovitch, R. M. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. _J. Phys. Condens. Matter_ 2009, _21_ , 395502
  * Giannozzi et al. 2017 Giannozzi, P.; Andreussi, O.; Brumme, T.; Bunau, O.; Nardelli, M. B.; Calandra, M.; Car, R.; Cavazzoni, C.; Ceresoli, D.; Cococcioni, M.; Colonna, N.; Carnimeo, I.; Corso, A. D.; de Gironcoli, S.; Delugas, P.; DiStasio, R. A.; Ferretti, A.; Floris, A.; Fratesi, G.; Fugallo, G.; Gebauer, R.; Gerstmann, U.; Giustino, F.; Gorni, T.; Jia, J.; Kawamura, M.; Ko, H.-Y.; Kokalj, A.; Küçükbenli, E.; Lazzeri, M.; Marsili, M.; Marzari, N.; Mauri, F.; Nguyen, N. L.; Nguyen, H.-V.; de-la Roza, A. O.; Paulatto, L.; Poncé, S.; Rocca, D.; Sabatini, R.; Santra, B.; Schlipf, M.; Seitsonen, A. P.; Smogunov, A.; Timrov, I.; Thonhauser, T.; Umari, P.; Vast, N.; Wu, X.; Baroni, S. Advanced capabilities for materials modelling with Quantum ESPRESSO. _J. Phys. Condens. Matter_ 2017, _29_ , 465901
  * Grimme et al. 2010 Grimme, S.; Antony, J.; Ehrlich, S.; Krieg, H. A consistent and accurate ab initio parametrization of density functional dispersion correction (DFT-D) for the 94 elements H-Pu. _J. Chem. Phys._ 2010, _132_
  * Cococcioni and de Gironcoli 2005 Cococcioni, M.; de Gironcoli, S. Linear response approach to the calculation of the effective interaction parameters in the $\mathrm{LDA}+\mathrm{U}$ method. _Phys. Rev. B_ 2005, _71_ , 035105
  * Sun 2015 Sun, Q. Libcint: An efficient general integral library for Gaussian basis functions. _J. Comput. Chem._ 2015, _36_ , 1664–1671
  * Sun et al. 2018 Sun, Q.; Berkelbach, T. C.; Blunt, N. S.; Booth, G. H.; Guo, S.; Li, Z.; Liu, J.; McClain, J. D.; Sayfutyarova, E. R.; Sharma, S.; Wouters, S.; Chan, G. K.-L. PySCF: the Python-based simulations of chemistry framework. _Wiley Interdiscip. Rev. Comput. Mol. Sci_ 2018, _8_ , e1340
  * Sun et al. 2020 Sun, Q.; Zhang, X.; Banerjee, S.; Bao, P.; Barbry, M.; Blunt, N. S.; Bogdanov, N. A.; Booth, G. H.; Chen, J.; Cui, Z.-H.; Eriksen, J. J.; Gao, Y.; Guo, S.; Hermann, J.; Hermes, M. R.; Koh, K.; Koval, P.; Lehtola, S.; Li, Z.; Liu, J.; Mardirossian, N.; McClain, J. D.; Motta, M.; Mussard, B.; Pham, H. Q.; Pulkin, A.; Purwanto, W.; Robinson, P. J.; Ronca, E.; Sayfutyarova, E. R.; Scheurer, M.; Schurkus, H. F.; Smith, J. E. T.; Sun, C.; Sun, S.-N.; Upadhyay, S.; Wagner, L. K.; Wang, X.; White, A.; Whitfield, J. D.; Williamson, M. J.; Wouters, S.; Yang, J.; Yu, J. M.; Zhu, T.; Berkelbach, T. C.; Sharma, S.; Sokolov, A. Y.; Chan, G. K.-L. Recent developments in the PySCF program package. _J. Chem. Phys._ 2020, _153_ , 024109
  * Hutter 2019 Hutter, J. New Optimization of GTH Pseudopotentials for PBE, SCAN, PBE0 Functionals. GTH Pseudopotentials for Hartree-Fock. NLCC Pseudopotentials for PBE. https://github.com/juerghutter/GTH, 2019
  * Goedecker et al. 1996 Goedecker, S.; Teter, M.; Hutter, J. Separable dual-space Gaussian pseudopotentials. _Phys. Rev. B_ 1996, _54_ , 1703–1710
  * Hartwigsen et al. 1998 Hartwigsen, C.; Goedecker, S.; Hutter, J. Relativistic separable dual-space Gaussian pseudopotentials from H to Rn. _Phys. Rev. B_ 1998, _58_ , 3641–3662
  * Ye and Berkelbach 2022 Ye, H.-Z.; Berkelbach, T. C. Correlation-Consistent Gaussian Basis Sets for Solids Made Simple. _J. Chem. Theory Comput._ 2022, _18_ , 1595–1606
  * Ye and Berkelbach 2021 Ye, H.-Z.; Berkelbach, T. C. Fast periodic Gaussian density fitting by range separation. _J. Chem. Phys._ 2021, _154_ , 131104
  * Ye and Berkelbach 2021 Ye, H.-Z.; Berkelbach, T. C. Tight distance-dependent estimators for screening two-center and three-center short-range Coulomb integrals over Gaussian basis functions. _J. Chem. Phys._ 2021, _155_ , 124106
  * Bintrim et al. 2022 Bintrim, S. J.; Berkelbach, T. C.; Ye, H.-Z. Integral-Direct Hartree-Fock and Møller-Plesset Perturbation Theory for Periodic Systems with Density Fitting: Application to the Benzene Crystal. _J. Chem. Theory Comput._ 2022, _18_ , 5374–5381
  * Paier et al. 2006 Paier, J.; Marsman, M.; Hummer, K.; Kresse, G.; Gerber, I. C.; Ángyán, J. G. Screened hybrid density functionals applied to solids. _J. Chem. Phys._ 2006, _124_ , 154709
  * Broqvist et al. 2009 Broqvist, P.; Alkauskas, A.; Pasquarello, A. Hybrid-functional calculations with plane-wave basis sets: Effect of singularity correction on total energies, energy eigenvalues, and defect energy levels. _Phys. Rev. B_ 2009, _80_ , 085114
  * Sundararaman and Arias 2013 Sundararaman, R.; Arias, T. A. Regularization of the Coulomb singularity in exact exchange by Wigner-Seitz truncated interactions: Towards chemical accuracy in nontrivial systems. _Phys. Rev. B_ 2013, _87_ , 165122
  * Rolik and Kállay 2011 Rolik, Z.; Kállay, M. A general-order local coupled-cluster method based on the cluster-in-molecule approach. _J. Chem. Phys._ 2011, _135_
  * Rolik et al. 2013 Rolik, Z.; Szegedy, L.; Ladjánszki, I.; Ladóczki, B.; Kállay, M. An efficient linear-scaling CCSD(T) method based on local natural orbitals. _J. Chem. Phys._ 2013, _139_
  * Nagy and Kállay 2017 Nagy, P. R.; Kállay, M. Optimization of the linear-scaling local natural orbital CCSD(T) method: Redundancy-free triples correction using Laplace transform. _J. Chem. Phys._ 2017, _146_
  * Lehtola and Jónsson 2014 Lehtola, S.; Jónsson, H. Pipek-Mezey Orbital Localization Using Various Partial Charge Estimates. _J. Chem. Theory Comput._ 2014, _10_ , 642–649
  * Møller and Plesset 1934 Møller, C.; Plesset, M. S. Note on an Approximation Treatment for Many-Electron Systems. _Phys. Rev._ 1934, _46_ , 618–622
  * Hass et al. 1998 Hass, K. C.; Schneider, W. F.; Curioni, A.; Andreoni, W. The Chemistry of Water on Alumina Surfaces: Reaction Dynamics from First Principles. _Science_ 1998, _282_ , 265–268
  * Hass et al. 2000 Hass, K. C.; Schneider, W. F.; Curioni, A.; Andreoni, W. First-Principles Molecular Dynamics Simulations of H2O on $\alpha$-Al2O3 (0001). _J. Phys. Chem. B_ 2000, _104_ , 5527–5540
  * Ranea et al. 2009 Ranea, V. A.; Carmichael, I.; Schneider, W. F. DFT Investigation of Intermediate Steps in the Hydrolysis of $\alpha$-Al2O3(0001). _J. Phys. Chem. C_ 2009, _113_ , 2149–2158
  * Wang et al. 2011 Wang, B.; Hou, H.; Luo, Y.; Li, Y.; Zhao, Y.; Li, X. Density Functional/All-Electron Basis Set Slab Model Calculations of the Adsorption/Dissociation Mechanisms of Water on $\alpha$-Al2O3(0001) Surface. _J. Phys. Chem. C_ 2011, _115_ , 13399–13411
  * Wirth and Saalfrank 2012 Wirth, J.; Saalfrank, P. The Chemistry of Water on $\alpha$-Alumina: Kinetics and Nuclear Quantum Effects from First Principles. _J. Phys. Chem. C_ 2012, _116_ , 26829–26840
  * Lindan and Zhang 2005 Lindan, P. J. D.; Zhang, C. Exothermic water dissociation on the rutile $\mathrm{Ti}{\mathrm{O}}_{2}(110)$ surface. _Phys. Rev. B_ 2005, _72_ , 075439


# Leveraging Synthetic Data to Learn Video Stabilization Under Adverse
Conditions ††thanks: The paper is under review.

Abdulrahman Kerim
School of Computing and Communications
Lancaster University, UK
<EMAIL_ADDRESS>
&Washington L. S. Ramos
Computer Science Department
Universidade Federal de Minas Gerais, Brazil
<EMAIL_ADDRESS>
Leandro Soriano Marcolino
School of Computing and Communications
Lancaster University, UK
<EMAIL_ADDRESS>
&Erickson R. Nascimento
Computer Science Department
Universidade Federal de Minas Gerais, Brazil
<EMAIL_ADDRESS>
&Richard Jiang
School of Computing and Communications
Lancaster University, UK
<EMAIL_ADDRESS>

###### Abstract

Video stabilization plays a central role to improve videos quality. However,
despite the substantial progress made by these methods, they were, mainly,
tested under standard weather and lighting conditions, and may perform poorly
under adverse conditions. In this paper, we propose a synthetic-aware adverse
weather robust algorithm for video stabilization that does not require real
data and can be trained only on synthetic data. We also present Silver, a
novel rendering engine to generate the required training data with an
automatic ground-truth extraction procedure. Our approach uses our specially
generated synthetic data for training an affine transformation matrix
estimator avoiding the feature extraction issues faced by current methods.
Additionally, since no video stabilization datasets under adverse conditions
are available, we propose the novel VSAC105Real dataset for evaluation. We
compare our method to five state-of-the-art video stabilization algorithms
using two benchmarks. Our results show that current approaches perform poorly
in at least one weather condition, and that, even training in a small dataset
with synthetic data only, we achieve the best performance in terms of
stability average score, distortion score, success rate, and average cropping
ratio when considering all weather conditions. Hence, our video stabilization
model generalizes well on real-world videos and does not require large-scale
synthetic training data to converge.

_K_ eywords Video Stabilization $\cdot$ Synthetic Data $\cdot$ Affine
Transformation

##  1 Introduction

Over the past several years, we have witnessed an explosion of videos being
recorded and shared on the Internet. However, most shared videos are unedited
and shaky, which makes them unpleasant to watch. Therefore, video
stabilization techniques became an essential step in the video processing
pipeline, gaining momentum as more unedited videos are being created and
shared. State-of-the-art video stabilization approaches perform well under
standard conditions but struggle at adverse conditions. Furthermore,
collecting training videos in these adverse conditions is hard, dangerous, and
time-consuming.

The training data bottleneck mentioned above causes many video stabilization
methods to be essentially non-learning-based commonly adopting affine or
homography matrix estimation in the camera motion estimation step, to extract
the camera trajectory. Usually, features extraction, description, and matching
are involved in this process. Feature extractors like SIFT [1] and the
learning-based ones, like R2D2 [2] and ASLFeat [3], can perform well under
standard weather conditions and enough illumination, but they may fail under
challenging conditions such as foggy, rainy, and snowy weather, as well as
night-time scenes and abrupt changes of illumination. For instance, rain and
snow drop particles and textureless scenes under fog or at night-time pose a
clear challenge to find sufficient robust features. Failing to accurately
estimate the actual camera trajectory, in the motion estimation step, causes
the error to propagate to the later steps of the process, decreasing the
quality of the stabilized video.

Figure 1: Our key idea is to use specially-designed synthetic data to train an
affine transformation matrix estimation CNN.

Synthetic data have shown a great progress in the field of computer vision [4,
5, 6, 7]. Its increase in popularity started to attract many researchers to
apply it for different computer vision problems. Most importantly, synthetic
data seems a promising solution to overcome the lack of suitable data for
training supervised learning models. However, we believe that the potential of
synthetic data is not simply in the amount of available data for training. In
fact, in this paper we propose a novel synthetic-aware video stabilization
algorithm that leverages synthetic data and achieves state-of-the-art results
using only a small-scale synthetic dataset. Figure 1 depicts the main steps of
this approach. Leveraging the powerful tools of the Unity game engine, we
built Silver, which creates three-dimensional, photo-realistic virtual worlds
procedurally at run-time. The system can automatically diversify many
essential scene attributes like weather conditions, time of the day,
crowdedness, to name a few. Most importantly, our system can generate the
required ground-truth training data for our learning-based video stabilization
method. Our novel video stabilization model can be trained with a small number
of synthetic videos. With no training or fine-tuning on real data, our model
is more robust and accurate than the state-of-the-art across different weather
conditions. To the best of our knowledge, this is the first work to study
video stabilization under adverse weather conditions and utilize synthetic
videos for video stabilization.

Despite supervised learning-based approaches [8, 9, 10, 11] being able to
learn parameters like cropping window, sensitivity, and even to extract
discriminative features, there is no sufficient labeled data for obtaining
high-quality results in any condition. Finding, collecting, and annotating
relevant data is cumbersome, time-consuming, error-prone, expensive, and
subject to privacy issues. It is widely accepted that collecting many videos
under adverse conditions is not easy since various attributes are needed to be
diversified, such as camera motion, scene type, time of the day, scene
crowdedness, and recording resolution. Even assuming that videos under these
attributes can be captured, there is still a problem capturing training data
for this task. Using two cameras, one with a mechanical stabilizer and another
without, is still not an ideal solution since the difference between the two
camera locations will cause the scene to be captured from two different
viewpoints. Thus, learning-based models may not be able to learn well the
video stabilization task, especially under adverse conditions.

Hence, our main contributions are three-fold: i) a novel synthetic-aware video
stabilization method achieving state-of-the-art results on real videos and
being trained only on synthetic videos. It is worth noting that our key
contribution is the idea of using specially designed synthetic data with a
simple yet powerful architecture for the task of video stabilization; ii) a
new synthetic training data generator, Silver 111A preliminary version of
Silver [12] was presented as an unarchived workshop paper., which is able to
generate an unlimited number of training videos for training our video
stabilization algorithm under a wide set of attributes; iii) a new video
stabilization dataset, the VSAC105Real, composed of real videos spanning
foggy, rainy, and snowy weather conditions, and night-time attributes. The
implementation of our proposed video stabilization algorithm, our simulator,
and the datasets are all available at
https://github.com/A-Kerim/SyntheticData4VideoStabilization.

##  2 Related Work

Our work utilizes synthetic data for the task of video stabilization. Thus,
video stabilization and synthetic data generation literature is briefly
reviewed in this section.

###  2.1 Video Stabilization

Video stabilization methods can be categorized broadly into non-learning-based
and learning-based approaches. Non-learning-based video stabilization methods
do not perform training. For instance, Grundmann et al. [13] stabilize the
shaky camera trajectory using L1-norm optimization under constraints.
Similarly, Bradley et al. [14] address the stabilization task as a constrained
convex optimization problem. These non-learning-based methods stabilize videos
without the need for training data to tune the model’s parameters. However,
they tend to give less pleasant results, can work only under predefined
conditions, and the model’s parameters must be tuned manually.

The learning-based approaches, on their turn, can be classified into
unsupervised and supervised approaches. Non-supervised video stabilization
methods require training videos but do not demand for annotated pairs of
shaky-stable videos. Deep Iterative FRame INTerpolation (DIFRINT) [15] is an
unsupervised learning-based approach that can be trained in an end-to-end
manner. It can stabilize videos without cropping original frames. Essentially,
this method utilizes the frame interpolation technique to synthesize middle
frames for video stabilization. Supervised learning approaches, on the other
hand, require human labeled ground-truth data, which is the main limitation of
applying them to video stabilization. Some works approached this problem using
a mechanical stabilizer to generate ground-truth stable videos with their
shaky counterparts. As an example, StabNet [8] followed this procedure,
generating videos for training CNNs for video stabilization. The network
learns a warping transformation of multi-grids given the shaky video frames
and the previously stabilized ones. On the other hand, Liu et al. [9] apply a
learning-based hybrid-space fusion to compensate for optical flow inaccuracy.
They synthesize stabilized frames by fusing the warped content estimated from
neighboring frames. Yu et al. [10] learn to stabilize videos using optical
flow. They compute the per-pixel warp field from the optical flow of the shaky
video, allowing to better handle moving objects and occlusion. In contrast to
the work of Yu et al. [10], Ali et al. [11] propose a full-frame supervised
video stabilization method that does not require optical flow. Their novel
pipeline for dataset generation uses a linearly moving window on high-
resolution images.

Despite the substantial progress made by these methods, they present a partial
solution to the video stabilization problem since it is assumed that they will
work under normal weather conditions and sufficient illumination. However,
finding resilient features in adverse conditions is rather challenging. For
example, rain particles, foggy weather conditions, and low illumination pose
clear challenges to finding robust features. Thus, it leads to inaccurate
motion estimation and low-quality video stabilization. At the same time,
collecting diverse real-world videos for training video stabilization methods
is cumbersome and time-consuming. Moreover, obtaining data with perfect
ground-truth is not feasible given the physical limitation of recording the
same scene from the same perspective via steady and shaky cameras.

Our video stabilization method belongs to the supervised learning-based
category. However, unlike other methods, we use only synthetic data for
training. To the best of our knowledge, this is the first work to utilize
synthetic videos for video stabilization. No pre-training or fine-tuning on
real data is required, and by using only a small-scale training dataset, our
method is more robust than state-of-the-art methods.

###  2.2 Affine and Homography Transformation

Estimating the affine or homography transformation between two images are two
common approaches to align one image to another. For that aim, there are
different ways to find these matrices like applying a feature extractor (e.g.,
SIFT [1], ORB [16], SURF [17], and OAN [18]) and an outlier rejection
algorithm (e.g., RANSAC [19] and MAGSAC [20]). At the same time, there are
unsupervised [21] and supervised [22] methods that estimate, specifically, the
homography matrix.

Although the traditional approach, i.e., feature extraction, does well at
standard conditions, it performs poorly under challenging conditions such as
adverse weather conditions and low illumination. Moreover, supervised
approaches cannot reflect scene parallax [23] and generating a suitable
training data is rather hard. Unsupervised approaches tried to solve these
problems but they still fail under large baseline alignment which makes it
unpractical for applications such as image stitching and video stabilization
under sharp camera movements.

Unlike previous methods, our model learns the affine transformation in a
supervised manner using specially generated synthetic training data for this
task and it does not need any real data in training time. Estimating the
affine transformation is of a special importance in video stabilization
because it allows us to recover the camera translation, rotation, and scale.
Although it is possible to decompose the homography matrix to extract these
information, it is not accurate. Moreover, training a model to estimate the
affine transformation is much easier compared to estimating the homography
transformation. As have been discussed by other similar works [13, 24],
although homography can better model the camera motion between frames for a
limited number of frames, it starts to cause many artifacts like skew and
perspective as the number of frames becomes larger. At the same time, higher
degrees of freedom transformations (e.g., homography) overfit easily even with
some regularization. Thus, utilizing homography transformation is both harder
to train (more parameters and easier to overfit) and more subject to
artifacts.

###  2.3 Synthetic Data Generation

The use of synthetic data in the computer vision community has shown to be a
promising solution to overcome the lack of suitable data for training
supervised learning models [25, 26, 6, 4, 5, 7]. Adapting a specific video
game to generate synthetic data with its corresponding ground-truth for the
task of semantic segmentation was presented by Richter et al. [26], who
modified the game Grand Theft Auto V for that purpose. At the same time,
another work by Shafaei et al. [6] investigated the use of photo-realistic
video games to generate synthetic data and their corresponding ground-truths
for image segmentation and depth estimation. Using open-source animation
movies was another approach discussed by Butler et al. [25], who showed how to
obtain an optical flow large-scale dataset, MPI-Sintel, following a systematic
and easy process. Dosovitskiy et al. [27] presented CARLA, a popular open-
source simulator widely used for autonomous driving research. Recently, the
UrbanScene3D simulator was proposed by Liu et al. [4] and developed for
computer vision and robotics research. The system supports autonomous driving
and flying research in different environments. GANcraft is another work by Hao
et al. [28] for generating photorealistic images.

Despite the great advancements of the previous simulators, they present a
partial solution for the data generation issue because of the lack of control
on the environmental elements. They also fail to randomize the scene elements
which will lead to some clear repetitions (to scene elements) when a large-
scale dataset is required to be generated by these methods. In our simulator,
we utilize procedural content generation to generate 3D virtual worlds.
Additionally, our simulator can generate a special training data for the task
of video stabilization. In our experiments, we show that generating
appropriate training data and creating a pipeline that utilizes synthetic
data, can achieve superior results. In fact, our algorithm can still provide
better results in real videos even though our engine does not generate state-
of-the-art photo-realistic videos. Additionally, our synthetic aware algorithm
and our specially designed synthetic data both contribute to teaching the
model to accurately estimate the affine transformation while not overfitting
to the synthetic data distribution being trained on. Thus, it mitigates the
domain gap and achieves satisfactory results on real data.

##  3 Methodology

Let ${V=\\{v_{1},v_{2},\dots,v_{N}\\}}$ be a shaky video composed of $N$
frames. The aim of our approach is to generate a stabilized version
${V^{\prime}=\\{v^{\prime}_{1},v^{\prime}_{2},\dots,v^{\prime}_{N}\\}}$ while
preserving the original camera movement made by the recorder and maintaining
its trend. Our proposed method is composed of two major stages: i) Motion
Estimation; and ii) Trajectory Smoothing. In the first stage, we train a
motion estimation network using the ground-truth data from generated synthetic
videos to estimate an affine transformation matrix $\mathbf{A}_{i}$ for every
consecutive frames $v_{i}$ and $v_{i+1}$. Then, in the second stage, we
calculate the camera trajectory
${\hat{T}=\\{\hat{t}_{1},\hat{t}_{2},\dots,\hat{t}_{N}\\}}$, where
$\hat{t}_{i}=\sum_{j=1}^{i}\hat{\mathbf{x}}_{j}$ and $\hat{\mathbf{x}}_{j}$
represents the estimated parameters for the pair of frames
${(v_{j},v_{j+1})}$. Following this and after smoothing $\hat{T}$, we warp and
crop frames using the smoothed transformations retrieved from the smoothed
trajectory
${\tilde{T}=\\{\tilde{t}_{1},\tilde{t}_{2},\dots,\tilde{t}_{N}\\}}$. An
outline of our approach is described in Figure 2.

Figure 2: Video stabilization pipeline. Our method estimates the translation,
rotation, and scale for each pair of frames of the shaky video. After
computing the camera trajectory, upper (red) and lower (green) bounds are
found and averaged, and the Savitzky-Golay filter is applied to smooth the
trajectory. Finally, warping and cropping is performed to generated the
stabilized video.

###  3.1 Motion Estimation

The first stage of our pipeline consists in estimating the camera motion
throughout the video. Most existing 2D-based stabilization approaches apply
key-point feature extraction and tracking to solve this task [29, 13, 30].
However, both feature extraction and tracking may fail under adverse weather
conditions due to repetitive textures and partial occlusions caused by rain
and snow particles or textureless objects under foggy weather and at night-
time. To overcome this issue and properly recover the camera motion in $V$, we
propose estimating parameters $t_{x}$, $t_{y}$, $\theta$, and $s$ of an affine
transformation matrix

$\mathbf{A}=\begin{bmatrix}s\cos{\theta}&-s\sin{\theta}&t_{x}\\\
s\sin{\theta}&s\cos{\theta}&t_{y}\end{bmatrix}$ (1)

for every consecutive pairs of frames using deep neural networks with
synthetic data. Thus, we abdicate the feature extraction procedure entirely
since, using our proposed engine, we can generate the ground-truth affine
transformation needed for training as described in Section 3.3.

Two identical networks are utilized: the ${f_{tr}:\mathbb{R}^{4\times W\times
H}\rightarrow\mathbb{R}^{2}}$ network that estimates the $x$ and $y$
translations, i.e., ${\mathbf{x}^{tr}=[t_{x},t_{y}]}$; and
${f_{rs}:\mathbb{R}^{4\times W\times H}\rightarrow\mathbb{R}^{2}}$ that
outputs the rotation angle and scale values ${\mathbf{x}^{rs}=[\theta,s]}$,
where $W=H=256$ is the center cropped image width and height. It is important
to note that $f_{tr}$ and $f_{rs}$ share the same architecture, but not the
same weights.

Both networks consist of a feature extractor implemented as four convolutional
layers followed by a pooling and a dropout layer, and a regressor, which is a
fully connected network composed of three linear layers that process the
extracted features to estimate the parameters. The number of output channels
of each layer is illustrated in Figure 2-a. For each training step, we feed
the networks with an input ${I=[v_{i};v_{i+1};O_{i}]}\in\mathbb{R}^{4\times
W\times H}$, where ${v_{i},v_{i+1}\in\mathbb{R}^{W\times H}}$ are two
consecutive grayscale frames from the input video $V$ and
${O_{i}\in\mathbb{R}^{2\times W\times H}}$ is the optical flow map for the
pair $(v_{i},v_{i+1})$ obtained via the FlowNet2 [31, 32]. Then, we estimate
the parameters for $\mathbf{A}_{i}$ as ${\hat{\mathbf{x}}^{tr}=f_{tr}(I)}$ and
${\hat{\mathbf{x}}^{rs}=f_{rs}(I)}$.

To optimize the parameters of the networks $f_{tr}$ and $f_{rs}$, we train
separately each one using the Mean Squared Error (MSE) loss functions:

$\mathcal{L}_{tr}=\frac{1}{M}\sum_{m=1}^{M}(\hat{\mathbf{x}}^{tr}-\mathbf{x}^{tr})^{2}~{}\mathrm{and}~{}\mathcal{L}_{rs}=\frac{1}{M}\sum_{m=1}^{M}(\hat{\mathbf{x}}^{rs}-\mathbf{x}^{rs})^{2},$
(2)

where $M$ is the number of training samples in a batch of randomly selected
consecutive pair of frames to compose $I$. $\mathbf{x}^{tr}$ and
$\mathbf{x}^{rs}$ are the ground-truth parameters of the affine matrix
$\mathbf{A}_{i}$.

A key contribution of our approach is the usage of specially-designed
synthetic data to learn affine transformation. Let
$\mathcal{P}_{i}=\\{\mathbf{p}_{1},\mathbf{p}_{2},\dots,\mathbf{p}_{K}\\}$
denote the 2D coordinates of $K$ mark points at the frame $v_{i}$ from a
generated synthetic video. Since these mark points in frame $v_{i}$ and
$v_{i+1}$ are static in the world space, we can compute an affine
transformation $\mathbf{A}_{i}$ with $4$ degrees of freedom using
$\mathcal{P}_{i}$ and $\mathcal{P}_{i+1}$. Following this strategy, we get the
ground-truth values $t_{x}$, $t_{y}$, $\theta$, and $s$ for a single pair of
images. The detailed process of the ground-truth data generation is described
in Section 3.3.

Finally, with the estimated parameters
$\hat{\mathbf{x}}=[\hat{t}_{x},\hat{t}_{y},\hat{\theta},\hat{s}]$ for each
image pair in the video, we can compute the estimated camera trajectory
${\hat{T}=\\{\hat{t}_{1},\hat{t}_{2},\dots,\hat{t}_{N-1}\\}}$, where
$\hat{t}_{i}=\sum_{j=1}^{i}\hat{\mathbf{x}}_{j}$, and $\hat{\mathbf{x}}_{j}$
represents the estimated parameters for the pair of frames
${(v_{j},v_{j+1})}$. It is important to note that, similar to Grundmann et al.
[13], we do not use $\hat{t}_{i}$ directly to warp the shaky images; we warp
the frames by applying smoothed affine transformation composed using the
smoothed translation, ration, and scale parameters as detailed in the
following section.

###  3.2 Trajectory Smoothing

After estimating the shaky camera trajectory, we need to smooth it. In
contrast to other methods like the work of Grundmann et al. [13], where they
tackle the camera trajectory smoothing as an optimization problem, we deploy
the Savitzky-Golay filter [33] on the averaged envelop of the shaky camera
trajectory to smooth it, as described in the sequel.

#### Signal Envelop Calculation

Given the camera trajectory $\hat{T}$, we first calculate the extremes of
$\hat{T}$ by applying the first-order discrete derivative. Then, we
interpolate the trajectory maxima ($\hat{T}_{max}$) and minima
($\hat{T}_{min}$) values to extract the upper and lower envelop, respectively.
We empirically experimented with linear, cubic, and quadratic interpolations.
However, quadratic interpolation presented the best results in our experiments
since it makes smooth interpolations and tends to stay within the ranges of
the interpolation points. The final upper and lower signal envelops are
represented as ${E_{up}=\\{e^{up}_{1},e^{up}_{2},\dots,e^{up}_{N-1}\\}}$ and
${E_{low}=\\{e^{low}_{1},e^{low}_{2},\dots,e^{low}_{N-1}\\}}$, respectively.

#### Smoothing

After obtaining the upper and lower signal envelops, we apply the Savitzky-
Golay filter [33] on the average envelop ${\bar{E}=(E_{up}+E_{low})/2}$ to
remove the unwanted sudden camera shakiness and create the smooth camera
trajectory $\tilde{T}=\\{\tilde{t}_{1},\tilde{t}_{2},\dots,\tilde{t}_{N-1}\\}$
as shown in Figure 2-b. The Savitzky-Golay filter smooths the digital signal
by fitting a low-degree polynomial to consecutive signal points using linear
least squares. This strategy has an advantage over other techniques as it
preserves the signal tendency. Thus, $\tilde{T}$ still maintains the
properties of $\hat{T}$ while ensuring smooth camera transition over time.
After that, we calculate the difference between both trajectories
$\delta_{T}=\hat{T}-\tilde{T}$. Then, the smoothed affine transformation
parameters $\tilde{X}$ can be calculated as $\tilde{X}=\hat{X}-\delta_{T}$,
where
$\tilde{X}=\\{\tilde{\mathbf{x}}_{1},\tilde{\mathbf{x}}_{2},\dots,\tilde{\mathbf{x}}_{N-1}\\}$
and
$\hat{X}=\\{\hat{\mathbf{x}}_{1},\hat{\mathbf{x}}_{2},\dots,\hat{\mathbf{x}}_{N-1}\\}$,
with $\tilde{\mathbf{x}}_{i}$ being the smoothed parameters $\tilde{t}_{x}$,
$\tilde{t}_{y}$, $\tilde{\theta}$, and $\tilde{s}$.

#### Warping and Cropping

At last, we warp the video frames and crop them to compose the final video.
For each video frame $v_{i}$, we compute its warped version $\tilde{v}_{i}$ by
applying a transformation matrix to every pixel. Formally, we retrieve
$\tilde{\mathbf{x}}_{i}$ from the smoothed transformations
$\tilde{\mathbf{X}}$ and use the smoothed parameters $\tilde{t}_{x}$,
$\tilde{t}_{y}$, $\tilde{\theta}$, and $\tilde{s}$ to compose the smoothed
affine matrix $\tilde{\mathbf{A}}_{i}$. Finally, we crop the warped frames
using a predefined virtual cropping window similar to Grundmann et al. [13] to
generate the stabilized video.

Figure 3: Ground-truth Generation. $K$ points are randomly sampled from the
screen space (yellow circles in a). From each of these points we cast rays to
infinity in the 3D scene space (dashed yellow lines in b), and create
hypothetical objects at the intersection of these rays with the scene (red
circles in c).We obtain the affine transformation matrix $A_{i}$ using the
coordinates of the hypothetical objects in screen space $\mathcal{P}_{i}$ and
$\mathcal{P}_{i+1}$ since they remain stationary in the scene from the frame
$v_{i}$ to $v_{i+1}$.

###  3.3 Ground-truth Generation

The goal of our network is to infer the affine transformation matrix given two
consecutive frames. Thus, we need to generate the affine transformation
ground-truth to supervise the training process. Our idea is two-fold: a)
create stationary hypothetical labeled objects in the 3D world scene; b)
record their coordinates in the screen space of the recording camera. In that
way, we can guarantee that the coordinates of these objects in frames
${(v_{i},v_{i+1})}$ correspond to exactly the same static elements in the 3D
world seen by the recording camera at frames $v_{i}$ and $v_{i+1}$. In other
words, we create a number of invisible objects and save their coordinates in
the camera space for each frame. In order to do so, for each frame a number of
random points are sampled from the screen camera space. Then, a ray is cast
from each of these points to infinity. At each ray’s intersection point with
the scene, a hypothetical invisible object is created. The object remains
stationary for a number of seconds before being destroyed. For each frame, the
object position in world space is transferred to the camera space and saved in
XML format. Figure 3 demonstrates how these hypothetical objects are created.

If the current camera view does not include the hypothetical object, or if it
exceeds its time limit duration $\beta$, it is removed. Each object is given a
Unique Identifier (UID) over its lifetime. Later in the post processing stage,
for each two consecutive frames ${(v_{i},v_{i+1})}$ using the UIDs of these
hypthetical objects and their screen locations, the affine transformation
ground-truth is calculated for each pair ${(v_{i},v_{i+1})}$ as described in
Algorithm 1. It should be noted that generating millions of hypothetical
objects is still a valid option. However, our algorithm provides a solution to
create only objects in the field view of the recording agent improving the
overall performance.

Require : $N_{frames}$, $T$

Ensure : Per frame $XML~{}file$ containing camera screen locations of a number
of hypothetical objects stationary in scene space.

$N_{frames};$ $\triangleright$ Total Number of Frames to Generate

$T;$ $\triangleright$ Sampling Period

while  _Recording_ do

for  _$ID_{frame}=0;\ ID_{frame} <N_{frames};\ ID_{frame}{+}{+}$_ do

if  _$ID_{frame}\%T==0$_ then

$Points$ $\leftarrow$ $samplePoints($K$);$ $\triangleright$ Sampling $K$
points on camera screen space

foreach  _point $\in$ $Points$ _ do

$M$ $\leftarrow$ $castRayToInfinity($point$);$ $\triangleright$ Cast a ray
from $point$ to infinity

$point_{inters}$ $\leftarrow$ $findIntersPoint(M);$ $\triangleright$
Intersection point between the ray with scene objects

$O$ $\leftarrow$ $createHyptheticalObject(point_{inters});$

$O.UID$ $\leftarrow$ $assignObjectUID(O);$ $\triangleright$ Assign $O$ a
unique identifier

$O.ScreenPos\leftarrow Cam.WorldToScreen(O.WorldPos);$ $\triangleright$
Transfer coordinates from world to screen space

while  _ $O$ is visible and did not exceed its lifetime _ do

$Save(O.ScreenPos);$

$WaitFewFrames();$

$Destroy(O);$

else

$WaitFewFrames();$

Algorithm 1 Affine Transfromation Ground-truth Generation

##  4 Silver: Framework for Generating Synthetic Data for Computer Vision
Tasks

There are many photo-realistic synthetic data generators like CARLA [27] and
UrbanScene3D [4] that can be used to simulate photo-realistic, diverse, and
visually complex 3D worlds. However, generating special data in such engines
is cumbersome and most importantly, they do not support generating training
data for video stabilization. Silver, on the other hand, fills the gap and
generates the required training data for this task. Additionally, it supports
other computer vision tasks like semantic segmentation, instance segmentation,
depth estimation, pose estimation, surface normals estimation. However, in
this paper, we limit our discussion to the usability of our engine for video
stabilization task.

In this work, we show that more vital than photo-realistic and diverse 3D
scenes is designing computer vision models targeted at using synthetic data,
and generating the appropriate synthetic data for these models. In virtual
worlds, it is not only easy to control all scene aspects but also to generate
more suitable training data for supervised learning algorithms.

#### Simulator

Our work deploys synthetic data to train a synthetic-aware video stabilization
algorithm. For that aim, we developed Silver using the Unity game engine to
generate our synthetic training datasets VSAC65Synth and VSNC35Synth. We
employ the Procedural Content Generation (PCG) concept to create a full 3D
virtual world at run-time while the system’s extensibility is attained by
taking advantage of the modular approach followed as we built the system from
scratch. Although our simulator can provide clean, unbiased, and large-scale
training and testing data for various computer vision tasks, we focus on the
video stabilization task. A shaky synthetic video is recorded after
procedurally creating a 3D virtual world sampled from a predefined set of 3D
models, materials, and animations. Note that for each video, a new virtual
world is created to diversify the training data. A simplified flowchart
describing the scene creation process is shown in Figure 4.

Figure 4: Flowchart describing the scene creation in Silver. Figure 5: Samples
from the procedurally generated scenes using Silver.

#### Static Elements

Starting from the given parameters, Silver initially creates the static part
of the 3D virtual world. In this part, first the street length and the number
of crosses are set at random. Following this, the buildings are created where
buildings’ locations, types, and frequency are set at random. After that, the
other scene elements like benches, trash containers and bags, trees, and other
elements are created. To further improve the realism and diversity of the
generated scenes, we introduce a new variable called Anomaly Rate; higher
values will cause more artifacts to appear in the scene such as more street
lights being off at night, more being on at daytime, and some trash bags being
on roads.

#### Dynamic Elements

Once the static part of the scene is completed, the dynamic part of the scene
is initiated. Initially, the characters generator retrieves the locations of
buildings and benches, and instantiates characters based on the required
characters density. The Microsoft Rocketbox Avatar Library [34] is used to
define the character avatar, and the animations are selected based on
character pose (standing or sitting). Character animations were adopted from
Mixamo. In a similar way, the cars are created. However, number of cars and
models are selected at random. Additionally, car shader attributes:
Smoothness, Metallic, and BaseColour are all randomized at run-time to give
different visual appearance even to the same car model. After that, the plates
of the cars are selected at random from a large set collected manually from
the web. The main processes are summarized in Figure 4. In parallel to that,
the first-person videos are recorded using Cinemachine camera behaviour from
Unity, attached to an AI navigation agent. We use Cinemachine since it gives
unlimited sets of behaviours that enrich the diversity of the generated
synthetic data in terms of the camera view angle and transition.

#### Camera Shakiness

To introduce shakiness to the recording camera, we create noise by using a
predefined noise profile asset. The amplitude and frequency of the noise are
randomly sampled from a uniform distribution. The noise is applied to change
the translation and rotation of the recording agent camera. Figure 5
demonstrates examples of the generated scenes.

Figure 6: VSAC105Real versus other datasets. On top, each dashed box shows frames from other datasets. Our proposed dataset, VSAC105Real, is at the bottom. It includes more diverse and challenging attributes as compared to the other datasets. Table 1: Dataset statistics. Comparison among the available video stabilization datasets and VSAC105Real dataset. Dataset Name | #Videos |  | Average
---
#Frames
| Total
---
#Frames
DeepStab [8] | $61$ | $714$ | $43{,}585$
Stabfr [35] | $45$ | $471$ | $21{,}200$
Selfie Video [36] | $33$ | $251$ | $8{,}308$
LiuSigg2013 [37] | $144$ | $578$ | $83{,}257$
VSAC105Real | $105$ | $737$ | $77{,}477$

##  5 Real and Synthetic Datasets

###  5.1 Real Data Collection

The available video stabilization benchmarks such as DeepStab [8], Stabfr
[35], Selfie Video [36], and LiuSigg2013 [37] exclusively contain videos under
normal weather condition and at a sufficient illumination. To assess the
performance of the state-of-the-art video stabilization methods under foggy,
rainy, snowy, and night-time conditions, we created the VSAC105Real dataset.

Our dataset is composed of videos collected from YouTube using search queries
like “Fog”, “Rain”, “Snow”, “Night”, “Adverse”, and “Severe”. We manually
inspected all the videos and selected the ones with shaking camera movement.
Then, we cut the videos to ensure continuous temporal criteria and the query
attribute. VSAC105Real dataset comprises $105$ videos spanning normal, rainy,
foggy, snowy, and night-time attributes. The first four attributes were
selected to study the effect of severe weather conditions on video
stabilization quality. Similarly, the night-time was chosen to understand the
effect of low illumination on video stabilization. Table 1 shows a comparison
among different video stabilization datasets and VSAC105Real dataset. The
VSAC105Real dataset has the advantage in terms of the average number of
frames. Moreover, it includes a diverse set of challenging attributes where
videos are evenly distributed across the classes, i.e., $21$ videos per class.
A visual comparison among VSAC105Real and other video stabilization datasets
is depicted in Figure 6.

###  5.2 Synthetic Data

Using Silver, we generate two different synthetic training datasets:
VSNC35Synth under normal weather conditions and VSAC65Synth under both normal
and adverse weather conditions.

#### VSNC35Synth dataset

It is used in all training experiments, unless otherwise specified. It
includes $35$ videos at $24$ fps and $400$ average number of frames per video;
it covers only videos in normal weather conditions. The average number of
frames was set to $400$ to match the available real video stabilization
dataset average number of frames.

#### VSAC65Synth dataset

It is used in one of the ablation study’s experiments (Table 4, under More
Data column). It consists of $65$ videos spanning normal, rainy, foggy, and
snowy weather conditions at daytime and nigh-time. It has the same fps and
average number of frames as the VSNC35Synth dataset.

##  6 Experiments

###  6.1 Experimental Setup

#### Implementation Details

We trained our method using only the synthetic data provided by our simulator,
i.e., VSNC35Synth. We used the Adam optimizer [38] with a learning rate
${\alpha=1e\mathrm{-}4}$ with ${\beta_{1}=0.9}$, ${\beta_{2}=0.999}$, and
${\epsilon=1e\mathrm{-}8}$. Additionally, the hypothetical objects time limit
duration was $\beta=1$ second. We trained the ${t}_{x}$ and ${t}_{y}$
translation prediction model ($f_{tr}$) and rotation ${\theta}$ and scale
${s}$ prediction model ($f_{rs}$) for $65$ and $2$ epochs, respectively, using
batches of size ${M=40}$. After $10$ epochs, we decrease the learning rate of
$f_{tr}$ to ${1e\mathrm{-}5}$. Our architecture is fully implemented in
PyTorch, and our whole training procedure takes about $33$ hours on a Tesla
V100 GPU. For the smoothing step, we used a window length, i.e., number of
coefficients, equal to $51$ with $1$st order polynomial as parameters to the
Savitzky-Golay filter. We experimentally selected these values since they give
slightly better results than others.

#### Video Stabilization Methods

We evaluate five state-of-the-art video stabilization algorithms on two
datasets: i.e., VSAC105Real and Selfie Video [36]. The baselines span non-
learning based [13], supervised [8], and unsupervised [15] video stabilization
methods. Furthermore, we compare our method to the one proposed by Yu et al.
[10], which heavily relies on optical flow since we also use optical flow.
Similarly, we consider the work of Liu et al. [9] because they also use CNNs
for video stabilization close to our work.

Table 2: Comparison across different weather conditions in the VSAC105Real dataset. Our method presents the best average values in comparison to the other competitors for all metrics. Bold indicates the best and underline second best. Metric | Method |  | Weather Condition | Average
---|---|---|---|---
| Fog | Night | Normal | Rain | Snow
Stability Avg. Score $\uparrow$ | FuSta [9] |  | $0.226$ | $0.683$ | $0.715$ | $0.679$ | $0.824$ | $0.626$
Grundmann et al. [13] |  | $0.642$ | $0.549$ | $0.620$ | $0.580$ | $0.809$ | $\underline{0.640}$
StabNet [8] |  | $0.201$ | $0.469$ | $0.620$ | $0.577$ | $0.753$ | $0.524$
DIFRINT [15] |  | $0.121$ | $0.212$ | $0.321$ | $0.247$ | $0.446$ | $0.270$
Yu et al. [10] |  | $0.401$ | $0.682$ | $0.665$ | $0.572$ | $0.834$ | $0.631$
| Ours |  | $0.606$ | $0.619$ | $0.728$ | $0.687$ | $0.835$ | $\mathbf{0.695}$
Distortion Score $\ast$ | FuSta [9] |  | $0.202$ | $0.692$ | $0.725$ | $0.712$ | $0.798$ | $0.626$
Grundmann et al. [13] |  | $0.740$ | $0.617$ | $0.762$ | $0.667$ | $0.952$ | $\underline{0.748}$
StabNet [8] |  | $0.111$ | $0.518$ | $0.790$ | $0.597$ | $0.710$ | $0.545$
DIFRINT [15] |  | $0.367$ | $0.219$ | $0.351$ | $0.270$ | $0.476$ | $0.337$
Yu et al. [10] |  | $0.372$ | $0.729$ | $0.654$ | $0.593$ | $0.804$ | $0.631$
| Ours |  | $0.719$ | $0.746$ | $0.952$ | $0.809$ | $0.997$ | $\mathbf{0.845}$
Cropping Ratio $\uparrow$ | FuSta [9] |  | $0.286$ | $0.810$ | $0.905$ | $0.800$ | $0.950$ | $\underline{0.751}$
Grundmann et al. [13] |  | $0.759$ | $0.618$ | $0.760$ | $0.663$ | $0.948$ | $0.750$
StabNet [8] |  | $0.278$ | $0.579$ | $0.875$ | $0.667$ | $0.850$ | $0.650$
DIFRINT [15] |  | $0.399$ | $0.234$ | $0.392$ | $0.280$ | $0.490$ | $0.359$
Yu et al. [10] |  | $0.476$ | $0.810$ | $0.842$ | $0.650$ | $0.947$ | $0.745$
| Ours |  | $0.762$ | $0.760$ | $0.945$ | $0.820$ | $0.999$ | $\mathbf{0.857}$
Success Rate $\uparrow$ | FuSta [9] |  | $0.280$ | $0.816$ | $0.905$ | $0.762$ | $0.905$ | $0.734$
Grundmann et al. [13] |  | $0.762$ | $0.619$ | $0.762$ | $0.667$ | $0.952$ | $\underline{0.752}$
StabNet [8] |  | $0.238$ | $0.524$ | $0.667$ | $0.571$ | $0.810$ | $0.562$
DIFRINT [15] |  | $0.429$ | $0.238$ | $0.381$ | $0.286$ | $0.476$ | $0.362$
Yu et al. [10] |  | $0.480$ | $0.815$ | $0.762$ | $0.619$ | $0.857$ | $0.707$
| Ours |  | $0.765$ | $0.762$ | $0.950$ | $0.814$ | $1.000$ | $\mathbf{0.858}$
|  |  |  $\uparrow$Higher is better ∗Better closer to $1$

#### Evaluation Metrics

To evaluate our approach, we use three metrics commonly used to evaluate video
stabilization algorithms [9, 15, 8, 10]: i) Stability Score. It assesses the
smoothness of the stabilized video; the higher the value the better. It is
computed as the average between Stability Average Translation and Stability
Average Rotation Scores. To compute this score, we estimate the homography
matrix between $v_{i}$ and $v_{i+1}$ to obtain the translation and rotation
arrays. Following this, we calculate their Fast Fourier Transform (FFT).
Finally, we obtain the score by calculating the ratio between the $2$nd
through $6$th frequency components and all frequency components. Note that the
$0$th frequency component is neglected; ii) Distortion Score. It measures the
global distortion caused by a given video stabilization method. It fits a
homography matrix between the original and stabilized videos. Then, it finds
the anisotropic scaling among these frames; the closer to $1$, the better;
iii) Cropping Ratio. It describes the ratio of the remaining frame’s area
after stabilization to the original one; iv) Success Rate. We also measure the
success rate, which computes the ratio of videos that were successfully
processed and yielded a distortion score lower than or equal to one.

Table 3: Comparison in the Selfie Video dataset [36]. Our method achieves the best distortion score and comparable results to other baselines. Bold indicates the best, underline second best, and italic the third best result.  Method |  | Stability
---
Avg. Score$\uparrow$
| Distortion
---
Score$\ast$
| Cropping
---
Ratio$\uparrow$
| Success
---
Rate$\uparrow$
FuSta [9] | $0.818$ | $\mathit{0.777}$ | $\mathbf{0.970}$ | $\mathbf{0.970}$
Grundmann et al. [13] | $0.727$ | $0.828$ | $0.848$ | $0.848$
StabNet [8] | $0.763$ | $0.680$ | $\mathit{0.917}$ | $0.667$
DIFRINT [15] | $\mathbf{0.827}$ | $0.691$ | $0.912$ | $\mathit{0.915}$
Yu et al. [10] | $0.770$ | $0.739$ | $0.909$ | $0.909$
Ours | $\mathit{0.787}$ | $\mathbf{0.933}$ | $0.939$ | $0.939$
$\uparrow$Higher is better ∗Better closer to $1$

###  6.2 Results

Table 2 shows the results of comparing our method to several video
stabilization approaches. As can be seen, our method presented the best values
on average in comparison to all the baselines in terms of stability average,
distortion, cropping ratio, and success rate. Even though our method did not
surpass the baselines at each class individually, it still achieved
competitive results. Every baseline performs badly in at least one class,
while our method is more robust across classes, hence holding the final best
results on the VSAC105Real.

Preserving the content while compensating for camera shakiness is another
important feature of our algorithm. Our method achieved the best results as
compared to other state-of-the-art methods. The superiority of our method can
be linked to the accurate affine transformation matrix estimation and the
smoothing stage. Moreover, our method achieved the highest success rate
compared to the competitors as shown in Table 2. It is worth noting that all
baselines failed to stabilize most of the shaky videos at foggy weather
conditions. The reason for this outcome is that participating media, like fog,
work as a low pass filter that removes high-quality features that most videos
stabilization algorithms depend on to estimate the camera trajectory. We
highlight that even though our model was not trained on any samples under the
foggy weather condition, it was capable of learning useful features from both
raw images and optical flow.

Furthermore, we evaluate our model on Selfie Video dataset [36], contains
videos under normal weather condition and standard illumination. Our model
achieves comparable results to other methods while maintaining the best
distortion score as shown in Table 3. It should be noted that our model was
trained from scratch on synthetic data only.

###  6.3 Ablation Study

Table 4: Ablation study. Performance for different design choices (best in bold). |  | Single
---
Network
SIFT |  | More
---
Data
| No Optical
---
Flow
|  $l_{1}$ Directed
---
Smoothing
| Complete
---
Model
Stability Avg. Score $\uparrow$ | $0.443$ | $0.576$ | $0.690$ | $0.678$ | $0.675$ | $\mathbf{0.695}$
Distortion Score $\ast$ | $0.540$ | $0.650$ | $0.793$ | $0.781$ | $0.828$ | $\mathbf{0.845}$
Cropping Ratio $\uparrow$ | $0.557$ | $0.670$ | $0.850$ | $0.829$ | $0.844$ | $\mathbf{0.857}$
Success Rate $\uparrow$ | $0.543$ | $0.667$ | $0.829$ | $0.800$ | $0.838$ | $\mathbf{0.840}$
$\uparrow$Higher is better ∗Better closer to $1$

We analyzed different design options and showed the effectiveness of each
component of our proposed pipeline. The results are reported in Table 4.
First, since we use two networks in our pipeline, we trained using a single
CNN with four losses: horizontal translation loss, vertical translation loss,
rotation loss, and scale loss. As a result, the network was unable to converge
well. One problem could be that the translation losses were larger than
others. However, even after applying weights for the losses to make them
comparable to each other, the network could not learn well (column Single
Network in Table 4).

To emphasize the advantages of using our learning-based model, for affine
transformation matrix estimation over applying SIFT, we apply SIFT to find the
affine transformation matrix while keeping the smoothing part of our model
intact. As expected, SIFT did not perform well (column SIFT in Table 4).
Standard feature extractors like SIFT struggle to extract reliable and robust
features under adverse conditions. Rain and snow particles, low illumination
at night, and foggy weather make finding and matching features rather hard.
This leads to inaccurate affine transformations, thus low-quality stabilized
videos. To further evaluate the advantages of using our smoothing algorithm
over $l_{1}$ directed smoothing, as done in [13], we apply $l_{1}$ directed
smoothing on the predicted camera trajectory, while keeping our learning-based
model for the affine transformation matrix estimation. As expected, the model
does not preform very well as compared to using our proposed smoothing
algorithm (column $l_{1}$ directed smoothing in Table 4) because ours
considers more sophisticated camera paths and is not limited to constant,
linear and parabolic motions like [13].

To show the need for optical flow in the affine transformation learning, we
train using only graysale images rather than both grayscale and optical flow
modalities. As anticipated, not utilizing optical flow information decreased
the video stabilization quality (column No Optical Flow in Table 4).

At last, to demonstrate the effect of our training synthetic data on the
quality of video stabilization, we trained our model on more data, including
normal weather, adverse weathers, and night-time videos. The training was
performed using VSAC65Synth dataset from scratch, and no real data were used.
The results indicated no significant improvement on the baseline that was
trained on VSNC35Synth dataset. Thus, we can see in Table 4, under More Data
column, that a few synthetic videos, with accurate ground-truth, were
sufficient to train our model to learn affine transformation. Thus, increasing
the number and diversity of the training videos did not boost the overall
performance.

Table 5:  Affine matrix estimation. Comparison among different methods for affine matrix estimation on CA-Unsupervised dataset [21] using $l_{2}$ distance. Bold indicates the best result.  Method |  |  | RE$\downarrow$ |  |  | LT$\downarrow$ |  |  | LL$\downarrow$ |  |  | SF$\downarrow$ |  |  | LF$\downarrow$ |  |  | Average$\downarrow$
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
ORB [16] \+ RANSAC [19] |  |  | $9.24$ |  |  | $14.63$ |  |  | $12.27$ |  |  | $11.36$ |  |  | $7.20$ |  |  | $10.94$
ORB [16] \+ MAGSAC [20] |  |  | $10.11$ |  |  | $19.79$ |  |  | $12.48$ |  |  | $11.86$ |  |  | $7.85$ |  |  | $12.42$
ORB [16] \+ LMEDS |  |  | $9.78$ |  |  | $40.11$ |  |  | $12.02$ |  |  | $10.84$ |  |  | $7.01$ |  |  | $15.95$
SIFT [1] \+ RANSAC [19] |  |  | $10.63$ |  |  | $11.70$ |  |  | $13.37$ |  |  | $11.75$ |  |  | $6.44$ |  |  | $10.78$
SIFT [1] \+ MAGSAC [20] |  |  | $10.75$ |  |  | $10.97$ |  |  | $12.99$ |  |  | $11.09$ |  |  | $6.35$ |  |  | $10.43$
SIFT [1] \+ LMEDS |  |  | $10.47$ |  |  | $9.72$ |  |  | $13.04$ |  |  | $10.14$ |  |  | $5.88$ |  |  | $9.85$
Supervised [22] |  |  | $8.39$ |  |  | $9.33$ |  |  | $8.63$ |  |  | $10.29$ |  |  | $5.92$ |  |  | $8.51$
Unsupervised [21] |  |  | $7.05$ |  |  | $7.60$ |  |  | $6.84$ |  |  | $7.42$ |  |  | 3.84 |  |  | $6.55$
Ours |  |  | 4.55 |  |  | 5.58 |  |  | 5.68 |  |  | 5.17 |  |  | $9.73$ |  |  | 6.14
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  $\downarrow$Lower is better

To further investigate the accuracy of our estimated affine transformation
matrix, we carry a special set of experiments shown in Table 5. We compare our
learning-based affine transformation estimation model to three types of affine
estimation approaches: i) the traditional ones, including ORB and SIFT with
RANSAC, MAGSAC, and LMEDS for outliers rejection; ii) supervised; and iii)
unsupervised approaches. We note that traditional approaches can be used to
estimate the affine transformation directly, but supervised and unsupervised
methods are designed to estimate the homography matrix. Thus, for fair
comparison, we extract the affine transformation from the estimated homography
one. We utilize the dataset of [21] which contains $4{,}200$ pairs of images
where each image-pair includes 6-matching human-annotated pairs of points. The
dataset covers regular-texture (RE), low-texture (LT), low-light (LL), small-
foregrounds (SF), and large-foregrounds (LF). We use $l_{2}$ distance to
measure the error between the warped and ground-truth points similar to [21,
23]. Table 5 demonstrates that our method can estimate the affine
transformation better for both standard and challenging conditions. Figure 7
shows a qualitative comparison among our model, traditional (e.g., SIFT and
ORB), supervised [22] and unsupervised [21] methods. While other methods fail
under such challenging conditions, our method performs well because it learned
how to extract resilient features using our specially designed synthetic data.

Figure 7: Qualitative comparison for affine transformation. Comparison between
unsupervised [21], supervised [22], SIFT [1], ORB [16] and ours for affine
transformation estimation. Our method can handle challenging conditions such
as low textures where other methods perform poorly.

##  7 Conclusion and Limitations

We showed that most of the state-of-the-art video stabilization methods could
not perform well under adverse weather conditions. We proposed a novel
synthetic-aware video stabilization algorithm that requires only synthetic
data for training. Our experimental results demonstrated that our method
surpasses all other baselines. We also provided one real dataset (VSAC105Real)
for video stabilization under adverse conditions and two synthetic datasets
(VSNC35Synth and VSAC65Synth) for training purposes. Our ablation studies
demonstrated that current affine transformation matrix estimation methods fail
under challenging conditions. Despite the fact that repetitive textures caused
by rain and snowdrops, along with smoothing filters caused by fog and the
absence of enough features at night altogether, pose certain challenges on
current video stabilization algorithms, our video stabilization performed well
under these challenges.

Although our approach achieved state-of-the-art results, it has a few
limitations. Increasing the number of synthetic training samples did not boost
the performance of our video stabilization method. The reason could be that
our model has already learned how to estimate the affine transformation
matrix, and any further training videos were not advantageous. At the same
time, our affine estimation model did not perform well under straightforward
scenarios such as large-foreground images. Additionally, it achieved
comparable results on selfie video dataset. That is due to the fact that our
generated synthetic dataset did not include any selfie videos. Even though,
our video stabilization algorithm obtained satisfactory results. Hence, we
have demonstrated that synthetic data can be very useful, but not only as an
additional data source, as how it was usually expected, but also as an
essential element for novel synthetic-aware computer vision algorithms.

## References

  * [1] David G Lowe.  Distinctive image features from scale-invariant keypoints.  International journal of computer vision, 60(2):91–110, 2004.
  * [2] Jerome Revaud, Philippe Weinzaepfel, César Roberto de Souza, and Martin Humenberger.  R2D2: repeatable and reliable detector and descriptor.  In NeurIPS, 2019.
  * [3] Zixin Luo, Lei Zhou, Xuyang Bai, Hongkai Chen, Jiahui Zhang, Yao Yao, Shiwei Li, Tian Fang, and Long Quan.  Aslfeat: Learning local features of accurate shape and localization.  Computer Vision and Pattern Recognition (CVPR), 2020.
  * [4] Yilin Liu, Fuyou Xue, and Hui Huang.  Urbanscene3d: A large scale urban scene dataset and simulator.  arXiv preprint arXiv:2107.04286, 2021.
  * [5] Abdulrahman Kerim, Ufuk Celikcan, Erkut Erdem, and Aykut Erdem.  Using synthetic data for person tracking under adverse weather conditions.  Image and Vision Computing, 111:104187, 2021.
  * [6] Alireza Shafaei, James J Little, and Mark Schmidt.  Play and Learn: Using Video Games to Train Computer Vision Models.  arXiv:1608.01745, 2016.
  * [7] Apostolia Tsirikoglou.  Synthetic data for visual machine learning: A data-centric approach.  PhD thesis, Linköping University Electronic Press, 2022.
  * [8] Miao Wang, Guo-Ye Yang, Jin-Kun Lin, Song-Hai Zhang, Ariel Shamir, Shao-Ping Lu, and Shi-Min Hu.  Deep online video stabilization with multi-grid warping transformation learning.  IEEE Transactions on Image Processing, 28(5):2283–2292, 2018.
  * [9] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang.  Hybrid neural fusion for full-frame video stabilization.  arXiv preprint arXiv:2102.06205, 2021.
  * [10] Jiyang Yu and Ravi Ramamoorthi.  Learning video stabilization using optical flow.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8159–8167, 2020.
  * [11] Muhammad Kashif Ali, Sangjoon Yu, and Tae Hyun Kim.  Learning deep video stabilization without optical flow.  arXiv preprint arXiv:2011.09697, 2020.
  * [12] Abdulrahman Kerim, Leandro Soriano Marcolino, and Richard Jiang.  Silver: Novel rendering engine for data hungry computer vision models.  In 2nd International Workshop on Data Quality Assessment for Machine Learning, 2021.
  * [13] Matthias Grundmann, Vivek Kwatra, and Irfan Essa.  Auto-directed video stabilization with robust l1 optimal camera paths.  In CVPR 2011, pages 225–232. IEEE, 2011.
  * [14] Arwen Bradley, Jason Klivington, Joseph Triscari, and Rudolph van der Merwe.  Cinematic-L1 Video Stabilization with a Log-Homography Model.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1041–1049, 2021.
  * [15] Jinsoo Choi and In So Kweon.  Deep iterative frame interpolation for full-frame video stabilization.  ACM Transactions on Graphics (TOG), 39(1):1–9, 2020.
  * [16] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski.  ORB: An efficient alternative to SIFT or SURF.  In 2011 International conference on computer vision, pages 2564–2571. Ieee, 2011.
  * [17] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool.  SURF: Speeded up robust features.  In European conference on computer vision, pages 404–417. Springer, 2006.
  * [18] Jiahui Zhang, Dawei Sun, Zixin Luo, Anbang Yao, Lei Zhou, Tianwei Shen, Yurong Chen, Long Quan, and Hongen Liao.  Learning two-view correspondences and geometry using order-aware network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5845–5854, 2019.
  * [19] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [20] Daniel Barath, Jiri Matas, and Jana Noskova.  MAGSAC: marginalizing sample consensus.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10197–10205, 2019.
  * [21] Jirong Zhang, Chuan Wang, Shuaicheng Liu, Lanpeng Jia, Nianjin Ye, Jue Wang, Ji Zhou, and Jian Sun.  Content-aware unsupervised deep homography estimation.  In European Conference on Computer Vision, pages 653–669. Springer, 2020.
  * [22] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  Deep image homography estimation.  arXiv preprint arXiv:1606.03798, 2016.
  * [23] Nianjin Ye, Chuan Wang, Haoqiang Fan, and Shuaicheng Liu.  Motion basis learning for unsupervised deep homography estimation with subspace projection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13117–13125, 2021.
  * [24] Shiwei Li, Lu Yuan, Jian Sun, and Long Quan.  Dual-feature warping-based motion model estimation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4283–4291, 2015.
  * [25] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black.  A Naturalistic Open Source Movie for Optical Flow Evaluation.  In European conference on computer vision, 2012.
  * [26] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.  Playing for Data: Ground Truth from Computer Games.  In European conference on computer vision, 2016.
  * [27] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In Conference on robot learning, pages 1–16. PMLR, 2017.
  * [28] Zekun Hao, Arun Mallya, Serge Belongie, and Ming-Yu Liu.  GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds.  arXiv preprint arXiv:2104.07659, 2021.
  * [29] Ken-Yi Lee, Yung-Yu Chuang, Bing-Yu Chen, and Ming Ouhyoung.  Video stabilization using robust feature trajectories.  In 2009 IEEE 12th International Conference on Computer Vision, pages 1397–1404. IEEE, 2009.
  * [30] Shuaicheng Liu, Yinting Wang, Lu Yuan, Jiajun Bu, Ping Tan, and Jian Sun.  Video stabilization with a depth camera.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 89–95. IEEE, 2012.
  * [31] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017.
  * [32] Fitsum Reda, Robert Pottorff, Jon Barker, and Bryan Catanzaro.  Flownet2-pytorch: Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks.  https://github.com/NVIDIA/flownet2-pytorch, 2017.
  * [33] Abraham Savitzky and Marcel JE Golay.  Smoothing and differentiation of data by simplified least squares procedures.  Analytical chemistry, 36(8):1627–1639, 1964.
  * [34] Mar Gonzalez-Franco, Ofek, et al.  The Rocketbox Library and the Utility of Freely Available Rigged Avatars.  Frontiers in virtual reality, 1(article 561558), 2020.
  * [35] Lei Zhang, Qing-Zhuo Zheng, Hong-Kang Liu, and Hua Huang.  Full-reference stability assessment of digital video stabilization based on riemannian metric.  IEEE Transactions on Image Processing, 27(12):6051–6063, 2018.
  * [36] Jiyang Yu and Ravi Ramamoorthi.  Selfie video stabilization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 551–566, 2018.
  * [37] Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun.  Bundled camera paths for video stabilization.  ACM Transactions on Graphics (TOG), 32(4):1–10, 2013.
  * [38] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.