# Dual Role for Heterogeneity in Dynamic Fracture

Itamar Kolvin<EMAIL_ADDRESS>School of Physics, Georgia Institute of
Technology, 837 State St NW, Atlanta, 30332, GA, USA  Mokhtar Adda-Bedia
Laboratoire de Physique, CNRS, ENS de Lyon, Université de Lyon, F-69364 Lyon,
France

###### Abstract

Are heterogeneous materials harder to break than homogeneous ones? Predicting
rapid crack propagation in the presence of inhomogeneities remains a
challenge. Linear perturbations leave the net speed of fracture unchanged. We
obtain a nonlinear equation of motion for planar crack fronts to gain insight
into the dynamic fracture of moderately heterogeneous solids. The coupling of
the front geometry to the heterogeneity landscape renormalizes the energy
dissipation and the crack velocity. In materials whose toughness is weakly
sensitive to crack speed, such as silica glass, heterogeneity results in
increased dissipation and slower crack propagation. However, heterogeneity has
a reduced effect and may even facilitate fracture in strongly velocity-
dependent materials.

Fracture mechanics, heterogeneous materials, dynamic fracture, nonlinear
dynamics

Introduction. Heterogeneous materials that locally vary in their mechanical
properties are everywhere in geology [1, 2, 3, 4, 5, 6], biology [7], and
engineering [8, 9, 10]. The bulk properties of heterogeneous media, such as
the Young modulus, are known to be well-represented by macroscopic averaging
over the material microstructure [11]. In contrast, the consequences of
heterogeneity in fracture are only beginning to be ascertained. The challenge
in predicting the toughness of heterogeneous media is the need to resolve
three-dimensional crack propagation in space and time [12, 13, 14, 15, 16,
17].

Heterogeneity may produce toughening effects. Quasi-static cracks experience
local and transient arrests at tough asperities, also termed “strong pinning”
[18, 19, 20, 21]. In bulk fracture, the formation of complex out-of-plane
surface structure may be triggered by inhomogeneities [22, 23, 24, 25]. The
resultant growth in fracture surface area is associated with increased energy
dissipation [26, 25, 17]. Still, the impact of heterogeneity on the resistance
of materials to dynamic cracks remains an open question. Existing
computational methods, including the spectral boundary integral method [27],
phase field simulations [28, 29, 30, 25] and atomistic models [31, 32],
predict 3D crack motion by obtaining the full elastic fields. The need to
resolve 3D or 2D dynamic fields that are nearly singular close to the crack
tip limits the accessible system sizes. Thus, approaching the thermodynamic
limit of 3D heterogeneous fracture remains a challenge.

To make progress, we analytically approximated the elastic fields near planar
cracks at the 2nd-order in the crack front fluctuation [33]. The local balance
between the elastic energy flux and dissipation dictated a one-dimensional
equation of motion for the crack front. Solutions for cracks traversing
heterogeneous media showed that, beyond mere averaging, crack front
fluctuations renormalize the global dissipation and speed of fracture. In
materials of weakly velocity-dependent toughness, such as silica glass [34],
dynamic cracks are slower and dissipate more energy than predicted by the
average toughness. The opposite is true, however, for sufficiently rapid
cracks in materials of strongly velocity-dependent toughness, as are silicon
[35], plastics [34] and polymer gels [36, 16].

Crack front equation of motion. Dynamic cracks are governed by the balance
$G=\Gamma$, between the elastic energy per unit area flowing into the crack
tip, $G$, and the dissipation per unit area tied to the creation of new
surfaces, $\Gamma$ [13, 16, 33]. To investigate how dynamic cracks interact
with inhomogeneities, the dependence of $G$ and $\Gamma$ on the crack front
geometry, velocity, and history is needed. Consider a semi-infinite crack
lying in the $y=0$ plane of a linearly elastic solid subject to remote tensile
stresses. In the absence of heterogeneity, the crack front is straight and
propagates at velocity $V$ along the $x$ axis. Asymptotically ahead of the
crack tip, the tensile stress at the fracture plane is singular
$\sigma_{yy}\sim(x-Vt)^{-1/2}$. The energy released per unit area is
$G=G_{r}g(V)$, where $G_{r}$ depends on the loading conditions and $g(V)$ is a
universal function that regulates the flow of energy into the crack tip [37].
For a crack front whose position at time $t$ and at coordinate $z$ is
$x=Vt+f(z,t)$, energy balance becomes

$G_{r}g(V)(1+\delta G)=\Gamma(x,z;V_{\perp})\,,$ (1)

where $\Gamma(x,z;V_{\perp})$ captures the toughness heterogeneity and
dependence on the local normal crack velocity
$V_{\perp}(z,t)=(V+\partial_{t}f)/\sqrt{1+(\partial_{z}f)^{2}}$. The energy
release rate fluctuation $\delta G$ is a nonlinear functional of $f(z,t)$ and
its derivatives that may be approximated by a perturbation expansion [12, 13,
14, 38, 39, 33].

Figure 1: Crack front dynamics across a heterogeneous toughness landscape. (A)
Fracture fronts (black lines) obtained by sampling a solution of Eqs. (4) at
intervals $\Delta t=1.72$. Colors indicate the local toughness contrast
$D\eta$. A section of the periodic $xz$ fracture plane is shown. Inset:
geometry of the full fracture plane. The main panel section is highlighted in
blue. $L_{x}=1552,L_{z}=257.8$ and $\Delta z=0.1$. (B) A section of the
fracture plane corresponding to the dashed red rectangle in (A). Fronts are
$\Delta t=0.431$ apart. (C) The local velocity fluctuation of the fronts
depicted in (B) is plotted as a function of $x=Vt$ and $z$. Red dashed lines
represent the predicted front wave directions. Solution parameters: $\nu=0.3$,
$V=0.5c_{R}$, $D=0.2$, $\psi=0.5$.

How do small local variations in $\Gamma$ modify the crack dynamics? The
linear response of dynamic crack fronts to perturbations is known [13, 15,
38]. Let us decompose the crack front fluctuation into Fourier components
$\hat{f}(k,\omega)=\int\\!\mathrm{d}z\,\mathrm{d}t\,e^{-ikz-i\omega t}f(z,t)$.
For small $f$, Ramanathan and Fisher [13] obtained

$\widehat{\delta G}=-2|k|P_{1}(\omega/|k|;V,\nu)\hat{f}+\mathcal{O}(f^{2})\,,$
(2)

where $\nu$ is the Poisson ratio and $P_{1}$ is an explicit function of its
arguments [43]. Through Eq. (1), the kernel $-2|k|P_{1}$ defines the linear
response of the crack front to spatial variations in $\Gamma$. Particularly,
$P_{1}(\omega/|k|=s_{FW})=0$ for
$s_{FW}\equiv\xi(V/c_{R},\nu)\sqrt{c_{R}^{2}-V^{2}}$, where $c_{R}$ is the
Rayleigh wave speed and $\xi$ is a factor very close to, but smaller, than 1.
Thus, crack fronts support one-dimensional dispersion-less wave propagation,
whose existence was corroborated numerically and experimentally [40, 38, 41,
42]. The total dissipation, however, remains unchanged in the 1st-order.

To determine how moderate heterogeneity affects dissipation in dynamic
fracture, we extended the perturbation theory to the 2nd-order in $f$. The
three-dimensional elastodynamic fields were analytically resolved close to the
crack front and the energy release rate was computed through a self-consistent
expansion [33]. This calculation resulted in the perturbation expansion
$\widehat{\delta G}=\widehat{\delta G}_{1}[\hat{f}]+\widehat{\delta
G_{2}}[\hat{f},\hat{f}]+\mathcal{O}(f^{3})$, in which the linear part
$\widehat{\delta G}_{1}$ is given by Eq. (2), and the 2nd-order contribution
is

$\displaystyle\widehat{\delta G_{2}}$
$\displaystyle=2|k|P_{1}\\{\hat{f}*|k|P_{1}\hat{f}\\}-\frac{1}{2}\left\\{k^{2}P_{1}^{2}+iV\omega|k|P_{2}\right\\}\hat{f}*\hat{f}$
(3) $\displaystyle-$
$\displaystyle\hat{f}*\left(k^{2}P_{1}^{2}-iV\omega|k|P_{2}\right)\hat{f}+|k|P_{1}\hat{f}*|k|P_{1}\hat{f}$

where $P_{2}=P_{2}(\omega/|k|;V,\nu)$ is an explicit function of its arguments
[33], and the convolution is
$(f*g)(k,\omega)=(2\pi)^{-2}\int\\!\mathrm{d}k^{\prime}\,\mathrm{d}\omega^{\prime}f(k-k^{\prime},\omega-\omega^{\prime})g(k^{\prime},\omega^{\prime})$.
Eq. (3) reproduced known expressions at the limits of $k\rightarrow 0$ and
$\omega\rightarrow 0$ and was shown to agree with the history-dependent form
$G(z,t)=G_{r}g(V_{\perp})\left(1+H[{f(z^{\prime},t^{\prime}\leq t)}]\right)$
[13, 33, 43]. To determine the time evolution of the front in the presence of
heterogeneity, we modeled the fracture energy as a product
$\Gamma=\Gamma_{0}(V_{\perp})(1+D\eta(z,x))$ of a velocity-dependent part
$\Gamma_{0}(V)=1+\psi V$, and a fluctuating part. The heterogeneity landscape
$\eta$ was a Gaussian random field of zero mean and unit variance. The
definition of $\Gamma_{0}$ entailed an implicit choice of the energy units.
Energy balance, then, required that $\delta G=\delta\Gamma$, where
$\delta\Gamma\equiv\Gamma/\Gamma_{0}(V)-1$. To proceed, we expanded
$f=Df_{1}+D^{2}f_{2}+\mathcal{O}(D^{3})$ and solved the perturbation hierarchy
in Fourier space,

$\hat{f}_{1}=C_{ff}\hat{\eta};\;\;\;\hat{f}_{2}=C_{ff}(\widehat{\delta\Gamma}_{2}-\widehat{\delta
G}_{2}[\hat{f}_{1},\hat{f}_{1}])\,,$ (4)

where $C_{ff}=-(2|k|P_{1}+i\omega\psi)^{-1}$, except at $(k=0,\,\omega=0)$
where $C_{ff}=0$, and
$\delta\Gamma_{2}=f_{1}\partial_{x}\eta+\psi\eta\partial_{t}f_{1}-\psi\frac{V}{2}(\partial_{z}f_{1})^{2}$.

To gain insight, we numerically evaluated Eqs. (4) on a periodic rectangle
$(L_{x},L_{z})$ for an exponentially correlated heterogeneity field
$\langle\eta(z,x)\eta(z^{\prime},x^{\prime})\rangle=C(\sqrt{(z-z^{\prime})^{2}+(x-x^{\prime})^{2}})$
where $C(r)=e^{-r/\ell}$. The units of time and space were chosen such that
the shear wave speed $c_{s}=1$ and $\ell=1$. The fracture energy heterogeneity
gave rise to spatiotemporal crack front fluctuations (Fig. 1A, Supplementary
Video 1). Regions of increased (decreased) toughness locally slowed down
(accelerated) the crack front (Fig. 1B). For weakly velocity-dependent
$\Gamma_{0}$, the local velocity fluctuation
$D\partial_{t}f_{1}+D^{2}\partial_{t}f_{2}$ exhibited oblique patterns across
the fracture plane (Fig. 1C). The slopes predicted for crack front waves, $\pm
s_{FW}/\sqrt{s_{FW}^{2}+V^{2}}$, approximated those of the velocity patterns.

Figure 2: Static structure factors of the crack front fluctuations. Colors
correspond to
$D=0.0125\,\mathrm{(light)},0.025,0.05,0.1,0.2\,\mathrm{(dark)}$. Curves
averaged over 10 realizations of $\eta$. Black lines and adjacent numbers
depict the slopes of power-law curves. $\nu=0.3,\,V=0.5c_{R}$,$L_{z}=257.8$,
$L_{x}=1552$,$\Delta z=0.1$.

The heterogeneous landscape excited in-plane front roughness. To quantify the
roughness, we calculated the static structure factor (SSF) of the crack front
$S(k)=\int\mathrm{d}\omega\;|\hat{f}(k,\omega)|^{2}$. The SSF decayed for all
$k$. For small amplitude heterogeneity, $S(k)$ approximately followed a
$k^{-1}$ decay until $k\sim 0.5$ where it crossed-over to a $k^{-4}$ decay.
These scaling relations were confirmed by an analytical calculation [43]. At
higher $D$ values and weak velocity-dependence $\psi\ll 1$, the SSF decay at
short wavelengths weakened (Fig. 2A). Thus, the crack fronts in this regime
are populated by small-scale roughness that is not present in the linear
solution. For strong velocity-dependence ($\psi\gg 1$), the SSF shape was
approximately unchanged as $D$ was increased (Fig. 2B). When $S(k)\sim
k^{-\alpha}$, the scaling exponent $\alpha$ is related to the roughness
exponent $\zeta$ defined by $\sqrt{\langle(f(z+\Delta
z)-f(z))^{2}\rangle}\sim\Delta z^{\zeta}$ via the equation $\alpha=1+2\zeta$.
Accordingly, our calculations are consistent with $\zeta\sim 0$ when $D<0.1$
or $\psi>1$. For $\psi\ll 1$, we identify this exponent with the logarithmic
in-plane roughness associated with front waves [44]. For $\psi\ll 1$ and
$D\sim 0.2$, the SSF roughly follows a $\alpha\sim 1.3$ scaling which
corresponds to $\zeta\sim 0.15$. The SSF decay is then much weaker than that
observed in quasi-static fracture experiments [45], indicating a fundamental
difference with dynamic cracks.

Figure 3: Renormalization prefactors for (A) the fracture energy and (B) the
crack velocity. $L_{z}=75.2$, $L_{x}=5330.65$ (full symbols), $L_{z}=1386.4$,
$L_{x}=288.526$ (empty symbols). Inset: Variation of $\gamma_{2}$ with the
Poisson ratio $\nu$. $\psi=0.125$, $L_{z}=75.2$, $L_{x}=5330.65$.

Front fluctuations renormalize dissipation. To investigate how the 2nd-order
corrections, $\delta G_{2}$ and $\delta\Gamma_{2}$, modify the overall
dissipation, we re-examined the local energy balance, Eq. (1). The
substitution of the perturbative solution (Eqs. (4)) in the 2nd-order
expressions for $G$ and $\Gamma$ showed that $\langle\delta
G-\delta\Gamma\rangle\sim\mathcal{O}(D^{2})$, where $\langle\rangle$ denotes
ensemble averaging over realizations of $\eta$. Energy balance appeared, then,
to be globally violated. The origin of this apparent contradiction is the
decomposition of $f$ into periodic functions over the rectangle
$(L_{z},L_{x}/V)$, which precludes global changes to the crack velocity. To
recover energy balance, we add a constant velocity term by writing $f=\delta
V\,t+f_{r}$ where $\langle f_{r}\rangle=0$ and $\delta
V\sim\mathcal{O}(D^{2})$. Eqs. (4), therefore, hold for
$f_{r}=Df_{1}+D^{2}f_{2}+\mathcal{O}(D^{3})$. The renormalized crack velocity
is $V+\delta V=V+v_{2}D^{2}+\mathcal{O}(D^{3})$, where
$v_{2}=\frac{1}{\psi+2\pi_{1}}\\{\langle\left(k^{2}(|P_{1}|^{2}-P_{1}^{2})+iV\omega|k|P_{2}\right)|\hat{f}_{1}|^{2}\rangle-\gamma_{2}\\}$
and $\pi_{1}=\lim_{k\rightarrow 0}|k|P_{1}(\omega/|k|)/(i\omega)$ [33]. The
renormalization coefficient $\gamma_{2}=\langle
f_{1}\partial_{x}\eta+\psi\eta\partial_{t}f_{1}-\psi\frac{V}{2}(\partial_{z}f_{1})^{2}\rangle$
tunes the global dissipation via
$\langle\Gamma\rangle=\Gamma_{0}(V)(1+\gamma_{2}D^{2})+\mathcal{O}(D^{3})$.
The sign of $v_{2}$ determines whether heterogeneity assists or hinders
fracture.

What are the consequences of heterogeneity to the overall dissipation? To find
out, we computed $\gamma_{2}$ using an analytical integral, Eq. (S5), that was
numerically evaluated over the discretized Fourier domain $(k_{max}=\pi/\Delta
z,\omega_{max}=\pi V/\Delta z)$ where $\Delta z$ is the real space
discretization and $\Delta k=2\pi/L_{z},\,\Delta\omega=2\pi V/L_{x}$. The
fracture plane $(L_{x},L_{z})$ was made sufficiently large to obtain
approximately size-independent results. For $\psi\ll 1$, the dissipation
renormalization prefactor was positive and decreased with crack velocity (Fig.
3A). Calculations for a range of Poisson ratios $0\leq\nu\leq 0.5$ showed that
$\gamma_{2}$ depends on the material elasticity almost exclusively through the
ratio of $V$ to the Rayleigh wave speed $c_{R}$ (Fig. 3A inset). At the two
limiting velocities, $V\rightarrow 0$ and $V\rightarrow c_{R}$, $\gamma_{2}$
approached $\psi$-independent limits. Upon increasing $\psi$, $\gamma_{2}$
decreased and attained negative values in a widening range of $V$. Thus,
heterogeneity is predicted to increase dissipation in materials whose
toughness is weakly velocity-dependent. In strongly velocity-dependent
materials, however, dissipation is predicted to decrease above a
$\psi$-dependent velocity.

The renormalization of dissipation was associated with a renormalization of
crack velocity. The coefficient $v_{2}$ was negative for $\psi\ll 1$ (Fig.
3B). For larger values of $\psi$, $v_{2}$ increased and attained positive
values over a broad crack velocity range. At limiting velocities, $v_{2}$
approached a vanishing $\psi$-independent limit at $V\rightarrow c_{R}$ and a
$\psi$-dependent value at $V\rightarrow 0$. To estimate the interval
$0<D<\tilde{D}$ in which $\delta V\simeq v_{2}D^{2}$ is the leading correction
to the global crack velocity, we approximated $\tilde{D}\simeq\sqrt{\langle
f_{1}^{2}\rangle/\langle f_{2}^{2}\rangle}$, where $f_{1},\,f_{2}$ are defined
in Eqs. (4). In the range of velocities $0.1c_{R}<V<0.9c_{R}$, the interval
limit was $\tilde{D}\sim 20\%$ for $\psi\ll 1$, and increased to
$\tilde{D}\gtrsim 100\%$ when $\psi\gg 1$ (Fig. S1). These results indicate
that the 2nd-order approximation applies to a wide range of heterogeneity
fields, materials, and loading conditions.

Figure 4: Finite size effect. (A) The fracture energy prefactor increases with
system length. (B) Collapse of curves in (A) and Fig. S2 by
$\delta\gamma_{2}=(\gamma_{2}(L_{x})-\gamma_{2}(1551))/(\gamma_{2}(1)-\gamma_{2}(1551))$.
$\nu=0.3$, $L_{z}=257.8$, $\Delta z=0.1$.

The renormalized dissipation exhibited finite-size dependence. To investigate,
we numerically computed $\gamma_{2}(L_{x};V,\psi)$ for a fixed $L_{z}$.
$\gamma_{2}(L_{x})$ increased monotonically with $L_{x}<10\ell$ where $\ell$
is the heterogeneity correlation length (Fig. 4A, Fig. S2). At $L_{x}\sim
10\ell$, $\gamma_{2}$ approached an asymptotic size-independent value. Through
a linear transformation, all the functions $\gamma_{2}(L_{x};V,\psi)$
approximately collapsed on a single master curve, showing that the asymptotic
length scale $L_{x}\sim 10\ell$ was independent of $V$ and $\psi$ (Fig. 4B).
The manner of the asymptotic approach, however, changed with the parameters.
For $\psi\geq 1$, $\gamma_{2}$ approached the asymptote smoothly, without
appreciable deviations from it for $L_{x}>20\ell$. For $\psi\ll 1$,
$\gamma_{2}$ approached the asymptote while exhibiting spike-like fluctuations
that persisted until $L_{x}>10^{3}$. The amplitudes of the isolated spikes
increased inversely with the crack velocity. Since $\gamma_{2}$ was evaluated
through Eq. (S5), where ensemble averaging is analytically performed, there is
no stochastic component that could give rise to fluctuations. Instead, the
finite-size spikes are expected to arise from front wave propagation in a
finite geometry.

What is the origin of the renormalization of dissipation by heterogeneity? The
fracture energy $\Gamma$ contains asymmetric 2nd-order contributions that
arise from the coupling of front dynamics to the toughness landscape. The
first contribution to the renormalization coefficient $\gamma_{2}$ couples the
front position to gradients of the heterogeneity field. This effect is the
dynamic version of the weak pinning observed in quasi-static cracks [46]. In
the limit $L_{x},L_{z}\rightarrow\infty$, this term may be written as $\langle
f_{1}\partial_{x}\eta\rangle=(L_{x}L_{z})^{-1}\int\\!\mathrm{d}z\,(f_{1}\eta|_{x=L_{x}}-f_{1}\eta|_{x=0})-(L_{x}L_{z})^{-1}\int\\!\mathrm{d}t\mathrm{d}z\,\eta\partial_{t}f_{1}$.
The boundary terms vanishes as $L_{x}^{-1}$ when $L_{x}\rightarrow\infty$. The
second integral is positive since locally increased toughness $\eta>0$ retard
the front $\partial_{t}f<0$. The second contribution
$\psi\langle\eta\partial_{t}f_{1}\rangle$ is negative by the same argument.
The third contribution $-\frac{1}{2}\psi
V\langle(\partial_{z}f_{1})^{2}\rangle$, that results from front propagation
along the local normal, is negative definite. Thus, velocity dependence of the
fracture energy acts to reduce toughness. The competition between terms of
opposite signs results in a dual effect which is confirmed by an analytical
calculation (Eq. (S5)). All terms vanish as $V\rightarrow c_{R}$ because
higher crack inertia lessens the retardance of the front by asperities. This
is a possible explanation for the decrease of fracture energy with velocity in
glass [34].

Figure 5: Renormalized dissipation of patterned planes. Crack fronts overlay
(black lines) (Eq. (4)) on the toughness patterns (colors): (A) checkered
diamonds, (B) checkered squares, (C) left-pointing and (D) right-pointing
triangles. Colors denote $\eta=1.2$ (light red) $\eta=-0.92$ (light blue).
Side of a square in (B), 20. Time interval between crack fronts, 1.742.
$\psi=0.5$. $D=0.2$ (E) Differences between $\gamma_{2}$ values for the
patterns (A-D) and the random exponentially-correlated field. System size,
$L_{z}=257.8$, $L_{x}=1552$. All panels, $V=0.5c_{R}$, $\nu=0.3$, , $\Delta
z=0.1$.

Control of dynamic fracture by patterning. Our semi-analytical framework can
be used to probe the influence of toughness design on crack propagation. To
shed light, we solved Eq. (1) for four periodic patterns with
$\langle\eta\rangle=0$ and $\langle\eta^{2}\rangle=1$ (Fig. 5A-D,
Supplementary Videos 2-5). Two of them, checkered diamonds and squares, were
symmetric to the crack propagation direction. The two other patterns were
mirror images of an asymmetric triangular pattern. In the diamond and
triangular cases, the crack fronts exhibited large spatial and temporal
gradients, whereas deformations were relatively small in the case of the
squares pattern. To find how $\gamma_{2}$ depend on design, we quantified
their deviations from the $\gamma_{2}$ values for random heterogeneity (Fig.
5E). The diamond pattern produced similar $\gamma_{2}$ values except for large
$\psi$. The checkerboard pattern, however, exhibited much reduced
$\gamma_{2}$. Surprisingly, front propagation through the two asymmetric
triangular patterns resulted in approximately identical $\gamma_{2}$. These
results indicate that, unlike quasi-static cracks, dynamic fracture may be
insensitive to pattern asymmetry. In contrast, since the diamond and square
patterns are related by a $45^{\circ}$ rotation, symmetric patterning may be
used to create “hard” and “easy” directions for crack propagation.

Discussion. Our study leaves the question of transients open. To encompass
such events, $G$ may be transformed into the time domain [33]. In
heterogeneous media with $\psi=0$, front fluctuations grow linearly in time
[15]. An inspection of $C_{ff}$ reveals that, for $0<\psi\ll 1$, fronts should
approach steady state at $t\sim|P_{1}^{\prime}(s_{FW})|(\pi
s_{FW}\psi)^{-1}L_{z}$, the lifetime of the longest front wave. However, local
front arrests, $G(z,t)<\Gamma(z,t)$, may modify the steady-state predictions.

The dominance of front waves when $\psi\ll 1$ raises questions about nonlinear
wave-wave interactions. Since $P_{2}(s_{FW})\neq 0$ [33] waves have a non-zero
contribution to $G$ at the 2nd-order. Resonant wave interactions may then
contribute to $G$ in the 1st-order in $D$ and populate small wavelengths in
the SSF (Fig. 2A). How would the interactions translate into dissipation is an
open question.

How do the predictions change for finite-size bodies [47, 48], besides the
size effect described in Fig. 4? In strip-like geometries, where
$L_{x},L_{z}\ll L_{y}$ the supplied energy per unit crack length is a constant
[49]. The integral dissipation $\int\\!\mathrm{d}l_{c}\,\Gamma(z,t;V_{\perp})$
over the front contour $l_{c}(t)$ is therefore fixed by the loading [16].
Crack front fluctuations will increase $l_{c}$, resulting in decreased
velocities. Loading at a finite distance $b$ from the crack tip adds
$\mathcal{O}(||f||/b)$ terms in $G$, which give rise to front wave dispersion
[38, 33].

Cracks in heterogeneous materials often produce rough surfaces whose excess
area increases the local dissipation. By projecting the dissipation to a
plane, $\Gamma(x,z)$, Eq. (1) can be used to predict the in-plane front
dynamics. Such a methodology may open a way to predict the propagation of
rough fractures at realistic scales.

###### Acknowledgements.

The authors acknowledge enlightening discussions with the participants of the
CECAM flagship workshop 3D cracks and crack stability (June 2023, Lausanne).
I.K. is grateful for fruitful conversations with Sharad Ramanathan and Shmuel
Rubinstein.

## References

  * Bedford  _et al._ [2022] J. D. Bedford, D. R. Faulkner, and N. Lapusta, Fault rock heterogeneity can produce fault weakness and reduce fault stability, Nature Communications 13, 326 (2022).
  * Ray and Viesca [2017] S. Ray and R. C. Viesca, Earthquake Nucleation on Faults With Heterogeneous Frictional Properties, Normal Stress, Journal of Geophysical Research: Solid Earth 122, 8214 (2017).
  * Taufiqurrahman  _et al._ [2023] T. Taufiqurrahman, A.-A. Gabriel, D. Li, T. Ulrich, B. Li, S. Carena, A. Verdecchia, and F. Gallovič, Dynamics, interactions and delays of the 2019 Ridgecrest rupture sequence, Nature 618, 308 (2023).
  * Gounon  _et al._ [2022] A. Gounon, S. Latour, J. Letort, and S. El Arem, Rupture Nucleation on a Periodically Heterogeneous Interface, Geophysical Research Letters 49, e2021GL096816 (2022).
  * Faure and Bayart [2023] Y. Faure and E. Bayart, Experimental evidence of seismic ruptures initiated by aseismic slip (2023), arXiv:2312.17511 [cond-mat, physics:physics] .
  * Garagash [2021] D. I. Garagash, Fracture mechanics of rate-and-state faults and fluid injection induced slip, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 379, 20200129 (2021).
  * Barthelat [2023] F. Barthelat, The Fracture Mechanics of Biological Materials, in  _Mechanics and Physics of Fracture: Multiscale Modeling of the Failure Behavior of Solids_, edited by L. Ponson (Springer International Publishing, Cham, 2023) pp. 255–282.
  * Amini  _et al._ [2021] A. Amini, A. Khavari, F. Barthelat, and A. J. Ehrlicher, Centrifugation and index matching yield a strong and transparent bioinspired nacreous composite, Science 373, 1229 (2021).
  * Shaikeea  _et al._ [2022] A. J. D. Shaikeea, H. Cui, M. O’Masta, X. R. Zheng, and V. S. Deshpande, The toughness of mechanical metamaterials, Nature Materials 21, 297 (2022).
  * [10] S. Gupta, H. S. Esmaeeli, and R. Moini, Tough and Ductile Architected Nacre-Like Cementitious Composites, Advanced Functional Materials n/a, 2313516.
  * Hashin [1962] Z. Hashin, The Elastic Moduli of Heterogeneous Materials, Journal of Applied Mechanics 29, 143 (1962).
  * Willis and Movchan [1995] J. R. Willis and A. B. Movchan, Dynamic weight functions for a moving crack. I. Mode I loading, Journal of the Mechanics and Physics of Solids 43, 319 (1995).
  * Ramanathan and Fisher [1997] S. Ramanathan and D. S. Fisher, Dynamics and instabilities of planar tensile cracks in heterogeneous media, Physical Review Letters 79, 877 (1997).
  * Movchan  _et al._ [1998] A. B. Movchan, H. Gao, and J. R. Willis, On perturbations of plane cracks, International Journal of Solids and Structures 35, 3419 (1998).
  * Morrissey and Rice [2000] J. W. Morrissey and J. R. Rice, Perturbative simulations of crack front waves, Journal of the Mechanics and Physics of Solids 48, 1229 (2000).
  * Wang  _et al._ [2022] M. Wang, M. Adda-Bedia, J. M. Kolinski, and J. Fineberg, How hidden 3D structure within crack fronts reveals energy balance, Journal of the Mechanics and Physics of Solids 161, 104795 (2022).
  * Wei  _et al._ [2024] X. Wei, C. Li, C. McCarthy, and J. M. Kolinski, Complexity of crack front geometry enhances toughness of brittle solids, Nature Physics , 1 (2024).
  * Sanner  _et al._ [2024] A. Sanner, N. Kumar, A. Dhinojwala, T. D. B. Jacobs, and L. Pastewka, Why soft contacts are stickier when breaking than when making them, Science Advances 10, eadl1277 (2024).
  * Démery  _et al._ [2014] V. Démery, A. Rosso, and L. Ponson, From microstructural features to effective toughness in disordered brittle solids, EPL (Europhysics Letters) 105, 34003 (2014).
  * Chopin  _et al._ [2018] J. Chopin, A. Bhaskar, A. Jog, and L. Ponson, Depinning Dynamics of Crack Fronts, Physical Review Letters 121, 235501 (2018).
  * Albertini  _et al._ [2021] G. Albertini, M. Lebihain, F. Hild, L. Ponson, and D. S. Kammer, Effective Toughness of Heterogeneous Materials with Rate-Dependent Fracture Energy, Physical Review Letters 127, 035501 (2021).
  * Pham and Ravi-Chandar [2016] K. H. Pham and K. Ravi-Chandar, On the growth of cracks under mixed-mode I+ III loading, International Journal of Fracture 199, 105 (2016).
  * Steinhardt and Rubinstein [2022] W. Steinhardt and S. M. Rubinstein, How Material Heterogeneity Creates Rough Fractures, Physical Review Letters 129, 128001 (2022).
  * Lebihain  _et al._ [2020] M. Lebihain, J.-B. Leblond, and L. Ponson, Effective toughness of periodic heterogeneous materials: The effect of out-of-plane excursions of cracks, Journal of the Mechanics and Physics of Solids 137, 103876 (2020).
  * Lubomirsky and Bouchbinder [2023] Y. Lubomirsky and E. Bouchbinder, Quenched disorder and instability control dynamic fracture in three dimensions (2023), arXiv:2311.11692 [cond-mat, physics:nlin] .
  * Sharon  _et al._ [1996] E. Sharon, S. P. Gross, and J. Fineberg, Energy Dissipation in Dynamic Fracture, Physical Review Letters 76, 2117 (1996).
  * Roch  _et al._ [2023] T. Roch, M. Lebihain, and J.-F. Molinari, Dynamic Crack-Front Deformations in Cohesive Materials, Physical Review Letters 131, 096101 (2023).
  * Pons and Karma [2010] A. J. Pons and A. Karma, Helical crack-front instability in mixed-mode fracture, Nature 464, 85 (2010).
  * Henry and Adda-Bedia [2013] H. Henry and M. Adda-Bedia, Fractographic aspects of crack branching instability using a phase-field model, Physical Review E 88, 060401 (2013).
  * Chen  _et al._ [2015] C.-H. Chen, T. Cambonie, V. Lazarus, M. Nicoli, A. J. Pons, and A. Karma, Crack front segmentation and facet coarsening in mixed-mode fracture, Physical review letters 115, 265503 (2015).
  * Möller and Bitzek [2015] J. J. Möller and E. Bitzek, On the influence of crack front curvature on the fracture behavior of nanoscale cracks, Engineering Fracture Mechanics 150, 197 (2015).
  * Buehler [2022] M. J. Buehler, Modeling Atomistic Dynamic Fracture Mechanisms Using a Progressive Transformer Diffusion Model, Journal of Applied Mechanics 89, 10.1115/1.4055730 (2022).
  * Kolvin and Adda-Bedia [2024] I. Kolvin and M. Adda-Bedia, A comprehensive study of nonlinear perturbations in the dynamics of planar crack fronts (2024), arXiv:2404.18633 [cond-mat, physics:physics] .
  * Sharon and Fineberg [1999] E. Sharon and J. Fineberg, Confirming the continuum theory of dynamic brittle fracture for fast cracks, Nature 397, 333 (1999).
  * Hauch  _et al._ [1999] J. A. Hauch, D. Holland, M. P. Marder, and H. L. Swinney, Dynamic fracture in single crystal silicon, Physical Review Letters 82, 3823 (1999).
  * Baumberger  _et al._ [2006] T. Baumberger, C. Caroli, and D. Martina, Solvent control of crack dynamics in a reversible hydrogel., Nature Materials 5 (2006).
  * Freund [1990] L. B. Freund,  _Dynamic Fracture Mechanics_ (Cambridge University Press, Cambridge; New York, 1990).
  * Norris and Abrahams [2007] A. N. Norris and I. D. Abrahams, A multiple-scales approach to crack-front waves, Journal of Engineering Mathematics 59, 399 (2007).
  * Adda-Bedia  _et al._ [2013] M. Adda-Bedia, R. E. Arias, E. Bouchbinder, and E. Katzav, Dynamic Stability of Crack Fronts: Out-Of-Plane Corrugations, Physical Review Letters 110, 014302 (2013).
  * Sharon  _et al._ [2001] E. Sharon, G. Cohen, and J. Fineberg, Propagating solitary waves along a rapidly moving crack front, Nature 410, 68 (2001).
  * Fekak  _et al._ [2020] F. Fekak, F. Barras, A. Dubois, D. Spielmann, D. Bonamy, P. H. Geubelle, and J. F. Molinari, Crack front waves: A 3D dynamic response to a local perturbation of tensile and shear cracks, Journal of the Mechanics and Physics of Solids 135, 103806 (2020).
  * Das  _et al._ [2023] S. Das, Y. Lubomirsky, and E. Bouchbinder, Dynamics of crack front waves in three-dimensional material failure, Physical Review E 108, L043002 (2023).
  * [43] I. Kolvin and M. Adda-Bedia, Supplemental Material for ”Dual role for heterogeneity in dynamic fracture”.
  * Bouchaud  _et al._ [2002] E. Bouchaud, JP. Bouchaud, DS. Fisher, S. Ramanathan, and JR. Rice, Can crack front waves explain the roughness of cracks?, Journal of the Mechanics and Physics of Solids 50, 1703 (2002).
  * Schmittbuhl and Måløy [1997] J. Schmittbuhl and K. J. Måløy, Direct Observation of a Self-Affine Crack Propagation, Physical Review Letters 78, 3888 (1997).
  * Dalmas  _et al._ [2009] D. Dalmas, E. Barthel, and D. Vandembroucq, Crack front pinning by design in planar heterogeneous interfaces, Journal of the Mechanics and Physics of Solids 57, 446 (2009).
  * Xue  _et al._ [2024] N. Xue, R. Long, E. R. Dufresne, and R. W. Style, Elastomers Fail from the Edge, Physical Review X 14, 011054 (2024).
  * Bazant and Pfeiffer [1987] Z. P. Bazant and P. A. Pfeiffer, Determination of Fracture Energy from Size Effect and Brittleness Number, Materials Journal 84, 463 (1987).
  * Goldman  _et al._ [2010] T. Goldman, A. Livne, and J. Fineberg, Acquisition of Inertia by a Moving Crack, Physical Review Letters 104, 114301 (2010).


utf8

# A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech
Recognition: the Arman-AV Dataset

Javad Peymanfard 1, Samin Heydarian 1§, Ali Lashini 1§, Hossein Zeinali 2,
Mohammad Reza Mohammadi 1, and Nasser Mozayani 1  1 School of Computer
Engineering
Iran University of Science and Technology, Tehran, Iran
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>2 Department of Computer Engineering
Amirkabir University of Technology, Tehran, Iran
Email<EMAIL_ADDRESS>
###### Abstract

In recent years, significant progress has been made in automatic lip reading.
But these methods require large-scale datasets that do not exist for many low-
resource languages. In this paper, we have presented a new multipurpose audio-
visual dataset for Persian. This dataset consists of almost 220 hours of
videos with 1760 corresponding speakers. In addition to lip reading, the
dataset is suitable for automatic speech recognition, audio-visual speech
recognition, and speaker recognition. Also, it is the first large-scale lip
reading dataset in Persian. A baseline method was provided for each mentioned
task. In addition, we have proposed a technique to detect visemes (a visual
equivalent of a phoneme) in Persian. The visemes obtained by this method
increase the accuracy of the lip reading task by 7% relatively compared to the
previously proposed visemes, which can be applied to other languages as well.

###### Index Terms:

persian dataset, audio-visual speech recognition, lip reading, viseme

§§footnotetext: These authors contributed equally.

##  I Introduction

Automatic speech recognition (ASR) is a task to understand speech from audio
signals. This task has been developed over the years and got mature enough to
be used on any device. But the trained models, apart from the high ability of
speech recognition, have weaknesses in special conditions like environments
with loud noises. That’s where audio-visual speech recognition (AVSR) comes in
to overcome this limitation. AVSR uses visual information alongside audio in
order to decode speech more effectively in noisy environments like inside
cars. AVSR is more like human comprehension, which uses visual and audio
perception to understand each other. Lip reading is another task which is only
consuming visual information.

The traditional approaches [1, 2, 3] used a two-stage algorithm to deal with
such problems. In the first stage, a hand-crafted feature extractor, extract
useful features from lip movements alongside another feature extractor that
extracts feature from audio signals and then fuse them. In the second stage, a
classifier such as the hidden Markov model or artificial neural networks is
used to classify digits, characters, etc. However, in the last few years, the
availability of large public datasets and emerge of deep neural networks had a
substantial impact on this field. Deep learning approaches usually consist of
two parts, which are front-end and back-end, like traditional ones except that
here they are end-to-end trainable. For front-end part usually use convolution
neural networks to extract visual and audio features and temporal networks
like RNNs, attention model, and transformers on the back-end side to model
temporal information. Recently, methods such as [4, 5, 6] used knowledge
distillation to train lip reading and AVSR models. To do this, usually use the
ASR model as the teacher and the lip reading model as the student.

The primary dataset was collected under laboratory conditions [7, 8] \-
Usually, a person stands in front of the camera and read some words or
sentences in a quiet place at normal speed. Emerge of deep learning and
automatic pipeline led to building datasets in the "wild" condition [9, 10,
11] which is larger and also more challenging and close to the real condition
than before. Unlike the old datasets which were restricted to digit,
character, or phrase classification, nowadays, datasets are collected for
word-level classification and sentence-level AVSR. Unfortunately, most of
these datasets are in English and there are a few datasets witches come in
other languages [12]. In this paper, we will more concentrate on sentence-
level AVSR, but we did not limit our dataset to just speech recognition tasks.
We make our dataset in a way that can be used for different tasks. In the
following sections, we will explain more about it.

Currently, LRS2 [13] is the most wildly used audio-visual dataset. The dataset
contains more than 240 hours of videos and about 118,000 utterances. Also,
they used BBC news and talk shows as sources for their dataset. The dataset is
publicly available to the research community. LSVSR [11] is the largest
dataset with over 3800 hours of data collected from YouTube-uploaded videos.
This dataset was collected by Google DeepMind and, unfortunately, they did not
make it publicly available.

In this paper we propose a novel large-scale dataset that is collected in the
"wild" condition from reviews, movies, etc. on the Aparat website. The dataset
contains over 220 hours of videos from 1760 Persian celebrities. Also, we
provide the name of the celebrity in each sample as labels to be used for
different purposes, such as speaker recognition. To the best of our knowledge,
this is the largest audio-visual dataset in the Persian language.

Since this is an audio-visual dataset, it could be used for a number of
different applications such as automatic speech recognition, lip reading,
speaker recognition, audio-visual speech synthesis, etc.

The organization of this paper is as follows. In section 2, we look at some of
the most recent approaches and datasets. In section 3, we discuss the
statistic of the data and the pipeline that we build for collecting data. In
section 4, we will use a base model to evaluate our dataset. Finally, conclude
the paper in section 5.

##  II Related Works

[b]  TABLE I: Statistical comparison between well-known datasets. Category
Dataset General info Transcription info Video info Speaker info Category Name
Year Lang. Task Classes Utter. Availability Source Vocab. Words Num. Duration
Resolution FPS View(°) Num. SFAVD [14] 2013 FA Sent. - 600 - Lab environment
1000 - - - 131×105 30 Frontal 1 LRW [9] 2016 ENG Words 500 400K Avail.*1 BBC
Programmes 500 400K - - 256×256 25 -30 30 1000+ LRS2 [13] 2017 ENG Sent. -
118K Avail.* BBC Programmes 174K 807K - 246h 160×160 25 -30 30 1000+ MV-LRS
[15] 2017 ENG Sent. - 74K Avail.* Programs 14K - - 165h 160×160 25 -90 90
1000+ Audio-Visual VLRF [16] 2017 SPA Sent. - 600 Public Lab environment 1374
10K - 180min 1280×720 50 Frontal 24 Speech LRS3-TED [10] 2018 ENG Sent. - 165K
Avail.* TED Talks 57K - - 475h 224×224 25 -90 90 1000+ Recognition LSVSR [11]
2018 ENG Sent. - 2,934K Private YouTube 127K - - 3,886h 128×128 30 -30 30
1000+ LRW-1000 [17] 2019 CHI Words 1000 718K Avail.* Broadcast news 1K - - -
Distributed 25 -90 90 2000+ CMLR [12] 2019 CHI Words - 102K Public News
Broadcast - - - - - - - 11 LRWR [18] 2021 RUS Words 235 - - YouTube 235 117K -
- 112×112 25 0 20 135 GLips [19] 2022 GER Words 500 - Public Hessian
Parliament 500 250K 250K - 256×256 25 - 100 RUSAVIC [20] 2022 RUS - 62 62
Avail.* Vehicle environment - - 200 - 1920×1080 60 -30 30 20 PLRW [21] 2022 FA
Words 500 244K Public Aparat 500 - 244K 30h 224×224 25 - 1800 Our Dataset 2022
FA Sent. - 89K Public Aparat 42K 2.5M 89K 220h 224×224 25 - 1760 Acitive
Speaker AVA [22] 2019 - - 3 - Public YouTube - - 188 38.5h 128×128 - - -
Detection ASW [23] 2021 - - 2 - - VoxConverse dataset - - 212 30.9h - - - -
Voxceleb [24] 2017 ENG - 1251 153K Public YouTube - - 22K 352h - - - 1,251
Speaker Voxceleb2 [25] 2018 ENG - 6112 1,128K Public YouTube - - 150K 2442h -
- - 6,112 Recognition DeepMine [26] 2019 FA&ENG - 1850 544K Avail.*
Crowdsourcing - - 540 480h - - - 1,850 CN-Celeb [27] 2019 CHI - 1000 130K
Public bilibili.com - - - 273.73h - - - 1,000 1 Available by contact.

In this section, we will review the two tasks of speech recognition and
speaker recognition. Since we have introduced a new dataset and tested the
performance of the baseline models on it, we will explore the related works
from the perspective of the proposed methods and datasets.

###  II-A Methods

####  II-A1 Speech Recognition

Lip reading. There is a large body of research on automated lip reading. The
advent of deep learning and the availability of large-scale datasets cause
massive progress in the automation of this field. Here we discuss the works
which focused on the sentence-level task in lip reading as an open set
character. For sentence-level recognition, the recent works can be divided
into approaches. The first approach utilizes Connectionist Temporal
Classification (CTC). The model predicts frame-wise characters and tries to
maximize and marginalizes all possible paths to find optimal alignment between
prediction and ground truth. An example based on this approach is LipNet [28].
LipNet was the first sentence-level lip reading model that used spatiotemporal
CNNs as the front-end, followed by Bidirectional Gated Recurrent Unit (Bi-GRU)
as the back-end of architecture, and employed CTC loss to train the network.
Shillingford et al. [11] proposed the V2P model closest to previous work. The
model outputs a sequence of phoneme distributions and uses CTC loss at train
time. At inference time, a decoder based on finite-state transducers (FSTs)
maps a sequence of phoneme distributions to a word sequence.

The second approach follows the sequence-to-sequence model, in which output
characters are conditioned based on each other, unlike CTC-based models. WAS
[9] was the first sequence-to-sequence model. The model consists of two key
components called the image encoder Watch and the character decoder Spell and
has a unique dual attention mechanism. Chung et al. [15] extended the WAS
model to the MV-WAS model that can decode visual sequences across all poses
and show that it is possible to read lips in profile, but the standard is
inferior to reading frontal faces. Peymanfard et al. [29] suggested external
viseme decoding, which divides the sequence-to-sequence model into two stages,
video to viseme and viseme to character, respectively. The network outputs
character sequence given viseme through external text data. Afouras et al.
[30] proposed a visual transformer pooling (VTP) module that learns to track
and aggregate the lip movements representation. This work also utilizes a
wordPiece tokenizer (one of the subword algorithms) to learn the language
easily, resulting in time and memory efficiency

Petridis et al. [31] presented a hybrid architecture and used a joint decoder
including RNN, attention, and CTC. Afouras et al. [32] took a hybrid approach
as well but suggested replacing RNN with a transformer. This work proposed a
sequence-to-sequence and CTC on top of the transformer self-attention as the
back-end. Both models use spatiotemporal CNNs, followed by ResNet as a common
front-end. The extension of [31] is [33] that employed the conformer variant
of the transformer to model both local and global dependencies of an image or
audio sequence.

There is a trend in lip reading that benefits from the knowledge distillation
area to improve performance by extracting information from the audio-only
counterpart of datasets. In this approach, the Automatic Speech Recognition
(ASR) model plays a role as the teacher for the Visual Speech Recognition
(VSR) model as the student. The examples of this trend are [4], [5] and [6].
Automatic Speech Recognition. In ASR, self-supervision plays the main role in
the state-of-the-art models. This approach has two steps, pre-training and
fine-tuning. In the pre-training step, the model learns powerful speech
representation from large amounts of unlabeled data through a pretext task.
Then, the output of the previous step (speech representation) is fed to an
acoustic model to be fine-tuned for a downstream task. In wav2vec [34], the
pre-training step has two networks named encoder and context. The encoder
network converts raw audio input to feature representations, and the context
network converts feature representations to contextualized features. Both
networks are convolutional networks. The objective of this model is defined by
contrastive loss between future steps and distractors. The vq-wav2vec [35] is
the improved version of the previous work, which benefited from the BERT
model. This work is also composed of two stages. In the first stage, the
continuous audio feature representations are discretized via Gumbel-Softmax or
k-means clustering quantization methods. Thanks to this, the speech data takes
a structure similar to language, and the BERT model can be applied. So in the
second stage, the discretized representations are fed to a BERT model for
contextualized representations. The wav2vec 2.0 [36] showed better results by
joint learning of the two-stages pre-training pipeline mentioned above. The
model is pre-trained by new contrastive loss between contextualized output and
quantized representation. The HuBERT [37] is very similar to wav2vec 2.0 in
model architecture but different in the training process. First, HuBERT uses
the cross-entropy loss for model optimization. Second, Data discretization is
done through a k-means algorithm instead of a quantization process. Third, The
training process alternates between the hidden units discovery and target
prediction. The model re-uses embeddings from the BERT encoder in the
clustering step.
Audio-Visual Speech Recognition. The task of audio-visual speech recognition
is lip reading with the presence of audio. Shi et al. [38] presented the
audio-visual counterpart of HuBERT model as an AV-HuBERT. Also, this work
reported the result of visual HuBERT performance for the lip reading task. The
works like [11], [31] and [32] addressed both the VSR and the AVSR problems,
which are later explained in the lip reading section.

####  II-A2 Speaker Recognition

Since this is a classification problem, the general models consist of two
modules. The first is a feature extractor module and the second is a
classification module. Depending on the input data, the feature extractor
could be a convolutional network like VGG-M or ResNet [25] or any other neural
network that could be used for sequential data. For classification, a fully
connected network is the leading choice, and the output size is the number of
speakers in the dataset. Since the input data is an audio file, input data to
the network could be a spectrogram [25] of data or the output of a feature
extraction algorithm like MFCC, etc., which usually is a 1D feature vector of
data.

###  II-B Datasets

Figure 1: Data collection pipeline. Figure 2: Our dataset samples.

####  II-B1 Speech Recognition

As we know, the progress in deep learning is due to large datasets. The
performance of lip reading systems is affected by position, illumination and
speaker diversity, so the quality and quantity of the dataset are important.
In recent years some datasets like LRS2 [13], LRS3-TED [10] and LSVSR [11]
have tried to increase the number of utterances and video hours and MV-LRS
[15] has tried to cover more face angles. Table I shows three categories of
datasets named Audio-Visual Speech Recognition, Active Speaker Detection and
Speaker recognition. The table has general, transcription, video and speaker
information columns for each dataset. We can see useful statistics in this
information, such as the type of task run on the dataset, the source of the
collected videos and the number of speakers in the videos. The language of
most of the datasets in this table is English, but fortunately, other
languages have been published in recent years, like CMLR [12], LRWR [18] and
GLips [19]. Since 2013, no Persian dataset has been published for lip reading
until now; we released Audio-Visual Speech Recognition in Persian (Arman_AV)
with 89k utterances, 220 hours and 1760 speakers.

####  II-B2 Speaker Recognition

Most of the datasets in this field are collected under laboratory conditions
and are not available to be used freely for research purposes. One of the
first free datasets which were collected under the "wild" conditions is the
Speakers in the Wild (SITW) [39] dataset. This dataset contains multimedia
content of 299 speakers with hand-annotated speech samples. VoxCeleb [24] is a
large-scale dataset that contains speech samples of over 1,000 celebrities
with more than 100,000 utterances, extracted from YouTube videos; this dataset
is gender-balanced (45% of the speakers are female), along with speakers with
various accents, racial backgrounds, and ages. The dataset’s creators included
information about each speaker’s gender and country of origin from Wikipedia.
Videos contained in the dataset are recorded in a huge number of challenging
environments such as quiet and noisy studio interviews, open-air stadiums,
etc., and all are debased with real-world commotion, consisting of giggling,
covering discourse, foundation chatter, etc. Also, another goal of the authors
of the dataset was to propose a pipeline to create fully automated datasets.
VoxCeleb 2 [25] which came shortly after VoxCeleb (and is very similar but
much larger) is the largest dataset in this field which includes more than six
thousand speakers that speak more than a million utterances. The three
mentioned datasets are in English. CN-Celeb [27] is another large-scale
dataset which contains over 130,000 utterances and is very similar to the
VoxCeleb dataset except in the three following aspects: first, CN-Celeb covers
11 genres of speech such as singing, entertainment, etc., which is more than
VoxCeleb. Second, CN-Celeb concentrated on Chinese celebrities, containing
videos of 1,000 celebrities. And last but not least is that the dataset is not
automated completely, and they considered human supervision for the dataset.
To the best of our knowledge, there is no suitable Persian dataset in this
field.

##  III Dataset

###  III-A Automated Pipeline

In this subsection, we see the steps of data collection. Figure 1 shows the
overview of these steps.
Step 1: Video Collection. We pick three kinds of videos for our dataset
described in the following segments.

  1. 1.

Interviews. Interviews and biographies are well-suited for audio-visual
datasets. The main goal of these videos is to speak to artists, politicians or
famous people in general. These kinds of videos are divided into two groups.
In the first group, the shows include a host and narrator (which is not
desirable in this case), and the second group consists of those that don’t
have them.

  2. 2.

Series and Movies. These kinds of videos can be used if suitable preprocessing
is applied to them. The main challenges are dealing with different directions
and angles of the camera (When a speaker is talking or is not in the shot all
of the time, unlike in the interviews). Also, the entire video has a
significant amount of silence or music. Another challenge is multi-speaker
simultaneous speech. In general, this type of video is not ideal for
collecting data because the percentage of redundant data in them is relatively
high. In the investigation we did, less than 10% of the input data is
considered appropriate.

  3. 3.

Vods. One of the other ideas to collect more data is different keyword
searching in ugcs. Then we review the resulting videos and choose the top
keywords for the dataset.

To obtain data, multiple search terms are used

Step 2: Scene Detection. In this stage, we want to detect each scene. For this
purpose, the PySceneDetect111https://github.com/Breakthrough/PySceneDetect is
utilized. In this algorithm, we subtract the pixel values of two consecutive
frames to detect different scenes. This difference value is calculated in HSV
colour space.

Step 3: Face Detection and Face Tracking. Now each video is divided into
smaller pieces called a scene. Up to this point, the video had a temporal
clipping. Note that we need frames in which we see faces. So the S3FD model
[40] and algorithm based on IoU [41] are used for face detection and face
tracking respectively. Depending on the number of faces found in each part of
the video, each video may be split into one or more videos or removed if there
is no face.

Step 4: Active Speaker Detection.

In this step, we automatically found the parts of the video that contain the
speaker’s face.

Step 5: Speaker Diarization. As stated, up to stage 4, the video parts
including a speaker with a constant shooting angle have been chosen. But there
is another potential challenge in interviews, a situation where the host and
the guest talk at the same time. To address this problem, we can use Speaker
Diarization222https://github.com/taylorlu/Speaker-Diarization based on UIS-
RNN.

Step 6: Annotations. As we mentioned before, the source of our dataset videos
is Aparat. The videos on this video hosting website lack subtitles, so we
utilize the commercial Aipaa’s333https://aipaa.ir/ ASR service to construct
rough sentence-level transcripts of the videos.

Step 7: Face Recognition and Dataset Split. We use ArcFace [42] and produce
face feature embeddings for face recognition. This stage aims to create a
dataset with a speaker-independent property. A dataset like this is beneficial
for applications like lip reading. It should be noted that the results of the
face recognition algorithm were checked manually, and its errors were
corrected.

###  III-B statistics

First of all, the dataset will be available as mp4 face-cropped videos with
224*224 pixels resolution and a frame rate of 25 FPS. We leveraged Aparat (a
Persian video-sharing website like YouTube) as a source for our dataset. The
collected videos came from different channels and programs, which means our
dataset contains various kinds of challenging environments and circumstances.
The outcome of our work comprises 220 hours of video, which contains over 89
thousand utterances and 2.5 million words of 1760 celebrities. We split our
dataset into train/ validation, and test sections which each contain 211, 9
hours of data. Figure 2 shows some examples of the video samples. In table I,
in addition to the statistics of our dataset, we have also presented the
statistics of other recent datasets.

##  IV Viseme Analysis

The viseme is a significant challenge in the lip reading problem. Visemes
consist of a set of phonemes spoken by the same form of lips and aren’t
exactly known for any languages. This mapping may even be different in various
speakers. However, some research has addressed the issue in languages such as
English and Persian. Some proposed models for said languages. In this study,
we proposed a method to automatically identify the Persian visemes on large-
scale data that one can apply to other languages by only having the
appropriate dataset of the interested language. Some works have addressed the
visemes for the Persian language [43]. Collecting a set of laboratory datasets
and considering the similarities in speaking the various phonemes, the authors
in [43] proposed a categorization method for phonemes to identify the visemes
in the language. Combining the deep learning technique to automatically
extract the features and clustering in this study, we introduced a method to
categorize the phonemes. Then, we compared a viseme-based model for lip
reading with these visemes and the visemes addressed in previous works.
Employing the Kaldi tool, we first determined the phoneme level transcription
for each sample in this experiment. Each phoneme spoken by the speaker is
known every 30 milliseconds in this transcription. Then, we employed the av-
Hubert pre-trained model to obtain the visual embedding for each phoneme,
which was used as a feature for our clustering using the k-means algorithm. In
this step, we expected the phonemes having the same visual features, like the
same lip movements and forms, to be grouped in the same category. We used the
method in [29] to train our model. Employing the two models of the video-to-
viseme and the viseme-to-character models and finally merging the two, we
obtained the lip reading model. One benefit of such a method is that one can
use the textual data to improve the quality of the lip reading model. However,
the model requires a proper grapheme to phoneme (G2P) model to convert the
Persian text into the phonemes and then convert it into the visemes, for which
we used an appropriate G2P. For the video-to-viseme model dataset, we used the
phoneme level transcription to convert the phonemes into visemes with the help
of its corresponding mapping. We also employed about 1 million utterances for
the visemes into character conversion datasets. In this regard, we first
employed a G2P model to obtain the phoneme sequence for each utterance and
then used the phoneme-to-viseme mapping to extract the viseme sequence for
each one. To train the viseme-to-character model, we used the viseme sequence
as the input, whereas the persian text was considered the desired output.
Employing the attention mechanism to implement the video-to-viseme model, we
used a seq2seq one having a two-layer GRU. Furthermore, we employed the same
architecture with fewer parameters and a different input type to train our
viseme-to-character model. As seen in table II, we obtained a higher precision
in the lip reading problem by employing the new proposed model to identify the
Persian visemes for which we believe this phoneme-to-viseme mapping to be more
appropriate. Furthermore, one can use the method to identify visemes in any
given language. All the data and their related transcriptions are publicly
available.

TABLE II: Lip reading results on our dataset

Viseme Mapping CER (Greedy decoding) WER (Greedy decoding) CER (Beam search
decoding) WER (Beam search decoding) Traditional [43] %54.04 %76.32 %52.19
%73.24 AV-HuBERT + K-means (Ours) %51.24 %76.71 %48.54 %72.83

(a)  (b)

Figure 3: 2D and 3D projection of AV-HuBERT for persian phonemes using our dataset. a) 3D projection of embedding of phonemes using PCA. b) 2D projection of embedding of phonemes using tSNE. TABLE III: Persian visemes using clustering and AV-HuBERT cluster id | phonemes
---|---
1 | /F/ /V/
2 | /AA/ /KH/
3 | /B/ /P/ /M/
4 | /ZH/
5 | /CH/ /JH/ /SH/
6 | /O/ /U/
7 | /S/ /Z/ /D/ /T/
8 | /A/ /’/ /H/ /GH/
9 | /G/ /K/ /E/ /L/ /I/ /Y/ /N/ /R/
10 | /sil/

##  V Experiments

Here, the baseline methods are explained. We demonstrate how well they work
with our dataset and provide information on training setup. The image sequence
and audio sequence inputs are denoted as $x_{1:T}^{v}$ and $x_{1:T}^{a}$ in
all baselines, respectively.

###  V-A Automatic Speech Recognition (ASR)

The present section reviews the experiments conducted on speech recognition
using the produced datasets. Audio data alone was used in the first experiment
to train Persian speech recognition. To this end, the pre-trained AV-HuBERT
[44] model uses $x_{1:T}^{a}$ to extract a 768-dimensional feature vector
$e_{1:T}^{a}$ for each 40ms of audio input. In this experiment, we used the
pre-trained AV-HuBERT base model for feature extraction on the English
language data. The output embedding in this network for each window is a
representation of that audio segment, including speaker and speech features.
We expect the model to use speech-related features during the learning
process. As such, we expect features corresponding to the speaker to be
implicitly ignored, as they are irrelevant to our purpose. One further
consideration during the experiment is whether or not a model pre-trained via
the English language data can be used for the Persian language.
In the next step, the $e_{1:T}^{a}$ vector was fed to a two-stacked
Bidirectional LSTM, resulting in $o_{1:T}^{a}$. This network was trained using
CTC. We assume $y=(y_{1},y_{2},...,y_{n})$ is a transcription and $\pi$ are
paths that function B will map $\pi$ to y. We calculated the $p_{t}^{CTC}$
probability over 52 characters by applying Softmax on $o_{1:T}^{a}$. As our
CTC loss $L_{CTC}$ decreases, the network begins to learn.
As evident in table IV, using the proposed architecture, we achieved a %84.66
character accuracy rate (CAR) using only 211 hours of Persian speech data,
which is a suitable value, considering the data volume. Moreover, the
experiment has demonstrated that the AV-HuBERT model, trained on data in
English, could extract a proper embedding for speech data in Persian.

###  V-B Audio Visual Speech Recognition (AVSR)

During the second experiment, we trained an audio-visual speech recognition
model. Image pre-processing was first used in this model to crop only the
mouth ROI. Then the AV-HuBERT model was used per cropped video frame to obtain
a 768-dimensional vector. In this section, an architecture similar to the
previous stage was used, but we also gave the visual features as input to the
model. Moreover, we used less output for visual features as they presumably
contain less information than speech information. This model is presented in
Figure 4. The input image sequence $x_{1:T}^{v}$ is fed to the pre-trained
Single-modal visual HuBERT to obtain a 768-dimensional feature vector
$e_{1:T}^{v}$. The 768-dimensional feature vector $e_{1:T}^{a}$ is obtained
from input audio sequence $x_{1:T}^{a}$ through the pre-trained Singel-modal
audio HuBERT concurrently. In the next step, the separated two-stacked Bi-LSTM
are used to map $e_{1:T}^{v}$ and $e_{1:T}^{a}$ to $o_{1:T}^{v}$ and
$o_{1:T}^{a}$ respectively. Then the two 1-dimensional convolutional filters
are applied on $c_{t}$ as a result of the concatenation of $o_{t}^{v}$ and
$o_{t}^{a}$. The output of this step is $f_{t}$. As mentioned above, CTC loss
is used to train the model. First, we calculated the $p_{t}^{CTC}$
probability. This probability is obtained by applying Softmax and linear
projection on $f_{t}$ (Where $W\in{\rm I\\!R^{52\times(256+1024)}}$ and
$b\in{\rm I\\!R^{52}}$). Then, the model will be optimized through $L_{CTC}$
minimizing the value.

This model was trained with the exact mechanism as the previous one. As
visible in table IV, through this model, which provides both audio and visual
features, we have achieved higher accuracy than the ASR model. As visible in
this experiment, embeddings obtained in the pre-trained AV-HuBERT model, which
are trained on the English language data, can be used in the Farsi language,
and relatively desirable features of lip movement can be extracted from it.

TABLE IV: Speech recognition results on our dataset

Method CER (Greedy decoding) WER (Greedy decoding) CER (Beam search decoding)
WER (Beam search decoding) ASR(CTC) %15.77 %46.43 %15.34 %45.86 AVSR(CTC)
%13.76 %44.98 %13.08 %43.43

###  V-C Speaker recognition

Another problem we tested using the generated data was speaker recognition.
For this, we first separated the people with more than five samples. Then we
randomly selected five samples for each speaker. For each speaker, four
samples were considered as a reference, and only one of the samples was used
for testing. Then, using the TitaNet [45] pre-trained model, which was trained
on 5 datasets (Voxceleb [24] and Voxceleb2 [25], NIST SRE portion of datasets
from 2004-2008 (LDC2009E100), Switchboard-Cellular1 and Switchboard-Cellular2
[46], Fisher [47] and Librispeech [48]). We fed $x_{1:T}^{a_{t}}$ as a test
and $x_{1:T}^{a_{0}}$, …, $x_{1:T}^{a_{3}}$ as references to the TitaNet [45]
model. The model produces the 192-dimensional embeddings of each audio sample,
for example, $v_{1:T}^{a_{t}}$. In the next step, the test embedding vector is
compared with every reference of every person through cosine similarity (Here
$i\in{0,...,3}$). After that, we calculate the average with the previous
step’s results. Then, each sample in the test was assigned to the person who
had the most similarity with that in terms of the average cosine similarity
criterion. Using this method, we achieved an accuracy of 84.5.

Figure 4: The audio-visual architecture.

##  VI Conclusion

Arman-AV is a large-scale multipurpose dataset for Persian that can be used in
various tasks such as lip reading, automatic speech recognition, audio-visual
speech recognition, and speaker recognition, and it is publicly available. The
dataset consists of almost 220 hours of video data (about 89,000 samples) from
1760 speakers. Although the main goal of collecting this dataset was lip
reading, the results of a baseline method were reported for each of the
mentioned tasks. According to the obtained results, the character error rate
of speech recognition was reduced by 14.7% relatively using visual features.
We also proposed a method for obtaining Persian visemes. By using these
visemes, higher accuracy in lip reading was achieved. In addition, we have
provided various analyses for the dataset such as the age and gender of
speakers, and the synchronization of audio and video for each segment. All of
this metadata is available along with the dataset. The dataset can be used for
other tasks such as face recognition, face verification, and audio-visual
speaker recognition, which we did not cover in this paper and can be
considered for future work on the dataset.

##  VII Acknowledgement

The project was mainly supported by Arman Rayan Sharif company, an AI company
in Iran.

## References

  * [1] I. Matthews, T. F. Cootes, J. A. Bangham, S. Cox, and R. Harvey, “Extraction of visual features for lipreading,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 24, no. 2, pp. 198–213, 2002.
  * [2] S. S. Morade and S. Patnaik, “A novel lip reading algorithm by using localized acm and hmm: Tested for digit recognition,” _Optik_ , vol. 125, no. 18, pp. 5181–5186, 2014. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0030402614005786
  * [3] Z. Zhou, G. Zhao, and M. Pietikäinen, “Towards a practical lipreading system,” in _CVPR 2011_. IEEE, 2011, pp. 137–144.
  * [4] Y. Zhao, R. Xu, X. Wang, P. Hou, H. Tang, and M. Song, “Hearing lips: Improving lip reading by distilling speech recognizers,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 6917–6924.
  * [5] T. Afouras, J. S. Chung, and A. Zisserman, “Asr is all you need: Cross-modal distillation for lip reading,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 2143–2147.
  * [6] S. Ren, Y. Du, J. Lv, G. Han, and S. He, “Learning from the master: Distilling cross-modal advanced knowledge for lip reading,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 13 325–13 333.
  * [7] M. Cooke, J. Barker, S. Cunningham, and X. Shao, “An audio-visual corpus for speech perception and automatic speech recognition,” _The Journal of the Acoustical Society of America_ , vol. 120, no. 5, pp. 2421–2424, 2006.
  * [8] I. Anina, Z. Zhou, G. Zhao, and M. Pietikäinen, “Ouluvs2: A multi-view audiovisual database for non-rigid mouth motion analysis,” in _2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , vol. 1. IEEE, 2015, pp. 1–5.
  * [9] J. S. Chung and A. Zisserman, “Lip reading in the wild,” in _Asian conference on computer vision_. Springer, 2017, pp. 87–103.
  * [10] T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” _arXiv preprint arXiv:1809.00496_ , 2018\.
  * [11] B. Shillingford, Y. Assael, M. W. Hoffman, T. Paine, C. Hughes, U. Prabhu, H. Liao, H. Sak, K. Rao, L. Bennett, M. Mulville, M. Denil, B. Coppin, B. Laurie, A. Senior, and N. de Freitas, “Large-Scale Visual Speech Recognition,” in _Proc. Interspeech 2019_ , 2019, pp. 4135–4139.
  * [12] Y. Zhao, R. Xu, and M. Song, “A cascade sequence-to-sequence model for chinese mandarin lip reading,” in _Proceedings of the ACM Multimedia Asia_ , 2019, pp. 1–6.
  * [13] J. Son Chung, A. Senior, O. Vinyals, and A. Zisserman, “Lip reading sentences in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6447–6456.
  * [14] Z. Naraghi and M. Jamzad, “Sfavd: Sharif farsi audio visual database,” in _The 5th Conference on Information and Knowledge Technology_. IEEE, 2013, pp. 417–421.
  * [15] J. S. Chung and A. Zisserman, “Lip reading in profile,” 2017.
  * [16] A. Fernandez-Lopez, O. Martinez, and F. M. Sukno, “Towards estimating the upper bound of visual-speech recognition: The visual lip-reading feasibility database,” in _2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017)_. IEEE, 2017, pp. 208–215.
  * [17] S. Yang, Y. Zhang, D. Feng, M. Yang, C. Wang, J. Xiao, K. Long, S. Shan, and X. Chen, “Lrw-1000: A naturally-distributed large-scale benchmark for lip reading in the wild,” in _2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019)_. IEEE, 2019, pp. 1–8.
  * [18] E. Egorov, V. Kostyumov, M. Konyk, and S. Kolesnikov, “Lrwr: Large-scale benchmark for lip reading in russian language,” _arXiv preprint arXiv:2109.06692_ , 2021.
  * [19] G. Schwiebert, C. Weber, L. Qu, H. Siqueira, and S. Wermter, “A multimodal german dataset for automatic lip reading systems and transfer learning,” _arXiv preprint arXiv:2202.13403_ , 2022.
  * [20] D. Ivanko, A. Axyonov, D. Ryumin, A. Kashevnik, and A. Karpov, “Rusavic corpus: Russian audio-visual speech in cars,” in _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , 2022, pp. 1555–1559.
  * [21] J. Peymanfard, A. Lashini, S. Heydarian, H. Zeinali, and N. Mozayani, “Word-level persian lipreading dataset,” in _2022 12th International Conference on Computer and Knowledge Engineering (ICCKE)_. IEEE, 2022, pp. 225–230.
  * [22] J. Roth, S. Chaudhuri, O. Klejch, R. Marvin, A. Gallagher, L. Kaver, S. Ramaswamy, A. Stopczynski, C. Schmid, Z. Xi _et al._ , “Ava active speaker: An audio-visual dataset for active speaker detection,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 4492–4496.
  * [23] Y. J. Kim, H.-S. Heo, S. Choe, S.-W. Chung, Y. Kwon, B.-J. Lee, Y. Kwon, and J. S. Chung, “Look who’s talking: Active speaker detection in the wild,” _arXiv preprint arXiv:2108.07640_ , 2021.
  * [24] A. Nagrani, J. S. Chung, and A. Zisserman, “Voxceleb: a large-scale speaker identification dataset,” in _INTERSPEECH_ , 2017.
  * [25] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” in _INTERSPEECH_ , 2018.
  * [26] H. Zeinali, H. Sameti, and T. Stafylakis, “Deepmine speech processing database: Text-dependent and independent speaker verification and speech recognition in persian and english.” in _Odyssey_ , 2018, pp. 386–392.
  * [27] Y. Fan, J. Kang, L. Li, K. Li, H. Chen, S. Cheng, P. Zhang, Z. Zhou, Y. Cai, and D. Wang, “Cn-celeb: a challenging chinese speaker recognition dataset,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7604–7608.
  * [28] Y. M. Assael, B. Shillingford, S. Whiteson, and N. de Freitas, “Lipnet: Sentence-level lipreading,” _CoRR_ , vol. abs/1611.01599, 2016. [Online]. Available: http://arxiv.org/abs/1611.01599
  * [29] J. Peymanfard, M. R. Mohammadi, H. Zeinali, and N. Mozayani, “Lip reading using external viseme decoding,” in _2022 International Conference on Machine Vision and Image Processing (MVIP)_. IEEE, 2022, pp. 1–5.
  * [30] T. Afouras, A. Zisserman _et al._ , “Sub-word level lip reading with visual attention,” _arXiv preprint arXiv:2110.07603_ , 2021.
  * [31] S. Petridis, T. Stafylakis, P. Ma, G. Tzimiropoulos, and M. Pantic, “Audio-visual speech recognition with a hybrid ctc/attention architecture,” in _2018 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2018, pp. 513–520.
  * [32] T. Afouras, J. S. Chung, A. Senior, O. Vinyals, and A. Zisserman, “Deep audio-visual speech recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , 2018.
  * [33] P. Ma, S. Petridis, and M. Pantic, “End-to-end audio-visual speech recognition with conformers,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 7613–7617.
  * [34] S. Schneider, A. Baevski, R. Collobert, and M. Auli, “wav2vec: Unsupervised pre-training for speech recognition,” _arXiv preprint arXiv:1904.05862_ , 2019.
  * [35] A. Baevski, S. Schneider, and M. Auli, “vq-wav2vec: Self-supervised learning of discrete speech representations,” _arXiv preprint arXiv:1910.05453_ , 2019\.
  * [36] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [37] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [38] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” _arXiv preprint arXiv:2201.02184_ , 2022.
  * [39] M. McLaren, L. Ferrer, D. Castán, and A. D. Lawson, “The speakers in the wild (sitw) speaker recognition database,” in _INTERSPEECH_ , 2016.
  * [40] S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3fd: Single shot scale-invariant face detector,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 192–201.
  * [41] R. Tao, Z. Pan, R. K. Das, X. Qian, M. Z. Shou, and H. Li, “Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 3927–3935.
  * [42] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4690–4699.
  * [43] M. Aghaahmadi, M. M. Dehshibi, A. Bastanfard, and M. Fazlali, “Clustering persian viseme using phoneme subspace for developing visual speech application,” _Multimedia tools and applications_ , vol. 65, no. 3, pp. 521–541, 2013.
  * [44] B. Shi, W.-N. Hsu, K. Lakhotia, and A. Mohamed, “Learning audio-visual speech representation by masked multimodal cluster prediction,” _arXiv preprint arXiv:2201.02184_ , 2022.
  * [45] N. R. Koluguri, T. Park, and B. Ginsburg, “Titanet: Neural model for speaker representation with 1d depth-wise separable convolutions and global context,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 8102–8106.
  * [46] J. Godfrey and E. Holliman, “Switchboard-1 release 2 ldc97s62,” _Linguistic Data Consortium_ , 1993.
  * [47] C. Cieri, D. Graff, O. Kimball, D. Miller, and K. Walker, “Fisher english training speech part 1 transcripts,” _Philadelphia: Linguistic Data Consortium_ , 2004.
  * [48] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.