# Enhanced weak force sensing through atom-based coherent noise cancellation
in a hybrid cavity optomechanical system

S.K. Singh  School of Physical Sciences, Central University of Karnataka,
Kalaburagi-585367, India  M. Mazaheri<EMAIL_ADDRESS>Department
of Basic Science, Hamedan University of Technology, Hamedan, 65169-1-3733,
Iran  Jia-Xin Peng<EMAIL_ADDRESS>State Key Laboratory of Precision
Spectroscopy, Quantum Institute for Light and Atoms, Department of Physics,
East China Normal University, Shanghai 200062, People’s Republic of China  M.
Asjad  Electrical and Computer Engineering Department, Abu Dhabi University,
Abu Dhabi 59911, United Arab Emirates

###### Abstract

## Abstract

We investigate weak force sensing based on coherent quantum noise cancellation
in a nonlinear hybrid optomechanical system. The optomechanical cavity
contains a moveable mechanical mirror, a fixed semitransparent mirror, an
ensemble of ultracold atoms, and an optical parametric amplifier (OPA). Using
the coherent quantum noise cancellation (CQNC) process, one can eliminate the
back action noise at all frequencies. Also by tuning the OPA parameters, one
can suppress the quantum shot-noise at lower frequencies than the resonant
frequency. In the CQNC scheme, the damping rate of the mechanical oscillator
matches the damping rate of the atomic ensemble, which is experimentally
achievable even for a low-frequency mechanical oscillator with a high-quality
factor. Elimination of the back action noise and suppression of the shot noise
significantly enhance force sensing and thus overcome the standard quantum
limit of weak force sensing. This hybrid scheme can play an essential role in
the realization of quantum optomechanical sensors and quantum control.

##  I Introduction

Weak force sensing at the quantum limit of sensitivity has attracted a lot of
attention in recent decades Braginsky  _et al._ (1980); Braginsky and
Braginskiĭ . This tendency leads to the development of quantum cavity
optomechanics and the possibility of using quantum phenomena in macroscopic
dimensions Aspelmeyer  _et al._ (2014). Optomechanical sensors have given
accurate results to measure mass Bin  _et al._ (2019); Liu  _et al._ (2019),
acceleration Krause  _et al._ (2012); Qvarfort  _et al._ (2018), displacement
Wilson  _et al._ (2015); Rossi  _et al._ (2018), and force Caves  _et al._
(1980); Schreppler  _et al._ (2014); Ghobadi  _et al._ (2015). However, the
measurement accuracy of optomechanical sensors is limited by the shot noise
and the back-action noise, in which their competition are collectively called
the standard quantum limit (SQL) Bowen and Milburn (2015). The quantum non-
demolition measurement Thorne  _et al._ (1978) via quantum entanglement Ma
_et al._ (2017) or squeezing Clark  _et al._ (2016); Otterpohl  _et al._
(2019) can be used to surpass the SQL. Also, the squeezed states are
traditionally employed to detect gravitational waves via supersensitive
interferometry Hollenhorst (1979); Aasi  _et al._ (2013); Dodonov  _et al._
(1980).

In the optomechanical sensor, as the intensity of the optical field increases,
the measurement strength and the shot noise decrease while the back-action
noise increases. Therefore, to increase the force sensitivity, it is necessary
to reduce or eliminate the back-action noise. It is suggested Hertzberg  _et
al._ (2010) that measuring only a single quadrature of motion can increase the
force sensitivity. In this method, the effect of the back-action noise on the
orthogonal quadrature in the measurement is isolated. Woolly and Clerk Woolley
and Clerk (2013) proposed a three-mode optomechanical system to evade the
back-action noise, in which two mechanical oscillators are coupled to a single
cavity mode. Hammerer $\it{et~{}al.}$ Hammerer  _et al._ (2009) introduced an
anti-noise path in the system to cancel the back-action noise of light via
quantum interference. They showed that an ancillary atomic oscillator could
exhibit an opposite and equal response to the optical field in the
optomechanical cavity. In this way, the CQNC process can delete the back-
action noise, which is induced by the radiation pressure at all frequencies,
and surpass the SQL Tsang and Caves (2010, 2012). The atomic ensemble can be
coupled to the optomechanical systems, internally or externally, to improve
the optomechanical cooling Genes  _et al._ (2009); Hammerer  _et al._ (2010);
Genes  _et al._ (2011); Camerer  _et al._ (2011) and create the best regime
relevant to force sensing at the low mechanical frequency with a high-quality
factor Wimmer  _et al._ (2014).

In addition, the interaction between the atomic ensemble and mechanical
oscillator has been used to provide entanglement, entangled Einstein-Podolsky-
Rosen (EPR) states, and squeezed states in the hybrid optomechanical system
Hammerer  _et al._ (2009). The combination of the optomechanical system, the
atomic two-level system and also the OPA can be utilized as an applicable
candidate for force sensing Wasilewski  _et al._ (2010); Muschik  _et al._
(2011); Polzik and Hammerer (2015). Recently, the cavity optomechanical system
with the OPA is used to implement ultrahigh-precision position detection Peano
_et al._ (2015). The advantage of this scheme is that the information is
imprinted on the momentum quadrature, which is not amplified. It leads to a
limited suppression of signal but simultaneously reduces the noise
dramatically. Then, one can obtain the SQL without losing quantum efficiency
at a mechanical resonance in the squeezed resonator. On the other hand, using
the OPA, the SQL can be attained in a dissipative cavity optomechanical system
Huang and Agarwal (2017). So far, the hybrid optomechanical system, including
the OPA and the atomic ensemble, is not used to enhance force sensing.

He et al. He  _et al._ (2020) used the hybrid optomechanical system,
containing the degenerate OPA and the atomic ensemble, to investigate the
displacement spectrum of the moveable mirror and the spectrum of squeezed
cavity quadrature field. They showed that this hybrid optomechanical system
leads to a wider spacing between the two peaks in the normal mode splitting
and a more significant degree of the squeezing spectrum compared with the case
of the optomechanical system with just one of the atomic ensemble or the OPA.
Furthermore, the peak in the squeezed spectrum diminishes at lower
temperatures due to the reduction of mechanical thermal noise.

Under these considerations, the present paper aims to study the CQNC force
sensing in the hybrid optomechanical system containing both the OPA and the
atomic ensemble. The mechanical oscillator, coupled to the atomic ensemble as
the oscillator with the effective negative mass, is used for force sensing.
Based on the scheme of Tsang $\it{et~{}al.}$ Tsang and Caves (2010, 2012), the
coupling between the optomechanical cavity and the atomic ensemble is
necessary for the CQNC process. On the other hand, tuning the OPA parameters
can improve the precision of force sensing and reach the sub-SQL sensitivity
at small values of the optomechanical coupling without losing quantum
efficiency Zhao  _et al._ (2020). The rest of the present paper is organized
as follows. In section II, we present the hybrid optomechanical system and
model for optomechanical force sensing and derive the equations of motion. The
central conditions for ideal CQNC and shot noise reduction are given based on
the physical parameters of the system. In section III, we discuss force
sensing and the feasibility of physical parameters of the system. A summary of
our main results is provided in section IV.

##  II Hybrid optomechanical system and model

We consider the hybrid optomechanical system, which contains a moveable mirror
known as a mechanical oscillator, a trapped ultracold atomic ensemble and an
optical parametric amplifier (OPA), as shown in Fig. 1. The atomic ensemble is
driven by the intracavity field and a classical field. It can generate a
strong atom-field coupling to store and transfer quantum states through the
collective atomic excitation Akulshin  _et al._ (1998); Li and Sun (2004); Ian
_et al._ (2008); Chang  _et al._ (2011). The optical amplifier is used to
prevent a loss in the injected squeezed light. The single-mode Fabry-Perot
optical cavity includes the fixed semitransparent mirror and the movable fully
reflective mirror, equivalent to a quantum harmonic oscillator with mass $m$
and frequency $\omega_{m}$. The optical cavity interacts with the moveable
mirror through the radiation pressure. The optical cavity is driven by
applying a strong laser field to the semitransparent mirror.

Figure 1: (Color online) Overview of the hybrid optomechanical system which
consists of an optomechanical cavity, a degenerate optical parametric
amplifier (OPA), and an ensemble of ultracold atoms.

Hamiltonian of the optomechanical system (OMS), Hamiltonian of the OPA Zhao
_et al._ (2020), Hamiltonian of the atomic ensemble, and the Hamiltonian of
the laser driving field Motazedifard  _et al._ (2019) can be written as the
following expressions:

$\displaystyle
H_{OMS}=\hbar\Delta_{a}a^{{\dagger}}a+\frac{\hbar\omega_{m}}{2}(X^{2}+P^{2})+\hbar
g_{0}a^{{\dagger}}aX,$ (1) $\displaystyle H_{OPA}=i\hbar
G(e^{i\theta}a^{{\dagger}2}-e^{-i\theta}a^{2}),$ (2) $\displaystyle
H_{at}=-\hbar\omega_{m}d^{{\dagger}}d+\frac{\hbar
G^{\prime}}{2}(a+a^{{\dagger}})(d+d^{{\dagger}}),$ (3)

and

$\displaystyle H_{L}=i\hbar E_{L}(a^{{\dagger}}-a),$ (4)

where $a$ is the annihilation operator of the optical field, $X$ and $P$ are
the dimensionless position and momentum operators of the mechanical oscillator
which are normalized to zero point motion $X_{zp}=\sqrt{\hbar/(m\omega_{m})}$
and zero point momentum $P_{zp}=\hbar/X_{zp}$, respectively. The parameters
$g_{0}$ and $G$ are the single photon optomechanical coupling and the
nonlinear gain of the degenerate OPA, and $\theta$ is the phase of the OPA
pump field. $E_{L}=\sqrt{P\kappa/\omega_{L}}$ stands for the pump rate of
external driving laser in which $P$, $\omega_{L}$, and $\kappa$ are the input
laser driving power, the input laser frequency, and the input photon decay
rate, respectively. We also used $d$ and $G^{\prime}$ to denote the effective
atomic bosonic annihilation operator and the effective atom-field coupling for
the atomic ensemble which are thoroughly discussed in Ref. Motazedifard  _et
al._ (2016). Hamiltonian of the system in the interaction picture containing
the two level atomic ensemble and the OPA can be written as

$\displaystyle
H_{I}=\hbar\Delta_{a}a^{{\dagger}}a+\frac{\hbar\omega_{m}}{2}(X^{2}+P^{2})+\hbar
g_{0}a^{{\dagger}}aX$ $\displaystyle+i\hbar
G(e^{i\theta}a^{{\dagger}2}+e^{-i\theta}a^{2})-\hbar\omega_{m}d^{{\dagger}}d$
$\displaystyle+\frac{\hbar
G^{\prime}}{2}(a+a^{{\dagger}})(d+d^{{\dagger}})+i\hbar
E_{L}(a^{{\dagger}}-a).$ (5)

Here, $\Delta_{a}=\omega_{a}-\omega_{L}$ is the detuning of the optical mode
from the driving laser frequency in which $\omega_{a}$ is the frequency of the
optical cavity Bariani  _et al._ (2015); Motazedifard  _et al._ (2016). Using
the quantum Heisenberg-Langevin equations of motion, the equations of motion
for the present system can be given as the following nonlinear equations

$\displaystyle\dot{X}=\omega_{m}P,$ (6)
$\displaystyle\dot{P}=-\omega_{m}X-g_{0}a^{{\dagger}}a-\gamma_{m}P+\sqrt{2\gamma_{m}}(\hat{F}_{th}+F_{ext}),$
(7)
$\displaystyle\dot{a}=-(i\Delta_{a}+\frac{\kappa}{2})-ig_{0}Xa+2Ge^{i\theta}a^{{\dagger}}$
$\displaystyle-\frac{iG^{\prime}}{2}(d+d^{\prime})+\sqrt{\kappa}a_{in},$ (8)
$\displaystyle\dot{d}=i\omega_{m}d-\frac{G^{\prime}}{2}(a+a^{{\dagger}})-\frac{\Gamma}{2}+\sqrt{\Gamma}d_{in},$
(9)

where $\gamma_{m}$, $\kappa$ and $\Gamma$ are the mechanical damping rate, the
cavity photon decay rate and the collective atomic dephasing rate,
respectively. The mechanical noise term is $\hat{F}_{th}$, and $F_{ext}$ is
the external force to be measured which is applied to the mechanical
oscillator, as shown in Fig. 1. The external force is normalized to
$\sqrt{\hbar m\omega_{m}\gamma_{m}}$ that is expressed in units of $\sqrt{\rm
Hz}$. The equations of motion contain the dissipation and fluctuation noise
terms associating with the parameters of the system.

In the presence of strong optical drive, the quantum dynamics of the system
can be well described by the linearizing the nonlinear Heisenberg-Langevin
equations of motion, as shown in Refs. Genes  _et al._ (2008); Vitali  _et
al._ (2007). Therefore, we expand each operator, in Eqs. (6)-(9), as a steady
state part and a quantum fluctuating part, as discussed theoretically Huang
and Agarwal (2017); Gebremariam  _et al._ (2020) and experimentally Safavi-
Naeini  _et al._ (2013); Shomroni  _et al._ (2019). Then, we suppose that each
operator in Eqs. (6)-(9) contains the steady state values ($\bar{O}$) and the
fluctuating part ($\hat{O}$), including $P=\bar{P}+\hat{P}$,
$X=\bar{X}+\hat{X}$, $a=\alpha+\hat{a}$, and $d=\bar{d}+\hat{d}$. We take the
amplitude of intra cavity mode as a c-number, $\alpha=|{\alpha}|e^{i\phi}$,
with a phase $\phi$. In order to summarize the equations of motion, the
standard position and momentum quadratures of the cavity and atomic fields are
defined as $\hat{x}_{i}=(\hat{a}_{i}+\hat{a}_{i}^{{\dagger}})/\sqrt{2}$ and
$\hat{p}_{i}=(\hat{a}_{i}-\hat{a}_{i}^{{\dagger}})/(i\sqrt{2})$, and in
addition, there are similar definitions for the amplitude and phase of the
input quantum noises as
$\hat{x}_{i}^{in}=(\hat{a}_{i}^{in}+\hat{a}_{i}^{in,{\dagger}})/\sqrt{2}$ and
$\hat{p}_{i}^{in}=(\hat{a}_{i}^{in}-\hat{a}_{i}^{in,{\dagger}})/(i\sqrt{2})$,
where $i$ stands for the cavity field mode ($\hat{a}$) and the atomic field
mode ($\hat{d}$).

After straightforward calculations in the strong driving regime ($\\\
vert{\alpha}|\gg 1$), the quantum fluctuation parts of the linearized quantum
Heisenberg-Langevin equations of motion are obtained in terms of the standard
quadratures as

$\displaystyle\dot{\hat{X}}=\omega_{m}\hat{P},$ (10)
$\displaystyle\dot{\hat{P}}=-\omega_{m}\hat{X}-\gamma_{m}\hat{P}-g\cos{\phi}\hat{x}_{a}+g\sin{\phi}\hat{p}_{a}$
$\displaystyle+\sqrt{2\gamma_{m}}(\hat{F}_{th}+F_{ext}),$ (11)
$\displaystyle\dot{\hat{x}}_{a}=g\sin{\phi}\hat{X}+c_{-}\hat{x}_{a}+s_{+}\hat{p}_{a}+\sqrt{\kappa}\hat{x}^{in}_{a},$
(12)
$\displaystyle\dot{\hat{p}}_{a}=-g\cos{\phi}\hat{X}+s_{-}\hat{x}_{a}-c_{+}\hat{p}_{a}-G^{\prime}\hat{x}_{d}+\sqrt{\kappa}\hat{p}^{in}_{a},$
(13)
$\displaystyle\dot{\hat{x}}_{d}=-\frac{\Gamma}{2}\hat{x}_{d}-\omega_{m}\hat{p}_{d}+\sqrt{\Gamma}\hat{x}^{in}_{d},$
(14)
$\displaystyle\dot{\hat{p}}_{d}=-G^{\prime}\hat{x}_{a}+\omega_{m}\hat{x}_{d}-\frac{\Gamma}{2}\hat{p}_{d}+\sqrt{\Gamma}\hat{p}^{in}_{d}.$
(15)

where parameters are defined as $g=\sqrt{2}g_{0}|{\alpha}|$ known as the
effective linearized optomechanical coupling,
$c_{\pm}=\pm\frac{\kappa}{2}+2G\cos{\theta}$ and
$s_{\pm}=\pm\Delta+2G\sin{\theta}$. The laser driving power, $P$, is
proportional to $g^{2}$ as the measurement strength. The equations of motion,
Eqs. (10)-(15), describe the CQNC which are similar to the equations of motion
in Refs. Wimmer  _et al._ (2014); Bariani  _et al._ (2015).

The atomic ensemble, as the quantum oscillator, is coupled to the
optomechanical cavity but interacts with the cavity field in the exactly
opposite way with an effective negative mass. Now, in order to cancel the back
action noise we assume that the system is set at the resonance condition,
$\Delta=0$, the $\alpha$ is real which means the intracavity phase is zero,
$\phi=0$, and finally we assume the case of zero pump phase, $\theta=0$, of
the optomechanical system, then Eqs. (10)-(15) are changed to

$\displaystyle\dot{\hat{X}}=\omega_{m}\hat{P},$ (16)
$\displaystyle\dot{\hat{P}}=-\omega_{m}\hat{X}-\gamma_{m}\hat{P}-g\hat{x}_{a}+\sqrt{2\gamma_{m}}(\hat{F}_{th}+F_{ext}),$
(17)
$\displaystyle\dot{\hat{x}}_{a}=(-\frac{\kappa}{2}+2G)\hat{x}_{a}+\sqrt{\kappa}\hat{x}^{in}_{a},$
(18)
$\displaystyle\dot{\hat{p}}_{a}=-g\hat{X}+(-\frac{\kappa}{2}-2G)\hat{p}_{a}-G^{\prime}\hat{x}_{d}+\sqrt{\kappa}\hat{p}^{in}_{a},$
(19)
$\displaystyle\dot{\hat{x}}_{d}=-\frac{\Gamma}{2}\hat{x}_{d}-\omega_{m}\hat{p}_{d}+\sqrt{\Gamma}\hat{x}^{in}_{d},$
(20)
$\displaystyle\dot{\hat{p}}_{d}=-G^{\prime}\hat{x}_{a}+\omega_{m}\hat{x}_{d}-\frac{\Gamma}{2}\hat{p}_{d}+\sqrt{\Gamma}\hat{p}^{in}_{d}.$
(21)

It can be seen from Eq. (19) that the optical field momentum quadrature
($p_{a}$) depends on the position ($X$) of the mechanical oscillator and in
turn the position is controlled by the momentum ($P$) of the mechanical
oscillator. The external force can be detected by monitoring the optical mode.
The equations of motion, Eqs. (16)-(21), can be solved for the operators in
the frequency domain to calculate $p_{a}(\omega)$ as a function of the input
noises. A quantum operator in the Fourier domain can be written as

$\displaystyle\hat{O}(\omega)=\frac{1}{\sqrt{2\pi}}\int{dt\hat{O}(t)e^{-i\omega
t}},$ (22)

then using Eq. (22), Eqs. (16)-(21) can be given in the Fourier space as

$\displaystyle\hat{X}=\chi_{m}(-g\hat{x}_{a}+\sqrt{2\gamma_{m}}(\hat{F}_{th}+F_{ext})),$
(23) $\displaystyle\hat{P}=\frac{i\omega}{\omega_{m}}\hat{X},$ (24)
$\displaystyle\hat{x}_{a}=\sqrt{\kappa}\lambda_{+}\hat{x}_{a}^{in},$ (25)
$\displaystyle\hat{p}_{a}=g^{2}\chi_{m}\lambda_{-}\lambda_{+}\sqrt{\kappa}\hat{x}_{a}^{in}-g\chi_{m}\lambda_{-}\sqrt{2\gamma_{m}}(\hat{F}_{th}+F_{ext})$
$\displaystyle-G^{\prime}\lambda_{-}\hat{x}_{d}+\lambda_{-}\sqrt{\kappa}\hat{p}_{a}^{in},$
(26)
$\displaystyle\hat{x}_{d}=-\omega_{m}\chi_{d}\hat{p}_{d}+\chi_{d}\sqrt{\Gamma}\hat{x}^{in}_{d},$
(27)
$\displaystyle\hat{p}_{d}=-G^{\prime}\xi\lambda_{+}\sqrt{\kappa}\hat{x}_{a}^{in}-\chi^{\prime}_{d}\sqrt{\Gamma}\hat{x}_{d}^{in}+\xi\sqrt{\Gamma}\hat{p}^{in}_{d},$
(28)

where, the parameters are defined as

$\displaystyle\lambda_{\pm}=(\chi_{a}^{-1}\mp 2G)^{-1},$ (29)
$\displaystyle\xi=(i\omega+\Gamma/2+\omega_{m}^{2}\chi_{d})^{-1}.$ (30)

Also, the effective susceptibilities can be written as

$\displaystyle\chi_{m}=\frac{\omega_{m}}{\omega_{m}^{2}-\omega^{2}+i\omega\gamma_{m}},$
(31) $\displaystyle\chi_{a}=\frac{1}{i\omega+\kappa/2},$ (32)
$\displaystyle\chi_{d}=\frac{1}{i\omega+\Gamma/2},$ (33)
$\displaystyle\chi^{\prime}_{d}=-\omega_{m}\xi\chi_{d}=\frac{-\omega_{m}}{\omega_{m}^{2}-\omega^{2}+i\omega\Gamma/2+\Gamma^{2}/4},$
(34)

for the mechanical oscillator, the meter optical cavity, the atomic ensemble,
and the last one stands for the generalized susceptibility of the atomic
ensemble, respectively.

##  III Force sensing and CQNC

Solving Eqs. (23)-(28),we can find the phase quadrature of the cavity field,
$p_{a}$, and finally using the standard input-output relation as

$\displaystyle\hat{p}_{a}^{out}=\sqrt{\kappa}\hat{p}_{a}-\hat{p}_{a}^{in},$
(35)

the detected phase quadrature of the cavity field mode can be expressed in
terms of the input noises as the following expression

$\displaystyle\hat{p}_{a}^{out}=-g\chi_{m}\lambda_{-}\sqrt{2\gamma_{m}\kappa}(\hat{F}_{th}+F_{ext})$
$\displaystyle+(g^{2}\chi_{m}+G^{{}^{\prime}2}\chi^{\prime}_{d})\lambda_{+}\lambda_{-}\kappa\hat{x}_{a}^{in}+(\lambda_{-}\kappa-1)\hat{p}_{a}^{in}$
$\displaystyle-G^{\prime}\lambda_{-}\sqrt{\kappa\Gamma}[\chi_{d}(\omega_{m}\chi^{\prime}_{d}+1)\hat{x}_{d}^{in}+\chi_{d}^{\prime}\hat{p}_{d}^{in}].$
(36)

The first term in Eq. (III) represents the thermal noise and the external
force where the external force is independent of the detected frequency, the
second line contains the back-action noise and the shot noise of the field,
and the third line describes the atomic noise. If we choose the conditions
such that

$\displaystyle g^{2}\chi_{m}+G^{{}^{\prime}2}\chi^{\prime}_{d}=0$ (37)

for all frequencies, then the first term in the second line of Eq. (III), the
back-action term, will be entirely canceled in Eq. (III) and also in the
external force measurement. In the CQNC scheme, we essentially have
$g=G^{{}^{\prime}}$ and $\chi_{m}=-\chi^{\prime}_{d}$ or, in other words, the
contributions to the back-action noise from the mechanical oscillator and the
atomic ensemble cancel each other for all frequencies. They are the noise and
anti-noise contributions to the signal obtained by assuming the use of an
atomic ensemble as an oscillator with the effective negative mass. In addition
to the above assumptions, we assume that the mechanical resonator and the
atomic ensemble have the same damping rates ($\gamma_{m}=\Gamma/2$) and also
the mechanical oscillator has a high-quality factor such that
$\Gamma\ll\omega_{m}$, then we can ignore the last term in the denominator of
the generalized atomic susceptibility in Eq. (34). Therefore, the
susceptibilities of mechanics and generalized atomic ensemble completely
match, which leads to the perfect coherent back-action noise cancellation.

By rewriting Eq. (III), the relation between the external force, $F_{ext}$,
and the measured phase quadrature, $\hat{p}_{a}^{out}$, can be given as the
following expression

$\displaystyle
F_{ext}+\hat{F}_{add}=\frac{-1}{g\chi_{m}\lambda_{-}\sqrt{2\gamma_{m}\kappa}}\hat{p}_{a}^{out}$
(38)

where the added force noise, $\hat{F}_{add}$, is defined as

$\displaystyle\hat{F}_{add}=\hat{F}_{th}-\frac{\lambda_{-}\kappa-1}{g\chi_{m}\lambda_{-}\sqrt{2\gamma_{m}\kappa}}\hat{p}_{a}^{in}$
$\displaystyle+\frac{G^{\prime}\chi_{d}^{\prime}\sqrt{\kappa\Gamma}}{g\chi_{m}\sqrt{2\gamma_{m}\kappa}}[\frac{\chi_{d}}{\chi_{d}^{\prime}}(\omega_{m}\chi^{\prime}_{d}+1)\hat{x}_{d}^{in}+\hat{p}_{d}^{in}].$
(39)

It should be noted that the back-action noise term is canceled. Simplifying
Eq. (III), we have

$\displaystyle\hat{F}_{add}=\hat{F}_{th}-\frac{\lambda_{-}\kappa-1}{g\chi_{m}\lambda_{-}\sqrt{2\gamma_{m}\kappa}}\hat{p}_{a}^{in}$
$\displaystyle+[-\frac{i\omega+\Gamma/2}{\omega_{m}}\hat{x}_{d}^{in}+\hat{p}_{d}^{in}].$
(40)

The added noise contains the thermal Langevin force coupled to a thermal
reservoir at temperature $T$ (first term), the shot noise in the phase
quadrature of the optical field (second term) and the atomic noise (third
term).

Using the definition of spectral density of the added noise Wimmer  _et al._
(2014), $S_{F,add}(\omega)$, we can find the sensitivity of the force
measurement as

$\displaystyle
S_{F,add}(\omega)\delta(\omega-\omega^{\prime})=\frac{1}{2}(\left\langle\hat{F}_{add}(\omega)\hat{F}_{add}(-\omega^{\prime})\right\rangle+c.c.),$
(41)

then, using the conditions of the CQNC and for $\omega\ll\kappa$, we find the
spectral noise as

$\displaystyle
S_{F,add}(\omega)=\frac{k_{B}T}{\hbar\omega_{m}}+\frac{1}{2}\frac{1}{g^{2}|\chi_{m}(\omega)|^{2}(2\gamma_{m}\kappa)}|\frac{\lambda_{-}(\omega)\kappa-1}{\lambda_{-}(\omega)}|^{2}$
$\displaystyle+\frac{1}{2}(\frac{\omega^{2}+\Gamma^{2}/4}{\omega_{m}^{2}}+1).$
(42)

After some calculation, the spectral density can be more simplified as

$\displaystyle
S_{F,add}(\omega)=\frac{k_{B}T}{\hbar\omega_{m}}+\frac{1}{2}[\frac{(\kappa/2-2G)^{2}}{g^{2}|\chi_{m}(\omega)|^{2}(2\gamma_{m}\kappa)}$
$\displaystyle(\frac{\omega^{2}+\Gamma^{2}/4}{\omega_{m}^{2}}+1)],$ (43)

where obviously the back-action noise term, scaling to $g^{2}$, is deleted.
This result can be compared to that of the standard optomechanical system,
discussed in Refs. Aspelmeyer  _et al._ (2014); Wimmer  _et al._ (2014);
Bariani  _et al._ (2015), as the following equation

$\displaystyle
S_{F}(\omega)=\frac{k_{B}T}{\hbar\omega_{m}}+\frac{1}{2}[\frac{\kappa}{\gamma_{m}}\frac{1}{g^{2}|\chi_{m}|^{2}}\frac{1}{4}+4g^{2}\frac{1}{\kappa\gamma_{m}}]$
(44)

which contains the shot noise and the back-action noise terms proportional to
$1/g^{2}$ and $g^{2}$, respectively. The thermal Brownian noise results in a
constant background noise to the force sensitivity independent of the input
laser power. At $T=0$, we can minimize the right hand side of Eq. (44) with
respect to $g^{2}$ or proportionally the input laser power,
$P=2\hbar\omega_{L}\kappa(g/g_{0})^{2}$, gives the SQL for continuous force
sensing,

$\displaystyle S_{F,SQL}=\frac{1}{\gamma_{m}|\chi_{m}|}$ (45)

and equivalently the minimized noise spectral density can be found for the
CQNC at $T=0$ as

$\displaystyle
S_{F,CQNC}=\frac{1}{2}\frac{\omega^{2}+\omega_{m}^{2}+\Gamma^{2}/4}{\omega_{m}^{2}}.$
(46)

The noise spectral densities are normalized by the factor, $\hbar
m\omega_{m}\gamma_{m}$, then they are given in units of N2Hz-1. However, in
this work, we have considered the noise spectral densities as dimensionless
for simplicity.

The noise spectral densities for the standard optomechanical system (SQL), the
optomechanical system with OPA (dashed lines) and the CQNC schemes (CQNC) are
represented in Fig. 2. The spectral density for the CQNC scheme is limited to
the shot noise term over the whole detection bandwidth, which is the main
advantage of the CQNC scheme. Fig. 2 shows the variation of the noise spectral
density of the optomechanical system versus the detection frequency for
different values of the OPA pump gain, $G$. At the resonance condition
($\Delta=0$), the noise spectral density of the standard optomechanical system
and that of the optomechanical system with the OPA equal the noise spectral
density of the CQNC scheme. Fig. 2 shows that with the increase of relative
OPA pump gain, $G/\kappa$, the SQL can be suppressed at frequencies below and
as well above the mechanical resonance. The force noise spectral density is
decreased by nearly two orders of magnitude on both sides of the low and high-
frequency domains. Recently, Korobko et al. Korobko  _et al._ (2019) proposed
an optical approach to expand the detection bandwidth for gravitational-wave
observatories in which quantum uncertainty can be squeezed inside one of the
optical resonators at high frequencies. At the same time, the low-frequency
sensitivity was remained unchanged. On the other hand, as mentioned in section
I, Zhao et al. Zhao  _et al._ (2020) investigated weak-force sensing in a
squeezed cavity and theoretically showed that the SQL could be suppressed at
frequencies below the mechanical resonance; however, the back-action noise
remained unchanged in the system.

Figure 2: (Color online) Noise power spectral densities calculated for the
standard optomechanical system (black solid line), $S_{SQL}$, the
optomechanical system with the OPA (colored dashed line) with different $G$,
and the coherent quantum noise cancellation (red dashed line), $S_{CQNC}$, in
optimal power versus the detection frequency. The spectral densities are
normalized to $\hbar m\omega_{m}\gamma_{m}$. The parameters are given in Table
1 and we have $Q=\omega_{m}/\gamma_{m}=10^{4}$.

The spectral densities are depicted in Fig. 3(a) and Fig. 3(b) as functions of
the laser driving power, scaling to the optomechanical coupling $g^{2}$, for
on-resonance ($\omega=\omega_{m}$) and off-resonance
($\omega=\omega_{m}+4\gamma_{m}$) cases, respectively. Fig. 3(a) shows that at
the higher driving power, the noise spectral density decreases over the
bandwidth significantly. At lower values of the coupling strength, the shot
noise term is dominant, and the noise spectral density of the normal
optomechanical system (SQL) first decreases with the driving power increasing
until the minimum point. After that, the back-action noise is dominant, and
the noise spectral density of the normal optomechanical system increases. In
the normal optomechanical system, the noise spectral density of the SQL
represented by Eq. (45) can be reached to the minimum point at
$g_{SQL}=\sqrt{\kappa}/(2\sqrt{|\chi_{m}|})$, whereas this point in the normal
system can be lowered by using the OPA pump with gain $G$, which gives
$g_{SQ}^{\theta=0}=|\kappa-4G|/(2\sqrt{\kappa|\chi_{m}|})$ Zhao  _et al._
(2020). The CQNC scheme cancels the back-action noise. It improves the
sensitivity in a broad bandwidth at higher values of the driving power. In the
absence of OPA pumping ($G=0$), on-resonance, the CQNC system needs a higher
driving power to reach the same sensitivity of the standard setup. In
contrast, as the OPA pump gain $G$ increases, the sensitivity of the CQNC-OPA
arrangement rapidly becomes superior to the normal optomechanical system and
surpasses the SQL at lower values of the driving power. In addition to the
cancellation of the back-action noise, the CQNC-OPA arrangement reduces the
shot noise by about two orders of magnitude. It improves the measurement
accuracy at lower driving power domain. On the other side, away from the
resonance case with detuning $4\gamma_{m}$, the spectral density of the
system, $S_{F}$, decreases with increasing the values of the OPA pump gain
$G$, as depicted in Fig. 3(b). We have used experimentally accessible
parameters Bariani  _et al._ (2015); Bowen and Milburn (2015); Wimmer  _et
al._ (2014) in our calculations which are collected in Table 1.

Table 1: Parameters of the hybrid system. Parameter | Symbole | Value (unit)
---|---|---
Mirror mass | $m$ | 50 ng
Single photon optomechanical coupling | $g_{0}/2\pi$ | 300 Hz
Mechanical resonance | $\omega_{m}/2\pi$ | 300 kHz
Mechanical damping rate | $\gamma_{m}/2\pi$ | 30 Hz
Optical cavity damping rate | $\kappa/2\pi$ | 1 MHz
Laser source power | $P$ | 100 mW
Laser frequency | $\omega_{L}/2\pi$ | 384 THz

Figure 3: (Color online) Noise spectral density for weak force measurement (a)
on mechanical resonance ($\omega=\omega_{m}$) and (b) off-resonance with
$4\gamma_{m}$ detuning ($\omega=\omega_{m}+4\gamma_{m}$) as function of the
input driving power ($P=2\hbar\omega_{L}\kappa(g/g_{0})^{2}$) for the standard
optomechanical system and for the present hybrid system, containing the OPA
and the atomic ensemble, with different values of the relative parametric gain
as $G/\kappa=0$, 0.05, 0.1, 0.15, and 0.2. The parameters are given in Table
1.

To reach the SQL, we have considered the impact of $g^{2}/g^{2}_{SQL}$, the
frequency $\omega$, and the parametric gain $G$ on the noise spectral density
of the present hybrid system. Fig. 4(a) and Fig. 4(b) depict the noise
spectral densities for the hybrid system as functions of the relative
optomechanical coupling $g^{2}/g^{2}_{SQL}$ and relative frequency
$\omega/\omega_{m}$ with $G/\kappa=0$ and 0.1, respectively. Fig. 4(a) with
$G/\kappa=0$ indicates that the noise spectral density is close to the minimum
value in a limited range in the vicinity of the resonance, and it increases in
areas farther than the resonance region. While, the noise spectral density, as
shown in Fig. 4(b) with $G/\kappa=0.1$, is close to the minimum value in a
wider area, and it increases with a lower slope in areas farther than the
resonance region, especially in areas with a frequency lower than the resonant
frequency. Fig. 4(a) shows that the noise spectral density reaches the minimum
values in the narrow area in the vicinity of the resonant frequency, which
slightly increases at higher values of $g^{2}/g^{2}_{SQL}$. While, Fig. 4(b)
with $G/\kappa=0.1$ shows that the spectral noise density approaches to the
minimum values of the SQL in a wider range of lower values of
$g^{2}/g^{2}_{SQL}$ which is interesting for feasible experimental scenarios.

Figure 4: (Color online) Noise spectral density for weak force measurement as
function of the relative squared optomechanical coupling ($(g/g_{SQL})^{2}$)
and the relative detection frequency ($\omega/\omega_{m}$) for the present
hybrid system, containing the OPA and the atomic ensemble with different
values of the relative parametric gain as (a) $G/\kappa=0$ and (b)
$G/\kappa=0.1$. The parameters are given in Table 1.

##  IV Summary and conclusion

In summary, we have presented the analysis of the coherent quantum noise
cancellation (CQNC) and shot noise reduction scheme in the optomechanical
system assisted by the ensemble of ultracold atoms and the optical parametric
amplifier (OPA). The ensemble of ultracold atoms is chosen as the effective
oscillator with the effective negative mass, which destructively interacts
with the optical cavity mode leading to the perfect cancellation of the back-
action noise. On the other hand, the quantum squeezing in the optomechanical
cavity reduces shot noise in the low driving power and the broad range of
detection frequency. In this hybrid system, the CQNC and shot noise reduction
cooperatively occur when the effective linearized optomechanical coupling and
the nonlinear gain of the OPA, the mechanical susceptibility and the
generalized susceptibility of the atomic ensemble, and eventually the damping
rates of the mechanical oscillator and the atomic ensemble coincide with each
other, respectively. Finally, we have also used a set of possible experimental
parameters which shows the elimination of the back-action noise and the
reduction of the shot noise at frequencies below the mechanical resonance.

## References

## References

  * Braginsky  _et al._ (1980) V. B. Braginsky, Y. I. Vorontsov, and K. S. Thorne, Science 209, 547 (1980).
  * (2) V. B. Braginsky and Braginskiĭ,  _Quantum measurement_.
  * Aspelmeyer  _et al._ (2014) M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Reviews of Modern Physics 86, 1391 (2014).
  * Bin  _et al._ (2019) S.-W. Bin, X.-Y. Lü, T.-S. Yin, G.-L. Zhu, Q. Bin, and Y. Wu, Optics Letters 44, 630 (2019).
  * Liu  _et al._ (2019) S. Liu, B. Liu, J. Wang, T. Sun, and W.-X. Yang, Physical Review A 99, 033822 (2019).
  * Krause  _et al._ (2012) A. G. Krause, M. Winger, T. D. Blasius, Q. Lin, and O. Painter, Nature Photonics 6, 768 (2012).
  * Qvarfort  _et al._ (2018) S. Qvarfort, A. Serafini, P. F. Barker, and S. Bose, Nature communications 9, 1 (2018).
  * Wilson  _et al._ (2015) D. J. Wilson, V. Sudhir, N. Piro, R. Schilling, A. Ghadimi, and T. J. Kippenberg, Nature 524, 325 (2015).
  * Rossi  _et al._ (2018) M. Rossi, D. Mason, J. Chen, Y. Tsaturyan, and A. Schliesser, Nature 563, 53 (2018).
  * Caves  _et al._ (1980) C. M. Caves, K. S. Thorne, R. W. Drever, V. D. Sandberg, and M. Zimmermann, Reviews of Modern Physics 52, 341 (1980).
  * Schreppler  _et al._ (2014) S. Schreppler, N. Spethmann, N. Brahms, T. Botter, M. Barrios, and D. M. Stamper-Kurn, Science 344, 1486 (2014).
  * Ghobadi  _et al._ (2015) R. Ghobadi, S. Gholizadeh, and M. Mazaheri, International Journal of Optics and Photonics 9, 19 (2015).
  * Bowen and Milburn (2015) W. P. Bowen and G. J. Milburn,  _Quantum optomechanics_ (CRC press, 2015).
  * Thorne  _et al._ (1978) K. S. Thorne, R. W. Drever, C. M. Caves, M. Zimmermann, and V. D. Sandberg, Physical Review Letters 40, 667 (1978).
  * Ma  _et al._ (2017) Y. Ma, H. Miao, B. H. Pang, M. Evans, C. Zhao, J. Harms, R. Schnabel, and Y. Chen, Nature Physics 13, 776 (2017).
  * Clark  _et al._ (2016) J. B. Clark, F. Lecocq, R. W. Simmonds, J. Aumentado, and J. D. Teufel, Nature Physics 12, 683 (2016).
  * Otterpohl  _et al._ (2019) A. Otterpohl, F. Sedlmeir, U. Vogl, T. Dirmeier, G. Shafiee, G. Schunk, D. V. Strekalov, H. G. Schwefel, T. Gehring, U. L. Andersen, _et al._ , Optica 6, 1375 (2019).
  * Hollenhorst (1979) J. N. Hollenhorst, Physical Review D 19, 1669 (1979).
  * Aasi  _et al._ (2013) J. Aasi, J. Abadie, B. Abbott, R. Abbott, T. Abbott, M. Abernathy, C. Adams, T. Adams, P. Addesso, R. Adhikari, _et al._ , Nature Photonics 7, 613 (2013).
  * Dodonov  _et al._ (1980) V. Dodonov, V. Man’ko, V. Rudenko, A. Zamolodchikov, B. Kopeliovich, L. Lapidus, M. Y. Amus’ ya, V. Ivanov, S. Sheinerman, S. Sheftel, _et al._ , Sov. Phys. JETP 51, 443 (1980).
  * Hertzberg  _et al._ (2010) J. Hertzberg, T. Rocheleau, T. Ndukum, M. Savva, A. A. Clerk, and K. Schwab, Nature Physics 6, 213 (2010).
  * Woolley and Clerk (2013) M. Woolley and A. Clerk, Physical Review A 87, 063846 (2013).
  * Hammerer  _et al._ (2009) K. Hammerer, M. Aspelmeyer, E. S. Polzik, and P. Zoller, Physical Review Letters 102, 020501 (2009).
  * Tsang and Caves (2010) M. Tsang and C. M. Caves, Physical review letters 105, 123601 (2010).
  * Tsang and Caves (2012) M. Tsang and C. M. Caves, Physical Review X 2, 031016 (2012).
  * Genes  _et al._ (2009) C. Genes, H. Ritsch, and D. Vitali, Physical Review A 80, 061803 (2009).
  * Hammerer  _et al._ (2010) K. Hammerer, K. Stannigel, C. Genes, P. Zoller, P. Treutlein, S. Camerer, D. Hunger, and T. W. Hänsch, Physical Review A 82, 021803 (2010).
  * Genes  _et al._ (2011) C. Genes, H. Ritsch, M. Drewsen, and A. Dantan, Physical Review A 84, 051801 (2011).
  * Camerer  _et al._ (2011) S. Camerer, M. Korppi, A. Jöckel, D. Hunger, T. W. Hänsch, and P. Treutlein, Physical review letters 107, 223001 (2011).
  * Wimmer  _et al._ (2014) M. H. Wimmer, D. Steinmeyer, K. Hammerer, and M. Heurs, Physical Review A 89, 053836 (2014).
  * Wasilewski  _et al._ (2010) W. Wasilewski, K. Jensen, H. Krauter, J. J. Renema, M. Balabas, and E. S. Polzik, Physical Review Letters 104, 133601 (2010).
  * Muschik  _et al._ (2011) C. A. Muschik, E. S. Polzik, and J. I. Cirac, Physical Review A 83, 052312 (2011).
  * Polzik and Hammerer (2015) E. S. Polzik and K. Hammerer, Annalen der Physik 527, A15 (2015).
  * Peano  _et al._ (2015) V. Peano, H. Schwefel, C. Marquardt, and F. Marquardt, Physical Review Letters 115, 243603 (2015).
  * Huang and Agarwal (2017) S. Huang and G. Agarwal, Physical Review A 95, 023844 (2017).
  * He  _et al._ (2020) Q. He, F. Badshah, A. Basit, P. Guo, X. Zhang, Z. Zhou, and L. Li, JOSA B 37, 911 (2020).
  * Zhao  _et al._ (2020) W. Zhao, S.-D. Zhang, A. Miranowicz, and H. Jing, SCIENCE CHINA Physics, Mechanics & Astronomy 63, 224211 (2020).
  * Akulshin  _et al._ (1998) A. Akulshin, S. Barreiro, and A. Lezama, Physical Review A 57, 2996 (1998).
  * Li and Sun (2004) Y. Li and C. Sun, Physical Review A 69, 051802 (2004).
  * Ian  _et al._ (2008) H. Ian, Z. Gong, Y.-x. Liu, C. Sun, and F. Nori, Physical Review A 78, 013824 (2008).
  * Chang  _et al._ (2011) Y. Chang, T. Shi, Y.-x. Liu, C. Sun, and F. Nori, Physical Review A 83, 063826 (2011).
  * Motazedifard  _et al._ (2019) A. Motazedifard, A. Dalafi, F. Bemani, and M. Naderi, Physical Review A 100, 023815 (2019).
  * Motazedifard  _et al._ (2016) A. Motazedifard, F. Bemani, M. Naderi, R. Roknizadeh, and D. Vitali, New Journal of Physics 18, 073040 (2016).
  * Bariani  _et al._ (2015) F. Bariani, H. Seok, S. Singh, M. Vengalattore, and P. Meystre, Physical Review A 92, 043817 (2015).
  * Genes  _et al._ (2008) C. Genes, D. Vitali, P. Tombesi, S. Gigan, and M. Aspelmeyer, Physical Review A 77, 033804 (2008).
  * Vitali  _et al._ (2007) D. Vitali, S. Gigan, A. Ferreira, H. Böhm, P. Tombesi, A. Guerreiro, V. Vedral, A. Zeilinger, and M. Aspelmeyer, Physical review letters 98, 030405 (2007).
  * Gebremariam  _et al._ (2020) T. Gebremariam, Y.-X. Zeng, M. Mazaheri, and C. Li, SCIENCE CHINA Physics, Mechanics & Astronomy 63, 210311 (2020).
  * Safavi-Naeini  _et al._ (2013) A. H. Safavi-Naeini, S. Gröblacher, J. T. Hill, J. Chan, M. Aspelmeyer, and O. Painter, Nature 500, 185 (2013).
  * Shomroni  _et al._ (2019) I. Shomroni, A. Youssefi, N. Sauerwein, L. Qiu, P. Seidler, D. Malz, A. Nunnenkamp, and T. J. Kippenberg, Physical Review X 9, 041022 (2019).
  * Korobko  _et al._ (2019) M. Korobko, Y. Ma, Y. Chen, and R. Schnabel, Light: Science & Applications 8, 1 (2019).


# MOS-FAD: IMPROVING FAKE AUDIO DETECTION VIA AUTOMATIC MEAN OPINION SCORE
PREDICTION

###### Abstract

Automatic Mean Opinion Score (MOS) prediction is employed to evaluate the
quality of synthetic speech. This study extends the application of predicted
MOS to the task of Fake Audio Detection (FAD) as we expect that MOS can be
used to assess how close synthesized speech is to the natural human voice. We
propose MOS-FAD, where MOS can be leveraged at two key points in FAD: training
data selection and model fusion. In training data selection, we demonstrate
that MOS enables effective filtering of samples from unbalanced datasets. In
the model fusion, our results demonstrate that incorporating MOS as a gating
mechanism in FAD model fusion enhances overall performance.

Index Terms—  MOS prediction, self-supervised learned (SSL) model, model
fusion, fake audio detection (FAD)

##  1 Introduction

Recent developments in text-to-speech (TTS) [1, 2, 3, 4] and voice conversion
(VC) [5] have made it feasible to create a human-like speech, which could be
misused for malevolent purposes such as spoofing attacks. As a countermeasure,
there is a growing emphasis on fake audio detection (FAD), which aims to
distinguish fake audio from real audio [6, 7, 8, 9, 10, 11].

Efforts for FAD tasks have focused on enhancing the acoustic front end to
improve the effectiveness of FAD systems. Research has shown that well-
designed acoustic features can effectively distinguish fake audio from real
audio [12, 13, 14, 15]. Concurrently, there are also studies concentrated on
designing effective classification models to distinguish between real and fake
audios [16, 17, 18, 19, 20, 21, 22, 14, 15].

Unlike the earlier approach, our core idea is to improve fake audio detection
by leveraging external knowledge. MOS (Mean Opinion Score) is used to assess
the quality of synthesized speech, so MOS will be a promising tool for
detecting fake audio. Moreover, compared to the binary classification of fake
audio detection, MOS offers a finer-grained evaluation criterion that can
yield additional information. In this paper, we adopt the self-supervised
learning (SSL) models [23], widely studied and performed well in various
speech classification tasks [24], as the foundational architecture for
constructing the FAD classifier. To ensure the robustness of our final results
and reduce dependence on the performance of a single SSL model, we conduct
experiments with various SSL models. Building upon this SSL-based
architecture, we explore various methods for integrating MOS scores into FAD
tasks. This investigation ultimately leads to developing our innovative MOS-
FAD framework, where MOS scores are used as gating mechanisms in model
construction. This framework has achieved state-of-the-art performance.

We also extend the use of MOS scores to guide training data selection. This
approach addresses two key challenges: Firstly, it alleviates the long-tail
issue associated with imbalanced training data. Secondly, by utilizing MOS
scores for data selection, the model can be trained on samples that pose a
greater challenge in distinguishing between real and fake audio, ultimately
enhancing the model’s classification capabilities.

Fig. 1: Proposed model structure.

##  2 Related work

###  2.1 Automatic MOS Predictor

The Mean Opinion Score (MOS) is a widely used subjective quality evaluation
criterion for synthesized speech. In MOS assessments, listeners rate speech
sample quality on a scale ranging from 1 (poor) to 5 (excellent). The average
of these ratings is used to gauge overall speech quality. However, traditional
MOS evaluations can be expensive and time-consuming.

Given these challenges, there has been a growing interest in training
automatic MOS predictors. Notably, the VoiceMOS Challenge [26] has advanced
this field recently.

The state-of-the-art automatic MOS predictors employ neural network-based
frameworks and use large-scale synthetic speech data for training a model to
predict MOS score [27, 28, 29, 30, 31, 32]. Another solution is to introduce
self-supervised learning (SSL) models [22, 33, 34, 33, 35], which have already
been demonstrated in the MOS prediction task. Since the choice of data and
various model configurations used for pre-training will affect the
performance, fusing diverse models with different architectures or training
strategies [32, 31] could help achieve better results on the MOS prediction
task.

###  2.2 Fake Audio Detection

The current FAD research focuses on acoustic front-end and classification
models.

Research has shown that well-designed acoustic features can effectively
distinguish fake audio from real audio. The traditional Mel frequency spectrum
coefficients (MFCC) were improved by using constant Q-transform instead of the
short-time Fourier transform [12]. The linear frequency spectrum coefficients
(LFCC) [13] was proposed to replace the Mel-scale filters with linear filters,
making it more focused on high-frequency band features. Wav2vec pre-trained
models [14, 15] were used as the feature extractors, which obtain more robust
acoustic features.

An effective classification model can also help distinguish between real and
fake audio. With the development of deep neural networks, the classifier has
been upgraded from convolutional neural networks (CNN) [16, 17], light
convolution neural network (LCNN) [18], residual network (ResNet) [19],
differentiable architecture search (DARTS) [20, 21] to state-of-the-art
Transformer-based models, e.g., Wav2vec2.0 [22, 14, 15].

##  3 Proposed Approach

As illustrated in Fig. 1. (a), our proposed method consists of three
components: SSL-FADs, Fused SSL-MOS sub-system, and MOS-FAD Fusion. In this
section, we introduce each of them.

###  3.1 SSL-FAD

SSL-FADs are SSL-based models that predict FAD scores. This work adopts the
idea in [30], which adds a mean pooling layer and a fully connected layer
after the feature extractor of an SSL model to construct an SSL-FAD model. In
this paper, we adopt $7$ different SSL models (Wav2Vec 2.0 Base, Wav2Vec 2.0
Large, Wav2Vec 2.0 (LV-60), HuBERT Base, WavLM Base, WavLM Base+ and WavLM
Large) to construct $7$ SSL-FADs, as shown in Table 2. By $n$ SSL-FADs, FAD
scores $\mathcal{Y}=\\{y_{i}\mid y_{i}\in[0,1],\,i=1,2,\ldots,n\\}$ are
computed.

###  3.2 Fused SSL-MOS

The fused SSL-MOS is an automatic MOS predictor proposed in [31], which
consists of several SSL-MOS models and an MOS fusion. Like SSL-FAD, SSL-MOS is
constructed by adding a mean pooling layer and a fully connected layer after
the feature extractor of the SSL model. The SSL models employed in
constructing SSL-MOSs are identical to those used in constructing SSL-FAD. The
MOS model fuser is a $2$-layer neural network consisting of a fully connected
layer without a bias for capturing the weighted information, and a linear
function for obtaining the residual information between the ground truth and
the predicted scores after a fully connected layer. The MOS fusion takes the
MOS scores $\mathcal{Z}=\\{z_{i}\mid z_{i}\in[0,5],\,i=1,2,\ldots,n\\}$
predicted by $n$ SSL-MOS models as inputs and outputs the final MOS score
$z_{f}\in[0,5]$.

###  3.3 MOS-FAD Fusion

The MOS-FAD fusion performs the fusion of FAD scores predicted by the SSL-FADs
and the MOS scores predicted by the Fused SSL-MOS. It outputs the FAD task’s
prediction $y_{f}\in[0,1]$.

We conducted experiments using both Multi-Layer Perceptron (MLP) and LightGBM
[36] as potential candidates for MOS-FAD. Through a comparative analysis of
their performance, we concluded that MOS can be effectively employed as a
gating mechanism within the fusion model. Building upon this insight, we
introduced Gated MLP, as depicted in Fig.1(b.1). Moreover, to further leverage
the prior information from MOS scores, the Fusion with thresholding is
proposed, as shown in Fig.1(b.2).

a. MLP: In our experiment, the MLP is a two-layer neural network with one
sigmoid activation function and three hidden neurons. The linear
transformation in the first layer is performed without a bias.

b. LightGBM: Light Gradient Boosting Machine (LightGBM) [36] is an ensemble
model that trains a series of decision trees sequentially in a leaf-wise
fashion and combines them. During sequential training, the decision trees use
the error from the previous tree to adjust their learning and eventually
minimize the loss function. Therefore, weak learners of the decision trees can
be combined as a high-performance model.

c. Gated MLP: Instead of feeding all the predicted scores (FAD scores and MOS
scores) to the MLP, we design a special gate layer for the predicted MOS
score. The gates control how much information from the FAD scores should be
fed to the later MLP. The first step is to decode the MOS scores into a tensor
of the same length as the FAD scores using a Linear Decoder layer. Then, the
values in this tensor are normalized to the range of 0 to 1 using a $Sigmoid$
activation function. These outputs are treated as gate $g$. Then, predicted
FAD scores $y_{1}$, $y_{2}$ … are pointwisely multiplied by $g$, and the
results are used as inputs to the MLP. The MLP component in Gated MLP remains
consistent with the description provided in 3.3.a.

d. Fusion with Thresholding: Building upon the definition of MOS scores, we
introduce a straightforward screening mechanism named as ‘fusion with
thresholding’. The underlying concept is to categorize samples with
excessively low MOS scores as fake audio and those with excessively high MOS
scores as genuine audio. We assign a score of $0$ to samples with MOS scores
below the threshold $M_{1}$, designating them as fake speech, and assign a
score of $1$ to samples with MOS scores above the threshold $M_{2}$ as real
speech. FAD scores are determined for the remaining samples based on the
outcomes of specific fusion models. It is worth highlighting that this fusion
with thresholding approach applies to any FAD model. In our experiments, all
fake speech of training and validation datasets received MOS scores below 4.0,
while 98.81% of real speech received MOS scores above 2.5. Consequently, we
set the threshold values as $M_{1}=2.5$ and $M_{2}=4.0$.

Fig. 2: Training process of our proposed model.

##  4 Experiment

###  4.1 Datasets

a. VoiceMOS dataset: We used the VoiceMOS Challenge 2022 [26] main track
dataset for training automatic MOS predictor. The main track data contains
only English and consists of $4,974$ samples for training, $1,066$ samples for
validation, and $1,066$ examples for testing. We quantized the main track’s
MOS scores ranging from $1$ to $5$ into $33$ intervals with a step of $0.125$.

b. ASVspoof dataset: ASVspoof datasets were leveraged for the FAD task. First,
we trained our proposed framework by using the training and validation
datasets from the ASVspoof2019 [9] LA dataset.

Next, we employed the ASVSpoof2021 challenge [8] evaluation set of the DF
track to evaluate the effectiveness of our approach. This DF evaluation set
comprises a total of $611,829$ samples. It shows audio coding and compression
artifacts, with approximately $600$k of audio processing with various
commercial audio codecs.

###  4.2 Experimental Settings

All neural network models were trained using a fixed learning rate of $0.001$,
Stochastic Gradient Descent as the optimizer, and $CrossEntropy$ as the loss
function. Training would stop if there were no decrease in the validation loss
for 20 consecutive epochs. During the training of LightGBM, the following
parameter configuration was employed: {‘objective’: ‘binary’, ‘metric’: ‘auc’,
‘num_leaves’: $16$, ‘max_bin’: $25$, ‘max_depth’: $4$, ‘learning_rate’:
$0.1$}.

###  4.3 Training Process

Fig. 2 shows the training details of our proposed approach. The training
process can be divided into two independent modules, the Automatic MOS
Predictor Training Process and the MOS-FAD Training Process, and an additional
data processing module, the MOS-based data-filter, for the FAD task training
dataset.

a. Automatic MOS Predictor Training Process: As depicted in the green block in
Fig. 2, we employed the MOS fusion model, as described in Section 3.2, to
generate MOS scores. This model undergoes training using the VoiceMos
Challenge 2022 Main Track dataset. It is important to note that this training
process is conducted separately from the MOS-FAD training process, and the
training specifics for SSL-MOS closely adhere to those detailed in [31].

Fig. 3: The MOS score distributions in ASVspoof2019 LA track. Table 1: The number of samples of ASVspoof2019 LA Track with or without the data-filter. With the data-filter we select data by choosing data with MOS scores ranging between $3.0$ and $4.0$.  | Training Dataset | Validation Dataset
---|---|---
| Total | Real | Fake | Total | Real | Fake
w/o filter | 25,380 | 2,580 | 22,800 | 22,438 | 2,548 | 22,438
w/ filter | 5,034 | 2,568 | 2,466 | 5,198 | 2,533 | 2665

b. MOS-based Data-Filter: The MOS-based data-filter serves as a selector for
training FAD task models, with two primary objectives. Firstly, it aims to
alleviate the issue of sample imbalance, as evident in Table 1, where the
original training set exhibits a significant imbalance, with nearly ten times
more fake speeches than real ones. Secondly, it enhances model training
effectiveness, as illustrated in Fig. 3. The distribution of MOS scores in the
LA track dataset suggests that samples with MOS scores below $2.5$ can be
confidently categorized as fake, while those with scores exceeding $4.0$ can
be considered as real. Consequently, regarding its authenticity, the FAD model
only requires training on data that falls within the range of uncertainty
(overlap part in the MOS distribution graph).

In summary, to achieve a balanced sample distribution and enhance training
effectiveness, we opted for samples from the ASVspoof2019 LA dataset with MOS
scores between 3.0 and 4.0 as the new training and validation datasets for the
FAD task. As depicted in Table 1, this filtering approach results in an
approximate balance between real and fake speeches.

c. MOS-FAD Training Process: We used the filtered dataset to finetune SSL-FADs
and train their fusions (FAD Fusion and MOS-FAD Fusion) for FAD tasks (as
shown in Fig. 2). The MOS-FAD Fusion in Fig. 2 represents a model fusion
approach that combines MOS and FAD scores. Concurrently, The FAD Fusion in
Fig. 2 serves as a control group, enabling us to evaluate the importance of
incorporating MOS scores into the fusion model.

###  4.4 Inference

To assess the authenticity of samples in the ASVspoof2021 DF evaluation set,
we first employ the Fused SSL-MOS system to generate its MOS scores.
Concurrently, we use individual SSL-FAD systems to derive the corresponding
FAD scores. Subsequently, the MOS-FAD Fusion process combines these FAD and
MOS scores to yield a unified score for each sample. Notably, a lower fused
score indicates a higher likelihood that the sample is fake.

Table 2: EER on 7 FAD scores of ASVSpoof2021 DF Eval Track by each SSL-FAD which trained with or without MOS Data-Filter.  | DF Eval
---|---
SSL Base | w/ filter | w/o filter
W2V Base | 0.2514 | 0.5510
HuBERT Base | 0.2353 | 0.4449
W2V large | 0.1891 | 0.4794
W2V Large(LV-6.0) | 0.2395 | 0.3776
WavLM Base | 0.2167 | 0.5126
WavLM Base+ | 0.2088 | 0.5310
WavLM Large | 0.2453 | 0.5881

##  5 Results and Analysis

###  5.1 Effectiveness of MOS-based Data-Filter

We conducted experiments by training SSL-FADs directly using the original
training dataset and compared their performance with models trained on the
filtered dataset, as displayed in Table 2. The results demonstrate that models
trained after data filtering outperform those trained on the unfiltered
dataset, indicating the effectiveness of filtered data in training.

###  5.2 Effect of MOS in Fusion

Table 3 shows the performance of different fusion models. Among them, the
‘Score Fusion’ investigates how to combine $7$ different FAD scores and $1$ of
MOS scores, while the ‘Embedding Fusion’ validates the effectiveness of ‘Score
Fusion’.

a. Score Fusion: Only using FAD scores, both FAD Fusion methods fail to
outperform the Wav2Vec Large-based SSL-FAD model. This could be attributed to
the high correlations among FAD scores, which indicate multicollinearity,
making it challenging to improve performance through score fusion.

In contrast, within the MOS-FAD Fusions that incorporate MOS scores, LightGBM
outperforms MLP and any SSL-FADs. Considering LightGBM’s tree-like structure,
we reason that MOS plays a gating role in the fusion process. Building on this
hypothesis, we introduce the Gated MLP, which outperforms LightGBM, providing
evidence of MOS as a gating mechanism.

For further improvement, we introduced a more rigorous gate filtering
mechanism named ‘fusion with thresholding,’ drawing upon prior MOS knowledge.
Among MOS-FAD Fusions, the adoption of fusion with thresholding yields
performance improvements. Notably, both LightGBM with thresholding and Gated
MLP with thresholding surpass the previous SOTA performance. Gated MLP with
thresholding demonstrates an impressive $13.6\%$ reduction in EER.

b. Embedding Fusion: We substituted the MOS and FAD scores with embeddings
generated by the SSL model, except for the MOS $z_{f}$ in ‘fusion with
thresholding.’ In the MLP approach of FAD Fusion, all embeddings from SSL-FADs
are concatenated into a 6,144-dimension tensor, which is then input into the
MLP. In the MLP approach of MOS-FAD Fusion, embeddings from both SSL-FAD and
SSL-MOS are concatenated to create a 12,288-dimension tensor as input for the
MLP. In the case of Gated MLP, the concatenated 6,144-dimension embeddings
generated by SSL-MOS undergo a linear layer followed by a Sigmoid activation
layer to produce gate units of 6,144 dimensions. These gate units control how
much information from the concatenated 6,144-dimensional embeddings produced
by SSL-FADs is passed into the MLP. Notably, the number of hidden units in the
MLP described in this section is half of the input dimension.

The results show that the embeddings-based MLP without MOS data failed to
surpass the performance of that with MOS data and the best SSL-FAD model,
highlighting the necessity of incorporating MOS information. Furthermore,
embeddings-based fusions with thresholding outperform their respective
baseline models, further validating MOS’s appropriateness as the gating
mechanism in fusion.

Finally, when the significance level $p$ is set to $0.01$, the best
performance of ‘Score Fusion’ significantly outperforms that of ‘Embeddings
Fusion.’ A possible reason is that the gate logic in score fusion serves as a
more powerful mechanism for combinations than the combinations learned through
the model in embedding fusion. This confirms the effectiveness of ‘Score
Fusion.’

Table 3: EER of each fusion on ASVSpoof2021 DF Eval Track. Scores stand for the fusion of FAD and MOS scores. Emb stands for the fusion of FAD and MOS embeddings. ${\dagger}$ is significantly better than the former SOTA when significance level $p$ is set at 0.01.  |  | DF Eval
---|---|---
Model Type | Model | Score | Emb
FAD Fusion | MLP | 0.2110 | 0.2049
LightGBM | 0.2221 | \
MOS-FAD Fusion | MLP | 0.3160 | 0.1763
LightGBM | 0.1732 | \
Gated MLP | 0.1661 | 0.1890
MLP + threshold | 0.2921 |  0.1509†
LightGBM + threshold |  0.1476† | \
Gated MLP + threshold | 0.1351† |  0.1509†
Former SOTA in ASVspoof 2021 (T23 in [8]) | 0.1564

##  6 Conclusion

In this paper, we have proposed MOS-FAD, a robust approach for FAD via
automatic MOS predictor. We used the MOS-based data-filter to resample the
training and validation dataset, which led to effective training for SSL-FADs.
Additionally, our results have shown that utilizing MOS as a gating mechanism
in the fusion with FAD scores improves performance, based on which we proposed
a novel fusion model, Gated MLP, with thresholding. This MOS-FAD fusion
method, significantly decreases the EER by $13.6\%$, compared with the best
system reported so far. Given that the misuse of fake audio generation can
potentially negatively impact society, we hope this study will encourage
further research into this rapidly evolving and crucial field and help
circumvent any problems.

## References

  * [1] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, and et al.,  “Tacotron: Towards end-to-end speech synthesis,”  in Proc. INTERSPEECH, 2017.
  * [2] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu,  “Fastspeech: Fast, robust and controllable text to speech,”  in Proc. NeurIPS, 2019.
  * [3] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T. Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,”  in Proc. ICLR, 2020.
  * [4] C. Yu, H. Lu, N. Hu, M. Yu, C. Weng, K. Xu, P. Liu, D. Tuo, S. Kang, G. Lei, and et al.,  “DurIAN: Duration informed attention network for multimodal synthesis,”  in Proc. INTERSPEECH, 2020.
  * [5] B. Sisman, J. Yamagishi, S. King, and H. Li,  “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
  * [6] J. Yi, Y. Bai, J. Tao, H. Ma, Z. Tian, C. Wang, T. Wang, and R. Fu,  “Half-Truth: A Partially Fake Audio Detection Dataset,”  in Proc. INTERSPEECH, 2021, pp. 1654–1658.
  * [7] H. Ma, J. Yi, J. Tao, Y. Bai, Z. Tian, and C. Wang,  “Continual Learning for Fake Audio Detection,”  in Proc. INTERSPEECH, 2021, pp. 886–890.
  * [8] J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. Lee, T. Kinnunen, N. Evans, and et al.,  “Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,”  in Proc. ASVspoof Workshop, 2021.
  * [9] M. Todisco, X. Wang, V. Vestman, M. Sahidullah, H. Delgado, A. Nautsch, J. Yamagishi, N. Evans, T. Kinnunen, and K. Lee,  “ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection,”  in Proc. INTERSPEECH, 2019, pp. 1008–1012.
  * [10] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K. Lee,  “The ASVspoof 2017 Challenge: Assessing the Limits of Replay Spoofing Attack Detection,”  in Proc. INTERSPEECH, 2017, pp. 2–6.
  * [11] J. Yi, R. Fu, J. Tao, S. Nie, H. Ma, C. Wang, T. Wang, Z. Tian, Y. Bai, C. Fan, and et al.,  “Add 2022: the first audio deep synthesis detection challenge,”  in Proc. IEEE-ICASSP, 2022, pp. 9216–9220.
  * [12] M. Todisco, H. Delgado, and N. Evans,  “Constant q cepstral coefficients: A spoofing countermeasure for automatic speaker verification,”  Computer Speech & Language, vol. 45, pp. 516–535, 2017.
  * [13] M. Sahidullah, T. Kinnunen, and C. Hanilçi,  “A comparison of features for synthetic speech detection,”  in Proc. INTERSPEECH, 2015, pp. 2087–2091.
  * [14] Y. Xie, Z. Zhang, and Y. Yang,  “Siamese network with wav2vec feature for spoofing speech detection,”  in Proc. INTERSPEECH, 2021, pp. 4269–4273.
  * [15] J. Martín-Doñas and A. Álvarez,  “The vicomtech audio deepfake detection system based on wav2vec2 for the 2022 add challenge,”  in Proc. IEEE-ICASSP, 2022, pp. 9241–9245.
  * [16] C. Lai, N. Chen, J. Villalba, and N. Dehak,  “ASSERT: Anti-Spoofing with Squeeze-Excitation and Residual Networks,”  in Proc. INTERSPEECH, 2019, pp. 1013–1017.
  * [17] M. Alzantot, Z. Wang, and M. Srivastava,  “Deep Residual Neural Networks for Audio Spoofing Detection,”  in Proc. INTERSPEECH, 2019, pp. 1078–1082.
  * [18] G. Lavrentyeva, S. Novoselov, E. Malykh, A. Kozlov, O. Kudashev, and V. Shchemelinin,  “Audio replay attack detection with deep learning frameworks.,”  in Proc. INTERSPEECH, 2017, pp. 82–86.
  * [19] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in Proc. CVPR, 2016, pp. 770–778.
  * [20] H. Liu, K. Simonyan, and Y. Yang,  “DARTS: differentiable architecture search,”  in Proc. ICLR, 2019.
  * [21] W. Ge, M. Panariello, J. Patino, M. Todisco, and N. Evans,  “Partially-connected differentiable architecture search for deepfake and spoofing detection,”  in Proc. INTERSPEECH, 2021.
  * [22] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Proc. NeurIPS, 2020, vol. 33, pp. 12449–12460.
  * [23] A. Oord, Y. Li, and O. Vinyals,  “Representation learning with contrastive predictive coding,”  in Proc. NeurIPS, 2018.
  * [24] S. Yang and et al.,  “SUPERB: Speech processing universal performance benchmark,”  in Proc. INTERSPEECH, 2021, pp. 1194–1198.
  * [25] R. Huang and Y. Li,  “MOS: towards scaling out-of-distribution detection for large semantic space,”  in Proc. CVPR, 2021, pp. 8710–8719.
  * [26] W. Huang, E. Cooper, Y. Tsao, H. Wang, T. Toda, and J. Yamagishi,  “The voicemos challenge 2022,”  in Proc. INTERSPEECH, 2022.
  * [27] C. Lo, S. Fu, W. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H. Wang,  “Mosnet: Deep learning based objective assessment for voice conversion,”  in Proc. INTERSPEECH, 2019.
  * [28] Y. Choi, Y. Jung, and H. Kim,  “Neural mos prediction for synthesized speech using multi-task learning with spoofing detection and spoofing type classification,”  in Proc. IEEE-SLT, 2021, pp. 462–469.
  * [29] W. Huang, E. Cooper, J. Yamagishi, and T. Toda,  “LDNet: Unified listener dependent modeling in mos prediction for synthetic speech,”  in Proc. IEEE-ICASSP, 2022.
  * [30] E. Cooper, W. Huang, T. Toda, and J. Yamagishi,  “Generalization ability of mos prediction networks,”  arXiv preprint arXiv:2110.02635, 2021.
  * [31] Z. Yang, W. Zhou, C. Chu, S. Li, R. Dabre, R. Rubino, and Y. Zhao,  “Fusion of self-supervised learned models for mos prediction,”  in Proc. INTERSPEECH, 2022.
  * [32] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari,  “Utmos: Utokyo-sarulab system for voicemos challenge 2022,”  in Proc. INTERSPEECH, 2022.
  * [33] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, and et al.,  “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,”  IEEE Journal of Selected Topics in Signal Processing, vol. 16, pp. 1505–1518, 2021.
  * [34] W. Hsu, B. Bolte, Y. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [35] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli,  “Data2vec: A general framework for self-supervised learning in speech, vision and language,”  in Proc. ICML, 2022.
  * [36] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T. Liu,  “Lightgbm: A highly efficient gradient boosting d’e’c’i’s’i’o’n tree,”  in Proc. NeurIPS, 2017, vol. 30.


# Silhouette-based View-embeddings for Gait Recognition under Multiple Views

###### Abstract

Gait recognition under multiple views is an important computer vision and
pattern recognition task. In the emerging convolutional neural network based
approaches, the information of view angle is ignored to some extent. Instead
of direct view estimation and training view-specific recognition models, we
propose a compatible framework that can embed view information into existing
architectures of gait recognition. The embedding is simply achieved by a
selective projection layer. Experimental results on two large public datasets
show that the proposed framework is very effective.

Index Terms—  Gait recognition, silhouette-based, view-embedding, cross-view,
multi-task

##  1 Introduction

Gait is a biometric presenting the walking style of people and has an edge
over other biometrics such as face, fingerprint because it can be recognized
at a distance with much less cooperation. Recently, due to the growing demand
of intelligent surveillance, gait recognition attracted more attentions.

Variations like carrying conditions, coat-wearing and viewpoint differences
may cause changes in gait appearance and bring significant challenges to gait
recognition. Solving these problems is of great significance to improve the
performance of gait recognition. Among these problems, viewpoint differences
is a very tricky problem, because it may bring greater visual differences than
the identity.

Over the years many methods have been proposed to solve the problem of
multiple view. Gait recognition methods include model-based [21, 22] and
appearance-based methods. Appearance-based methods can be divided into two
categories, i.e., regarding gait as a single image [1, 2, 3, 9, 13, 20, 23,
24] and regarding gait as a video or an image sequence [4, 5, 7, 6, 10]. The
first category often use gait energy images (GEIs) [8] as the representation
of gait, while the second category directly makes use of the gait silhouette
images. Between these two categories, silhouettes based methods show better
performance and become a main trend.

Although the aforementioned methods work well on representing gait in a multi-
view scenario, and the deep neural network can somehow learn view-robust
feature from mixed views, view itself, i.e. the explicit view estimation and
view-specific modeling, is overlook and underrated. We argue that explicit
embedding view information can effectively improve the performance of existing
approaches.

In this paper, we proposed a general framework for multi-view gait recognition
by explicit view angle embedding, based on which, two state-of-the-art gait
recognition backbones, i.e.Gaitset [4] and GaitGL [14] are enhanced. Compared
with the original ones, the enhanced ones, improve the performance. The
effectiveness is well demonstrated by the experiments on CASIA-B [11] and
OUMVLP [12] datasets.

The rest of this paper is organized as follows. In Section 2 we will briefly
describe relevant studies. The details of proposed methods are introduced in
Section 3, while the experimental validation is described om Section 4. And
the conclusion is given in Section 5.

Fig. 1: Pipeline of our method. The backbone can be replaced by any
silhouette-based network, such as Gaitset [4], Gaitpart [5], MT3D[6] and
GaitGL [14]. For the extracted feature map, we feed it into two branches. The
first branch performs horizontal pyramid pooling (HPP) [4] on the feature map.
In the second branch, the feature map is pooled to get the view classification
features, and the projection matrix is selected according to the predicted
view. Then for each feature calculated in HPP at an unique position, we
multiply the corresponding projection matrix of the corresponding view to get
the final view-robust feature.

##  2 Related work

Gait Recognition. With the emergence of deep learning, convolution neural
networks (CNNs) have been adopted to gait recognition and gain a great
success. Shiraga et al. [1] and Wu et al. [3] both used CNN to extract feature
from GEIs. However, the simple averaging operation of GEI results in a serious
detail loss and limits the learning effects of CNN. To this end, Wolf et al.
[7] extracted feature directly from original silhouettes by three dimensional
convolutions. Chao et al. [4] treated gait silhouette sequence as a set and
proposed a novel model named Gaitset. Fan et al. [5] further improved
silhouette gait feature by introducing part-based representations. Lin et al.
[6, 14] proposed a comprehensive model named GaitGL by integrating both global
and local feature. It is recognized as the state-of-the-art gait recognition
method.

Multi-task Learning. Although success, all the aforementioned approaches
overlook the view information which has a great influence on the performance.
Multi-task learning, which has been proven effective in many computer vision
tasks [17, 16, 18], could be a solution to learning view-specific gait
recognition models. This idea is validated by He et al. [9], however, their
approach is still based on energy images. To our knowledge, there is no
silhouette-based view-specific or aware deep model has been proposed for gait
recognition. Besides, multi-task learning is also adopted to gait for
recognizing attributes like age and gender [19, 15].

##  3 Proposed Method

What we propose is not a specific model, but a general and compatible
framework for multi-view gait recognition. As shown in Figure 1, the input is
a gait silhouettes sequence $X_{in}\in{\mathbb{R}^{T\times H\times W}}$ and a
backbone model $E$ is used to extract feature map
$X_{f}\in{\mathbb{R}^{C_{f}\times H_{f}\times W_{f}}}$. The backbone can be
any silhouettes-based network, such as Gaitset [4], Gaitpart [5], MT3D [16]
and GaitGL [14].

Followed by the backbone, the feature map will be fed into two branches. The
first one performs Horizontal Pyramid Pooling (HPP)[4] on the feature map and
the Horizontal Pyramid Mapping (HPM) $f_{HPM}\in{\mathbb{R}^{n\times D}}$ will
be obtained where $D$ is the dimension of output feature. In the second
branch, the feature map is pooled to get the view classification feature
$f_{v}\in{\mathbb{R}^{D_{v}}}$, and the projection matrices
$\\{W_{1},W_{2},W_{3},...,W_{n}\\}(W_{i}\in{\mathbb{R}^{D\times D}})$ are
selected according to the predicted view, where $n$ is the number of strips
cut in the HPP Module [4]. Then for each feature in HPM, we will multiply the
projection matrix of the corresponding view to get the final view-invariant
feature.

###  3.1 View projection matrix selection

The feature map $X_{f}$ is calculated from the input $X_{in}$ using the
backbone $E$, and then we use it to compute the view classification feature
$f_{v}$. The process can be expressed as:

$X_{f}=E(X_{in})~{}~{}and~{}~{}f_{v}=F(P_{Global\\_Avg}(X_{f})).$ (1)

Especially for Gaitset, another feature map $X_{g}$ that denotes global
feature is calculated from the extractor, then its view classification feature
can be defined as:

$\displaystyle f_{v}$ $\displaystyle=F(P_{Global\\_Avg}([X_{f};X_{g}])),$ (2)

where $F$ denotes a fully connect layer and $P_{Global\\_Avg}$ denotes global
average pooling.

The predicted view probability $\hat{p}\in{\mathbb{R}^{M}}$ of the input gait
silhouettes and the view of maximum probability $\hat{y}$ are calculated as:

$\hat{p}=W_{view}f_{v}+B_{view}~{}~{}and~{}~{}\hat{y}=\mathop{\arg\max}_{i}\hat{p_{i}},\vspace{-0.2cm}$
(3)

where $M$ is the number of discrete views, $W_{view}\in\mathbb{R}^{M\times
D_{v}}$ are weight matrices, $B_{view}$ are the bias terms and
$\hat{y}\in\\{0,1,2...M\\}$.

For predicted view $\hat{y}$, a corresponding view projection matrix group
$Z_{\hat{y}}|=\\{W_{i}|i=1,2,...,n\\}$ will be trained where
$W_{i}\in\mathbb{R}^{D\times D}$ is the projection matrix. And all the view
projection matrix can be expressed as $S=\\{Z_{i}|i=1,2,...,M\\}$.

###  3.2 HPP feature projection

For the convenience for explanation, the horizontal pyramid mapping (HPM)
$f_{HPM}\in{\mathbb{R}^{n\times D}}$ is expressed as
${f_{HPM,i}},i=1,2,3,...,n$, where ${f_{HPM,i}}\in{\mathbb{R}^{D}}$. Suppose
that the view $\hat{y}$ of the input gait silhouettes is predicted to be
$\theta$ in Equation (3), then the projected features can be expressed as:

$\displaystyle f_{final,i}$ $\displaystyle=W_{i}{f_{HPM,i}}$ (4)
$\displaystyle f_{final}$
$\displaystyle=[f_{final,1},f_{final,2}...,f_{final,n}],$

where $i=1,2,...n$, $W_{i}\in{Z_{\theta}}$ and the $f_{final}$ is used as the
representation for calculating the similarity between two input gait
silhouette sequences.

Table 1: Rank-1 accuracy(%)on CASIA-B under 11 probe views excluding identical-view cases. Gallery NM#1-4 | 0°-180°
---|---
Probe | 0° | 18° | 36° | 54° | 72° | 90° | 108° | 128° | 144° | 162° | 180° | mean
ST(24) | NM#5-6 |  ViDP[2] | - | - | - | 59.1 | - | 50.2 | - | 57.5 | - | - | - | -
CNN-LB[3] | 54.8 | - | - | 77.8 | - | 64.9 | - | 76.1 | - | - | - | -
GaitSet[4] | 64.6 | 83.3 | 90.4 | 86.5 | 80.2 | 75.5 | 80.3 | 86.0 | 87.1 | 81.4 | 59.6 | 79.5
Vi-GaitSet | 67.8 | 84.3 | 90.7 | 88.4 | 81.3 | 76.9 | 82.2 | 87.2 | 89.3 | 84.0 | 65.8 | 81.6
GaitGL[14] | 72.1 | 83.4 | 88.6 | 88.6 | 80.9 | 74.4 | 82.3 | 88.3 | 89.5 | 86.5 | 69.9 | 82.2
Vi-GaitGL | 70.7 | 83.6 | 89.0 | 89.1 | 78.5 | 71.8 | 79.6 | 86.1 | 88.8 | 84.7 | 66.5 | 80.7
BG#1-2 |  GaitSet[4] | 55.8 | 70.5 | 76.9 | 75.5 | 69.7 | 63.4 | 68.0 | 75.8 | 76.2 | 70.7 | 52.5 | 68.6
Vi-GaitSet | 61.8 | 74.2 | 78.9 | 77.9 | 72.7 | 67.1 | 71.1 | 78.6 | 78.4 | 71.4 | 58.8 | 71.9
GaitGL[14] | 64.2 | 73.8 | 79.3 | 80.8 | 71.3 | 65.3 | 72.3 | 79.2 | 82.5 | 79.7 | 60.3 | 73.5
Vi-GaitGL | 64.2 | 75.0 | 82.6 | 81.5 | 70.2 | 63.9 | 70.4 | 77.8 | 81.0 | 77.6 | 58.3 | 72.9
CL#1-2 |  GaitSet[4] | 29.4 | 43.1 | 48.5 | 48.7 | 42.3 | 40.3 | 44.9 | 47.4 | 43.0 | 35.7 | 25.6 | 40.9
Vi-GaitSet | 33.8 | 46.4 | 51.8 | 47.5 | 46.8 | 41.3 | 44.7 | 48.5 | 44.7 | 36.7 | 27.5 | 42.7
GaitGL[14] | 45.8 | 59.1 | 62.7 | 62.5 | 56.7 | 51.5 | 57.6 | 60.5 | 58.0 | 54.4 | 37.9 | 55.1
Vi-GaitGL | 50.8 | 64.3 | 68.6 | 67.1 | 60.4 | 54.2 | 59.6 | 63.9 | 62.9 | 59.9 | 41.5 | 59.4
MT(62) | NM#5-6 |  MGAN[9] | 54.9 | 65.9 | 72.1 | 74.8 | 71.1 | 65.7 | 70.0 | 75.6 | 76.2 | 68.6 | 53.8 | 68.1
GaitSet[4] | 86.8 | 95.2 | 98.0 | 94.5 | 91.5 | 89.1 | 91.1 | 95.0 | 97.4 | 93.7 | 80.2 | 92.0
Vi-GaitSet | 87.9 | 95.8 | 98.5 | 97.1 | 92.1 | 89.3 | 92.5 | 97.3 | 97.3 | 95.4 | 82.8 | 93.3
GaitGL[14] | 90.8 | 95.3 | 97.9 | 96.0 | 94.0 | 91.4 | 94.7 | 97.1 | 97.8 | 95.6 | 88.0 | 94.4
Vi-GaitGL | 90.8 | 95.9 | 97.7 | 95.9 | 93.3 | 91.5 | 94.4 | 97.3 | 97.3 | 95.4 | 86.9 | 94.2
BG#1-2 |  MGAN[9] | 48.5 | 58.5 | 59.7 | 58.0 | 53.7 | 49.8 | 54.0 | 51.3 | 59.5 | 55.9 | 43.1 | 54.7
GaitSet[4] | 79.9 | 89.8 | 91.2 | 86.7 | 81.6 | 76.7 | 81.0 | 88.2 | 90.3 | 88.5 | 73.0 | 84.3
Vi-GaitSet | 80.8 | 88.3 | 92.8 | 91.3 | 84.8 | 79.2 | 85.3 | 89.8 | 92.1 | 89.9 | 76.4 | 86.4
GaitGL[14] | 85.6 | 93.2 | 95.0 | 92.4 | 89.0 | 81.5 | 86.8 | 92.7 | 95.6 | 92.9 | 83.1 | 89.8
Vi-GaitGL | 83.6 | 92.9 | 94.7 | 93.1 | 89.4 | 83.6 | 88.6 | 93.6 | 96.1 | 93.3 | 81.5 | 90.0
CL#1-2 |  MGAN[9] | 23.1 | 34.5 | 36.3 | 33.3 | 32.9 | 32.7 | 34.2 | 37.6 | 33.7 | 26.7 | 21.0 | 31.5
GaitSet[4] | 52.0 | 66.0 | 72.8 | 69.3 | 63.1 | 61.2 | 63.5 | 66.5 | 67.5 | 60.0 | 45.9 | 62.5
Vi-GaitSet | 60.7 | 70.4 | 75.8 | 69.7 | 65.3 | 61.3 | 67.1 | 70.9 | 71.9 | 65.1 | 49.8 | 66.2
GaitGL[14] | 70.2 | 83.6 | 87.3 | 85.2 | 78.5 | 73.1 | 80.0 | 85.1 | 84.6 | 76.9 | 61.7 | 78.7
Vi-GaitGL | 71.2 | 86.5 | 90.9 | 89.0 | 83.9 | 77.2 | 84.8 | 89.1 | 88.6 | 81.0 | 63.7 | 82.3
LT(74) | NM#5-6 |  CNN-Ensemble[3] | 88.7 | 95.1 | 98.2 | 96.4 | 94.1 | 91.5 | 93.9 | 97.5 | 98.4 | 95.8 | 85.6 | 94.1
GaitSet[4] | 90.8 | 97.9 | 99.4 | 96.9 | 93.6 | 91.7 | 95.0 | 97.8 | 98.9 | 96.8 | 85.8 | 95.0
Vi-GaitSet | 93.1 | 98.0 | 99.1 | 97.1 | 93.8 | 92.7 | 95.8 | 97.7 | 99.0 | 97.7 | 87.1 | 95.6
GaitPart[5] | 94.1 | 98.6 | 99.3 | 98.5 | 94.0 | 92.3 | 95.9 | 98.4 | 99.2 | 97.8 | 90.4 | 96.2
GaitGL[14] | 94.6 | 97.3 | 98.8 | 97.1 | 95.8 | 94.3 | 96.4 | 98.5 | 98.6 | 98.2 | 90.8 | 96.4
Vi-GaitGL | 93.7 | 96.9 | 98.6 | 97.4 | 95.5 | 93.9 | 97.3 | 98.6 | 98.6 | 97.7 | 89.7 | 96.2
BG#1-2 |  CNN-LB[3] | 64.2 | 80.6 | 82.7 | 76.9 | 64.8 | 63.1 | 68.0 | 76.9 | 82.2 | 75.4 | 61.3 | 72.4
GaitSet[4] | 83.8 | 91.2 | 91.8 | 88.8 | 83.3 | 81.0 | 84.1 | 90.0 | 92.2 | 94.4 | 79.0 | 87.2
Vi-GaitSet | 86.5 | 93.8 | 93.4 | 91.6 | 86.5 | 83.0 | 86.7 | 91.5 | 93.2 | 93.1 | 81.9 | 89.2
GaitPart[5] | 89.1 | 94.8 | 96.7 | 95.1 | 88.3 | 84.9 | 89.0 | 93.5 | 96.1 | 93.8 | 85.8 | 91.5
GaitGL[14] | 90.3 | 94.7 | 95.9 | 94.0 | 91.9 | 86.5 | 90.5 | 95.5 | 97.2 | 96.3 | 87.1 | 92.7
Vi-GaitGL | 89.6 | 94.5 | 95.6 | 95.2 | 93.2 | 87.3 | 91.7 | 95.9 | 97.8 | 96.1 | 85.5 | 92.9
CL#1-2 |  CNN-LB[3] | 37.7 | 57.2 | 66.6 | 61.1 | 55.2 | 54.6 | 55.2 | 59.1 | 58.9 | 48.8 | 39.4 | 54.0
GaitSet[4] | 61.4 | 75.4 | 80.7 | 77.3 | 72.1 | 70.1 | 71.5 | 73.5 | 73.5 | 68.4 | 50.0 | 70.4
Vi-GaitSet | 68.3 | 78.4 | 83.4 | 79.7 | 72.1 | 69.5 | 70.9 | 75.0 | 77.7 | 74.0 | 58.1 | 73.4
GaitPart[5] | 70.7 | 85.5 | 86.9 | 83.3 | 77.1 | 72.5 | 76.9 | 82.2 | 83.8 | 80.2 | 66.5 | 78.7
GaitGL[14] | 76.7 | 88.3 | 90.7 | 86.6 | 82.7 | 77.6 | 83.5 | 86.5 | 88.1 | 83.2 | 68.7 | 83.0
Vi-GaitGL | 81.2 | 92.4 | 94.9 | 93.3 | 87.8 | 82.1 | 87.4 | 89.8 | 90.2 | 87.9 | 72.5 | 87.2

###  3.3 Joint losses

In the proposed multi-task framework, our loss consists of cross entropy (CE)
and triplet loss. Combining the Equation (3), the CE loss can be expressed as:

$\mathcal{L}_{CE}=-\sum^{N}_{j=1}\sum^{M}_{i=1}y_{j}log(p_{ji})~{}~{}w.r.t.~{}~{}p_{ji}=\frac{e^{{\hat{p}}_{ji}}}{\sum^{M}_{i=1}e^{\hat{p}_{ji}}},$
(5)

where $N$ is the number of all gait silhouette sequences and $y_{j}$ is the
discrete ground truth of view of the $j$-th sequence.

Let a triplet of gait silhouette sequences group be $(Q,P,N)$, where $Q$ and
$P$ are from the same subject and $Q$ and $N$ are from two different subjects.
Denote $K$ triplets of fixed identity as
$\\{T_{i}|T_{i}=(f^{Q_{i}}_{final},f^{P_{i}}_{final},f^{N_{i}}_{final}),i=1,2,...,K\\}$,
then combining the Equation (4), the triplet loss can be expressed as:

$\mathcal{L}_{trip}=\frac{1}{K}\sum^{K}_{i=1}\sum^{n}_{j=1}max(m-d^{-}_{ij}+d^{+}_{ij},0),$
(6)

where $d^{-}_{ij}=\|f^{Q_{i}}_{final,j}-f^{N_{i}}_{fina,j}\|^{2}_{2}$,
$d^{+}_{ij}=\|f^{Q_{i}}_{final,j}-f^{P_{i}}_{final,j}\|^{2}_{2}$. In this
paper we used full mining to make triplets.

Combine Equation (5) and (6), the joint loss can be defined as:

$\mathcal{L}=\lambda_{CE}\mathcal{L}_{CE}+\lambda_{trip}\mathcal{L}_{trip},$
(7)

where $\lambda_{CE}$ and $\lambda_{trip}$ are hyper-parameters.

##  4 Experiment

In order to prove the effectiveness of view embedding in gait recognition, two
silhouette-based architectures, i.e. Gaitset [4] and GaitGL [14] are used as
the backbone.

###  4.1 Datasets

CASIA-B dataset [11] contains 124 subjects, each contains 11 views and each
view contains 10 sequences. The sequences are obtained in three scenarios:
normal (NM) (six sequences per subject), walking with bag (BG) (two sequences
per subject) and wearing coat or jacket (CL) (two sequences per subject)
respectively. We conduct experiments following the settings in [3]. These
three settings are small-sample training (ST), medium-sample training (MT) and
large-sample training (LT), in which 24, 62 and 74 subjects are used for
training and the rest are used for test respectively. The first four sequences
of the NM condition (NM#1-4) are kept in gallery, and the rest sequences are
divided into three probe subsets, i.e. NM subsets containing NM #5-6, BG
subsets containing BG #1-2 and CL subsets containing CL #1-2.

OU-MVLP dataset [12] is the largest public gait dataset, which contains 10,307
subjects. 5,153 subjects are used for training and the rest 5,154 subjects are
used for test. Each subject contains 14 views (0°,15°,…,90°;180°,195°,…,270°)
and two sequences (#00-01) per view. In the test set, sequences with index#01
are kept in gallery and those with index #00 are used for probes.

Fig. 2: Examples of View projection matrices for strip 0 and strip 20. The
Diff column shows the absolute difference between the two matrices of
different views at the same strip.

###  4.2 Training Details

In all the experiments, the input is a set of aligned silhouettes of size
64$\times$44 processed by the approach in [12]. Adam optimizer is used for
training and the margin in separate triplet loss is set to 0.2. The
$\lambda_{CE}$ and $\lambda_{trip}$ in Equation (7) are set to 0.5 and 1.

For backbone Gaitset (Vi-Gaitset), the learning rate is set to 1e-4 for both
two datasets. The batch size is set to (8,16) for CASIA-B and (24,16) for OU-
MVLP as mentioned in [4]. For ST, MT and LT settings of CASIA-B, we train our
model for 50K, 60K and 80K iterations. For OU-MVLP, we train our model for
150K iterations.

For GaitGL (Vi-GaitGL), the learning rate is set to 3e-4 for both two datasets
at first. The batch size is set to (8,8) for both datasets and the structure
of the network is all the same as mentioned in [14]. For ST, MT and LT
settings of CASIA-B, we all train our model for 150K iterations and reset the
learning rate to 3e-5 for the next 30K iterations. For OU-MVLP, we train our
model for 120K iterations.

Especially, since the sequence number of OU-MVLP is 20 times of CASIA-B, we
put our view projection module in different places on the original networks,
in order to facilitate the training and prevent overfitting. For CASIA-B, we
replace the separate FC module with our view projection module and the
parameters of view projection matrices are shared. For OU-MVLP, we add our
view projection module after the separate FC module.

Models on CASIA-B are trained with six NVIDIA 2080Ti GPUs and models on OU-
MVLP are trained with two Tesla V100-PCIE-32GB GPUs.

###  4.3 Results and Analysis

The rank-1 accuracies (%) on CASIA-B are shown in Table 1. We compared the
proposed Vi-Gaitset and Vi-GaitGL with original Gaitset [4] and GaitGL [14] as
well as VIDP [2], MGAN [9] and GaitPart [5].

It can be seen that our Vi-Gaitset is more accurate than the original Gaitset
under all of the settings by a large margin. For MT and LT settings, the
performance of Vi-GaitGL is close to that of the original GaitGL in NM and BG
conditions. And the performance of Vi-GaitGL is much better than that of the
original GaitGL in CL conditions by 3.6% and 4.2%. For ST setting, performance
of Vi-GaitGL decreased slightly (1.6% and 0.6%) in NM and BG conditions, and
increased as high as 4.3% in CL condition. We argue that part of the
performance degradation of Vi-GaitGL in ST setting is due to the small size of
training data, which is important to multi-task learning. Another possible
explanation is the view recognition accuracy. For Vi-GaitGL, it is only 96.2%
under ST setting, while it is 97.7% in MT and 97.8% in LT.

Table 2: Rank-1 accuracy (%) on OU-MVLP under 14 probe views excluding
identical-view cases.

Probe angle | Gallery All 14 views
---|---
GEINet | Gaitset | Vi-Gaitset | GaitPart | GaitGL | Vi-GaitGL
0° | 11.4 | 79.5 | 81.8 | 82.6 | 84.3 | 85.6
15° | 29.1 | 87.9 | 89.2 | 88.9 | 89.8 | 90.2
30° | 41.5 | 89.9 | 90.5 | 90.8 | 90.8 | 91.2
45° | 45.5 | 90.2 | 90.5 | 91.0 | 91.0 | 91.5
60° | 39.5 | 88.1 | 89.2 | 89.7 | 90.5 | 91.1
75° | 41.8 | 88.7 | 89.5 | 89.7 | 90.5 | 90.9
90° | 38.9 | 87.8 | 89.0 | 89.9 | 90.3 | 90.4
180° | 14.9 | 81.7 | 83.9 | 85.2 | 88.1 | 88.3
195° | 33.1 | 86.7 | 88.1 | 88.1 | 87.9 | 88.7
210° | 43.2 | 89.0 | 89.7 | 90.0 | 89.6 | 90.6
225° | 45.6 | 89.3 | 89.8 | 90.1 | 89.8 | 90.6
240° | 39.4 | 87.2 | 88.6 | 89.0 | 88.9 | 90.1
255° | 40.5 | 87.8 | 88.5 | 89.1 | 88.9 | 89.9
270° | 36.3 | 86.2 | 87.6 | 88.2 | 88.2 | 89.4
mean | 35.8 | 87.1 | 88.3 | 88.7 | 89.1 | 89.9

The results in rank-1 accuracy (%) on OU-MVLP dataset are shown in Table 2.
The performance of Vi-Gaitset is better than the original Gaitset [4] under
all the probe views. The proposed Vi-GaitGL meets a new state-of-the-art under
various cross-view conditions and the mean rank-1 accuracy is 0.8% higher than
the original GaitGL [14]. When the data is sufficient, out method is
consistently better.

In order to explain the effectiveness of our framework, we compare the
projection matrices of different views in Vi-GaitGL (trained on OU-MVLP). As
illustrated in Figure 2, their difference has obvious vertical texture, which
indicates that the projection matrices of different views has view specificity
for feature mapping.

##  5 Conclusion

In this paper, we propose a general view embedding framework for improved
multi-view gait recognition, in which the view angle is explicitly estimated
and used for model refining. Experimental results on two leading backbone
models show that our idea of explicit view embedding is very effective. The
proposed framework with GaitGL [14] as the backbone meets the state-of-the-art
on two large-scale public gait datasets. It should be pointed out that the
proposed framework is not competitive but rather complementary to existing
works.

## References

  * [1] Kohei Shiraga, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi,  “Geinet: View-invariant gait recognition using a convolutional neural network,”  in ICB. IEEE, 2016, pp. 1–8.
  * [2] Maodi Hu, Yunhong Wang, Zhaoxiang Zhang, James J Little, and Di Huang,  “View-invariant discriminative projection for multi-view gait-based human identification,”  IEEE TIFS, vol. 8, no. 12, pp. 2034–2045, 2013.
  * [3] Zifeng Wu, Yongzhen Huang, Liang Wang, Xiaogang Wang, and Tieniu Tan,  “A comprehensive study on cross-view gait based human identification with deep cnns,”  IEEE TPAMI, vol. 39, no. 2, pp. 209–226, 2016.
  * [4] Hanqing Chao, Yiwei He, Junping Zhang, and Jianfeng Feng,  “Gaitset: Regarding gait as a set for cross-view gait recognition,”  in AAAI, 2019, vol. 33, pp. 8126–8133.
  * [5] Chao Fan, Yunjie Peng, Chunshui Cao, Xu Liu, Saihui Hou, Jiannan Chi, Yongzhen Huang, Qing Li, and Zhiqiang He,  “Gaitpart: Temporal part-based model for gait recognition,”  in CVPR, 2020, pp. 14225–14233.
  * [6] Beibei Lin, Shunli Zhang, and Feng Bao,  “Gait recognition with multiple-temporal-scale 3d convolutional neural network,”  in ACM MM, 2020, pp. 3054–3062.
  * [7] Thomas Wolf, Mohammadreza Babaee, and Gerhard Rigoll,  “Multi-view gait recognition using 3d convolutional neural networks,”  in ICIP. IEEE, 2016.
  * [8] Jinguang Han and Bir Bhanu,  “Individual recognition using gait energy image,”  TPAMI, vol. 28, no. 2, pp. 316–322, 2005.
  * [9] Yiwei He, Junping Zhang, Hongming Shan, and Liang Wang,  “Multi-task gans for view-specific feature learning in gait recognition,”  IEEE TIFS, vol. 14, no. 1, pp. 102–113, 2018.
  * [10] Yuqi Zhang, Yongzhen Huang, Shiqi Yu, and Liang Wang,  “Cross-view gait recognition by discriminative feature learning,”  IEEE TIP, vol. 29, pp. 1001–1015, 2019.
  * [11] Shiqi Yu, Daoliang Tan, and Tieniu Tan,  “A framework for evaluating the effect of view angle, clothing and carrying condition on gait recognition,”  in ICPR. IEEE, 2006, vol. 4, pp. 441–444.
  * [12] Noriko Takemura, Yasushi Makihara, Daigo Muramatsu, Tomio Echigo, and Yasushi Yagi,  “Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition,”  IPSJ TCVA, vol. 10, no. 1, pp. 4, 2018.
  * [13] Shuangqun Li, Wu Liu, Huadong Ma, and Shaopeng Zhu,  “Beyond view transformation: Cycle-consistent global and partial perception gan for view-invariant gait recognition,”  in ICME, 2018.
  * [14] Beibei Lin, Shunli Zhang, Xin Yu, Zedong Chu, and Haikun Zhang,  “Learning effective representations from global and local features for cross-view gait recognition,”  arXiv preprint arXiv:2011.01461, 2020.
  * [15] Manuel J Marín-Jiménez, Francisco M Castro, Nicolás Guil, F de La Torre, and R Medina-Carnicer,  “Deep multi-task learning for gait-based biometrics,”  in ICIP. IEEE, 2017, pp. 106–110.
  * [16] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun,  “Multi-task multi-sensor fusion for 3d object detection,”  in CVPR, 2019, pp. 7345–7353.
  * [17] Xi Yin and Xiaoming Liu,  “Multi-task convolutional neural network for pose-invariant face recognition,”  IEEE TIP, vol. 27, no. 2, pp. 964–975, 2017.
  * [18] Jifeng Dai, Kaiming He, and Jian Sun,  “Instance-aware semantic segmentation via multi-task network cascades,”  in CVPR, 2016, pp. 3150–3158.
  * [19] Shaoxiong Zhang, Yunhong Wang, and Annan Li,  “Gait-based age estimation with deep convolutional neural network,”  in ICB. IEEE, 2019, pp. 1–8.
  * [20] Worapan Kusakunniran, Qiang Wu, Jian Zhang, Hongdong Li, and Liang Wang,  “Recognizing gaits across views through correlated motion co-clustering,”  IEEE TIP, vol. 23, no. 2, pp. 696–709, 2013.
  * [21] Rijun Liao, Chunshui Cao, Edel B Garcia, Shiqi Yu, and Yongzhen Huang,  “Pose-based temporal-spatial network (ptsn) for gait recognition with carrying and clothing variations,”  in CCBR. Springer, 2017, pp. 474–483.
  * [22] Rijun Liao, Shiqi Yu, Weizhi An, and Yongzhen Huang,  “A model-based gait recognition method with body pose and human prior knowledge,”  Pattern Recognition, vol. 98, pp. 107069, 2020.
  * [23] Xianye Ben, Chen Gong, Peng Zhang, Xitong Jia, Qiang Wu, and Weixiao Meng,  “Coupled patch alignment for matching cross-view gaits,”  IEEE TIP, vol. 28, no. 6, pp. 3142–3157, 2019.
  * [24] Xianye Ben, Chen Gong, Peng Zhang, Rui Yan, Qiang Wu, and Weixiao Meng,  “Coupled bilinear discriminant projection for cross-view gait recognition,”  IEEE TCSVT, vol. 30, no. 3, pp. 734–747, 2019.