# Polarization-path-frequency entanglement using interferometry and frequency
shifters

Mrittunjoy Guha Majumdar<EMAIL_ADDRESS>Quantum Optics & Quantum
Information, Department of Instrumentation and Applied Physics, Indian
Institute of Science, Bengaluru, India  C. M. Chandrashekar
<EMAIL_ADDRESS>Quantum Optics & Quantum Information, Department of
Instrumentation and Applied Physics, Indian Institute of Science, Bengaluru,
India  The Institute of Mathematical Sciences, C. I. T. Campus, Taramani,
Chennai 600113, India  Homi Bhabha National Institute, Training School
Complex, Anushakti Nagar, Mumbai 400094, India

###### Abstract

Higher dimensional Hilbert space along with ability to control multiple
degrees of freedom of photon and entangle them has enabled new quantum
protocols for various quantum information processing applications. Here, we
propose a scheme to generate and control polarization-path-frequency
entanglement using the operative elements required to implement a
polarization-controlled quantum walk in the path(position) space and frequency
domain. Hyperentangled states manifests in the controlled dynamics using an
interferometric setup where half-wave plates, beam-splitters and frequency
shifters such as those based on the electro-optic effect are used to
manipulate the polarization, path and frequency degrees of freedom
respectively. The emphasis is on utilizing the polarization to influence the
movement to a specific value in the frequency and position space. Negativity
between the subspaces is calculated to demonstrate the controllability of the
entanglement between them. Progress reported with experimental demonstration
of realization of quantum walk using quantum states of light makes quantum
walks a practical approach to generate hyperentangled states.

##  I Introduction

Entanglement has been the underlying resource for various quantum mechanical
applications. Entanglement across and within varied degrees of freedom quantum
particle has been even more useful in optimizing the number of physical qubits
and the logical quantum mechanical operations and applications one can
undertake with it Horodecki et al. (2009); Plenio and Virmani (2014); Weiner
and Stringer (2002); Deng et al. (2017); Xie et al. (2015). When it comes to
photonic qubits, we have had hyperentanglement across various degrees of
freedom: polarization-path Ciampini et al. (2016), polarization-OAM Majumdar
(2020), polarization-frequency Shi et al. (2012); Prabhu et al. (2021) as well
as polarization-time bin Du et al. (2016); Chapman et al. (2019). The level of
advancement in the realm of hyperentanglement has reached a point where we can
now simultaneously use three degrees of freedom for quantum information
processing Wang et al. (2018); Xu et al. (2012). Applications of
hyperentanglement have included entanglement purification and
hyperdistillation Zhou and Sheng (2021); Wang et al. (2015), Bell state and
GHZ state analysis Walborn et al. (2003); Song et al. (2013), high-capacity
quantum secure direct communication Tie-Jun et al. (2011), quantum error
correction Wilde and Uskov (2009), quantum superdense coding Rui-Tong et al.
(2012), quantum secret sharing Gu et al. (2012) and quantum sensing Smith III
(2015). With continued progress being reported establishing the control over
various combination of degrees of freedom of light, combining them by
simultaneously controlling more than three degree of freedom for quantum
information processing task can be envisioned.

A quantum walk is the quantum version of the classical random walk, and in the
form of algorithm it has shown significant advantages over classical
algorithms Childs et al. (2002). The advantage of a quantum walk comes from
the way it evolves the walker in a superposition of position space states
entangling the Hilbert spaces associated with the walker. Such quantum walks
have been ubiquitous in various applications such as search algorithms
Travaglione and Milburn (2002); Shenvi et al. (2003). While Richard Feynman
proposed the possibility of implementation of universal quantum computation
with a time-dependent Hamiltonian Feynman (1985), its possibility was shown
when the Hamiltonian is limited to being the adjacency matrix of a low-degree
graph Childs (2009). The resultant continuous-time quantum walk was found to
be a universal quantum computation primitive. Recent works have shown that the
discrete-time quantum walks are also universal for quantum computation Lovett
et al. (2010); Singh et al. (2021a) as are multi-particle quantum walks in
interacting systems such as the Bose-Hubbard model Childs et al. (2013).
Quantum walks has also been one of the effective algorithmic approach to
generate controlled dynamics and engineer entanglement between subsystems that
are part of the dynamics Chandrashekar (2012). Recently, hyperentangled states
generated in discrete-time quantum walks have been used as a resource for
efficacious and viable quantum dialogue protocol Liu et al. (2020). New
schemes for realization of a quantum walk based hyperentangled states of
photons using J - (q)- plates and polarization beamsplitters has also been
proposed Yasir and Chandrashekar (2021).

To use frequency of photons as effective degree of freedom to generate higher
dimensional entangled states, conversion of frequency and ability to retain
them in superposition of different frequencies are important. The optical
frequency bands for storage and transmission of optical photon qubits are
distinct. The inter-convertibility between the two regimes is pivotal to a
seamless integration of quantum computing and communication technologies. A
recent experiment demonstrates the coherent frequency conversion of photons
entangled in their polarization Ramelow et al. (2012). This has naturally led
to the prospect of changing the photon’s spectral bandwidth during frequency
conversion by suitably designing phase matching. But what is even more
interesting is the possibility of accessing myriad states in the higher-
dimensional Hilbert Space of the frequency degree-of-freedom. The
addressability of these states can be made precise by specific hyperentangled
structures that establish a one-to-one correspondence between a higher
dimensional space qubit and an associated hyperentangled qubit that tags it.
In our scheme, we look at doing this selectively by applying an (hyper-
conditional) operation depending on what the value of the ‘tagging’ degree-of-
freedom is, in an interferometric setup. This augments the higher-dimensional
nature of hyperentangled states by harnessing the resources of multiple
degrees-of-freedom as well as the multiplicity inherent in single, specific
degrees-of-freedom.

Higher-dimensional Hilbert Spaces, particularly when tagged by one or more
associated hyperentangled degrees-of-freedom, can be effective in facilitating
quantum information processing tasks. Quantum hyperdense coding protocol in
higher dimensions was utilized for a quantum secure direct communication
(QSDC) protocol and was shown to have a high capacity and better security than
that we can obtain with a two-dimensional entangled quantum system Wang et al.
(2005). A platform for photon-efficient quantum communication with
energy–time–polarization high-dimensional encoding has been proposed using a
biphoton frequency comb technique Xie et al. (2015). Recently a higher-
dimensional measurement-device-independent quantum key distribution (MDI-QKD)
protocol in multiple DOFs which is based on a linear-optical hyperentangled
Bell state analysis has also been proposed Yang et al. (2021). Most
applications of the frequency degree-of-freedom in a higher-dimensional
entangled system are premised on distinct frequency shifts and usually on two
frequency states, such as in the case of remote state preparation of single-
photon states in polarization and frequency DoFs Wang et al. (2021a). While
there has been discussion on spectral linear optical quantum computation
Lukens and Lougovski (2017), the spectrum of frequency modes have not been
utilized in quantum information processing, particularly with discernment of
individual modes over a large spread of frequencies.

In this work, we propose a scheme for harnessing path-polarization-frequency
hyperentanglement by undertaking a polarization-controlled quantum walk in the
frequency domain and position space. We would like to make a note that the
path and position degree of freedom are treated equally in this work. Our
proposal has an interferometry approach where polarizing beam splitters and
frequency shifter are used with a control on polarization state using a half-
wave plate to control. Frequency shifters are like those that use the electro-
optic effect Minet et al. (2020); Preble et al. (2007); Wright et al. (2017);
Hu et al. (2021) to shift the frequency of light using variations in the
birefringence in an optical medium induced by an electric field. We also show
the way entanglement between the sub-spaces manifests with the dynamics.

In section II, we discuss discrete-time quantum walks and present our model
for polarization-dependent discrete-time quantum walk in the position and
frequency space and the way hyperentangled states manifests in the dynamics.
In the same section we also discuss the current state of art in resolving the
frequency shifts generated by frequency shifters like electro-optic
modulators. In section III we conclude with our remarks.

##  II The Model

The quantum walk is distinct from its classical counterpart, particularly in
that there is a ballistic spread of the probability distribution of the walker
due to interference effects, unlike when there is a diffusive effect as in the
case of the classical walk. Quantum walks are studied in both, continuous-time
and discrete time frame work. In this work we will focus on discrete-time
version of the quantum walk Meyer (1996); Ambainis et al. (2001);
Chandrashekar et al. (2008). In a discrete-time quantum walk, the walker is a
quantum system which resides on the composite Hilbert space
$\mathcal{H}_{T}=\mathcal{H}_{c}\otimes\mathcal{H}_{p}$. An internal degree of
freedom of the walker spans the basis state of the coin Hilbert space
$\mathcal{H}_{c}$ and the position space on which the walker spreads spans the
basis state of the position Hilbert space $\mathcal{H}_{p}$. For a walker on a
one-dimensional position space the basis state of $\mathcal{H}_{c}$ will be
$|0\rangle$ and $|1\rangle$ and the basis state of the $\mathcal{H}_{p}$ will
be $\\{\ket{x}\\}$ where $x\in\mathbb{Z}$ represents the labels on the
position space. The general form of initial state of the walker can be written
as,

$\ket{\Psi}_{in}=\cos(\delta)\ket{0}+e^{-i\eta}\sin(\delta)\ket{1}\otimes\ket{x=0}.$
(1)

Each step of the walk evolution in one-dimensional position space is defined
using a quantum coin operation acting on the coin space

$\mathcal{C}(\theta)=\begin{bmatrix}\cos(\theta)&-\sin(\theta)\\\
\sin(\theta)&~{}~{}\cos(\theta)\end{bmatrix}$ (2)

followed by a conditional position shift operation acting on the full Hilbert
space

$\mathcal{S}=\sum_{x\in\mathbb{Z}}\bigg{(}\ket{0}\bra{0}\otimes\ket{x-1}\bra{x}+\ket{1}\bra{1}\otimes\ket{x+1}\bra{x}\bigg{)}.$
(3)

The state after $t$ steps of the walk will be

$\ket{\Psi}_{t}=\Big{[}\mathcal{S}(\mathcal{C}(\theta)\otimes\mathbb{I}_{x})\Big{]}^{t}\ket{\Psi}_{in}.$
(4)

By using different variant of the coin and shift operators, different forms of
discrete-time quantum walks can be realized Chandrashekar (2012); Singh et al.
(2021b).

To generate hyperentanglement in polarization-frequency-position space of the
photon we extend the above described evolution to the higher dimensional
Hilbert space
$\mathcal{H}_{Hyp}=\mathcal{H}_{c}\otimes\mathcal{H}_{p}\otimes\mathcal{H}_{f}$
associated with the photons. The $\mathcal{H}_{c}$ will span the polorization
degree of freedom $\ket{H}$ and $\ket{V}$, the $\mathcal{H}_{p}$ will span the
position space $\ket{x_{i}}$ and the $\mathcal{H}_{f}$ will span the frequency
space $\ket{f_{i}}$. The operations to define the walk on these three degree
of freedom will be a composition of the coin operation on the polarization
space followed by a polarization conditioned shift operation on the position
and frequency space. The Half-Wave plate (HWP) executes the rotation
$C(\theta)$ given by the Eq. (2) on the polarization space. The polarization
Beamsplitter (PBS) executes a polarization conditioned shift operation of the
form

$\displaystyle\mathcal{S_{P}}=$
$\displaystyle\Bigg{[}\ket{H}\bra{H}\otimes\ket{x+1}\bra{x}$
$\displaystyle+\sum_{i\,\in\,\mathbb{Z}}\bigg{(}\ket{V}\bra{V}\otimes\ket{x-1}\bra{x}\bigg{)}\Bigg{]}\otimes\mathbb{I}_{f}.$
(5)

Electro-optics frequency shifters (EOFSs) are not polarization dependent
shifters and therefore cannot be directly used as a polarization conditioned
shift operator as in the form of Eq. (II). The electro-optic effect mixes
optical and microwave fields directly, and in recent on-chip realisations of
frequency shifting, the frequency of all inserted photons were seen to be
shifted to another frequency in the regime of full conversion Hu et al.
(2021). This allows us tune a single frequency shift and make it evolve in
superposition with the polarization state of the photon. A polarization
dependent shift operator using EOMs can be executed by placing the EOM (or any
frequency shifter) in one of the path after the two polarization states are
spatially separated. The operational representation will be of the form,

$\displaystyle\mathcal{S}_{f}=$
$\displaystyle\Bigg{[}\ket{H}\bra{H}\otimes\mathbb{I}_{x}\otimes\ket{f_{i+1}}\bra{f_{i}}$
$\displaystyle+\sum_{i\,\in\,\mathbb{Z_{+}}}\bigg{(}\ket{V}\bra{V}\otimes\mathbb{I}_{x}\otimes\ket{f_{i}}\bra{f_{i}}\bigg{)}\Bigg{]}.$
(6)

Figure 1: Schematic of optical elements with spatial and temporal markings to
realize polarization-controlled quantum walk in the frequency and position
degrees-of -freedom. Each step of walk is a composition of two HWP, two PBS
and one EOM. EOMs are placed across the path where only state $\ket{H}$ pass
through. Each detector will capture different frequencies with the difference
between them being a function of the electro-optic frequency shifts carried
out by the EOMs.

This form of shift operators has been effectively used in defining directed
quantum walks Hoyer and Meyer (2009); Chandrashekar and Busch (2014). Thus,
for a photon in initial state

$\displaystyle|\Psi_{p}\rangle_{in}=\bigg{(}\cos(\delta)|H\rangle-\sin(\delta)\ket{V}\bigg{)}\otimes\ket{x_{0}}\bra{x_{0}}\otimes\ket{f_{0}}\bra{f_{0}}$
(7)

each step of walk operation will be a composition of evolution in
superposition of position space followed by the evolution in superposition of
frequency space,

$\displaystyle\mathcal{W}=\mathcal{S}_{f}\mathcal{S_{P}}\bigg{(}\mathcal{C}(\theta)\otimes\mathbb{I}_{x}\otimes\mathbb{I}_{f_{i}}\bigg{)}.$
(8)

In Fig. 1 we give a schematic of the optical setup for four step of discrete-
time quantum walk composing of HWPs, PBSs, and EOMs to generate polarization-
path-frequency entanglement. Two HWP, two PBS and one EOM is required for each
step of the walk. As shown in the schematic the detector are placed at the end
identified with the position space and each position space will uniquely
capture different frequencies with the difference between the adjacent
detectors will be the frequency of the acoustic wave used to drive frequency
shift. After $t$ steps of walk evolution in network of optical setup will be

$\displaystyle|\Psi_{p}(t)\rangle$
$\displaystyle=\mathcal{W}^{t}|\Psi_{p}\rangle_{in}$
$\displaystyle=\sum_{k=0}^{t}\Bigg{[}\xi^{H}_{k}(\delta,\theta)|H\rangle|f_{t}\rangle|x_{-n+2t}\rangle$
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}+\xi^{V}_{k}(\delta,\theta)|V\rangle|f_{t}\rangle|x_{-n+2t}\rangle\Bigg{]}.$
(9)

The above state is a Schmidt rank vector of $(2,t,t)$. The probability
distribution in position space will be

$\displaystyle
P(x,t)=\sum_{x}\bra{x}\bigg{(}\ket{\Psi_{p}(t)}\bra{\Psi_{p}(t)}\bigg{)}\ket{x}$
(10)

and in frequency space it will be

$\displaystyle
P(f_{i},t)=\sum_{i}\bra{f_{i}}\bigg{(}\ket{\Psi_{p}(t)}\bra{\Psi_{p}(t)}\bigg{)}\ket{f_{i}}.$
(11)

Figure 2: Probability distribution in position space after 20 step of walk
using different rotation angles $\theta$. We can see that the probability of
finding the states is zero for alternate position space. When this is compared
with the schematic in Fig. 1, detectors are places only at odd(even) positions
after odd (even) number of steps. Figure 3: Probability distribution in
frequency space after 20 step of walk using different rotation angles
$\theta$. Each frequency $f_{i}$ will be identified with separate position
space with non-zero probability in Fig. 2.

In Figs. 2 and 3 we show the spread position Hilbert space and frequency space
after 20 steps of quantum walk when different rotations in HWPs are used.
These distribution can be further controlled by using different rotation
angles $\theta$ in each path in combination of $\delta$, initial state
parameter. By neglecting the position with zero probabilities in Fig. 2, each
position labelling can be identified with the frequency.

We characterize the entanglement between the different degrees of freedom
using negativity, $\mathcal{N}(\rho)=\frac{||\rho^{\Gamma_{A}}||_{1}-1}{2}$,
where $\rho^{\Gamma_{A}}$ is the partial transpose of $\rho$ with respect to
subsystem $A$ and $||A||$ is the trace norm.

Figure 4: Negativity between polarization and frequency space as function of
steps (time). Figure 5: Negativity between position and frequency space as
function of steps (time).

In Fig. 4 we show the negativity between polarization and frequency space,
$\mathcal{N}_{P-F}$ after taking trace over the position space and a similar
behavior can be seen for negativity between polarization and position space
$\mathcal{N}_{F-Pos}$. With increase in $\theta$ average value of negativity
increases and an inverse is seen for negativity between polarization and
frequency space obtained after taking trace over the polarization space.
Unlike $\mathcal{N}_{P-F}$ which oscillated around an average value,
$\mathcal{N}_{F-Pos}$ increases with number of steps and this is due to the
increase in the dimension of both, position and frequency with steps.

In this work, we have looked at the possibility of manipulating polarization
and path degree of freedom to generate an extended dimension of frequency
space and generate hyperentangled states. In general resolving closely placed
frequencies is an experimental hurdle. For example, a 1550 nm. laser beam has
frequency of around 193 THz, while an EOM can produce a frequency shift of the
order of 10-15 GHz MHz, which is around $0.005\%$ of the frequency of the
laser beam. Traditionally, unwanted symmetric sidebands are invariably
produced in EOMs, and we therefore need to filter undesireable components,
thereby leading to less efficient frequency shifts. Quadrature and in-phase
modulators remove such sidebands using destructive interference among multiple
units, while Serrodyne modulators utilise saw-tooth waveforms for the
generation of frequency shifts. This, however, is not easily extendible to the
GHz regime and requires high-power, broadband electronics. There are also
other methods such as spectral shearing Wright et al. (2017); Fan et al.
(2016) and adiabatic tuning of optical cavity resonance Preble et al. (2007),
which can undertake frequency shifts with pulsed operations having a known
timing reference. Recently, a quadrature phase-shift keying modulator was used
to attain a 15.65 GHz frequency shift Chen et al. (2021). When we have
multiple such devices, it get more complicated but in our scheme, the closely
placed frequencies are handled by tagging them with position space without
need for a special frequency resolver.

However, advancements in the field of frequency discriminators and
instantaneous frequency measurement subsystems Schilt et al. (2011); Hartnett
et al. (2001); Choondaragh and Masoumi (2014); Stec et al. (2006); Gruchala et
al. (2002) have been reported. Microstrip-based discriminators have been seen
to achieve high sensitivity and high dynamic range, with a recent work being
able to attain a frequency resolution of 16 MHz by using open-loop spiral
resonators instead of the more conventionally used coupled resonators
Rahimpour and Masoumi (2018). High resolution and accuracy have also been
obtained with quadrature microwave frequency discriminators (QMFD) Tsui and
McCormick (1990); Chatterjee et al. (2015); de Moura Barbosa and dos Santos
(2005); Lurz et al. (2018); Yingjiao and Wenfang (2018); Wang et al. (2021b).
Adaptive spectrum sensing techniques based on group delay induced by Brillouin
scattering in optical fiber has been shown to be able to temporally
discriminate two-tone signals with a resolution of 60 MHz Kadum et al. (2021).
Modern spectrum analyzers can also provide robust and efficient ways to
observe and discriminate between frequency modes. Recently, Pelusi et al used
compact planar rib waveguides based on $\mathrm{As}_{2}\mathrm{S}_{3}$ glass,
providing a virtual ‘lumped’ high nonlinearity in a monolithic platform that
was capable of integrating multiple functions, and output spectral resolution
of 1.25 GHz (0.01 nm) Pelusi et al. (2009). Above outlined experimental
progress in handling frequency in addition to it control through quantum walk
approach shows a huge potential of hyperentangled states in polarization-
frequency-position space in quantum information processing applications.

##  III Conclusion

In this work, we present a scheme to pass single photon through an intricate
network of concatenated units comprising of a combination of HWPs, PBSs and
AOMs and generate hyperentangled state in polarization-path-frequency domain.
BY harnessing the power of a polarization-controlled quantum walk one can also
tune to the desired frequency from the extended frequency space that can be
generated in the system. We have show the entanglement generated between
different degree of freedom associated with photon in quantum walk setup. We
particularly see a generally rising (normalised) negativity of entanglement
between the position and frequency degrees-of-freedom. The addition to
generation of hyperentangled states for various quantum information processing
tasks, the scheme can also be harnessed to generate closely shifted
frequencies and resolve them with tagging with position space.

Acknowledgement. MGM would like to acknowledge the insightful discussions with
Abhay Hegde and Ameen Yasir. MGM and CMC acknowledge the support from the
Office of Principal Scientific Advisor to Government of India, project no.
Prn.SA/QSim/2020.

## References

  * Horodecki et al. (2009) R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Reviews of modern physics 81, 865 (2009).
  * Plenio and Virmani (2014) M. B. Plenio and S. S. Virmani, Quantum information and coherence pp. 173–209 (2014).
  * Weiner and Stringer (2002) J. Weiner and C. Stringer, in _The Piltdown Forgery_ (Oxford University Press, 2002).
  * Deng et al. (2017) F.-G. Deng, B.-C. Ren, and X.-H. Li, Science bulletin 62, 46 (2017).
  * Xie et al. (2015) Z. Xie, T. Zhong, S. Shrestha, X. Xu, J. Liang, Y.-X. Gong, J. C. Bienfang, A. Restelli, J. H. Shapiro, F. N. Wong, et al., Nature Photonics 9, 536 (2015).
  * Ciampini et al. (2016) M. A. Ciampini, A. Orieux, S. Paesani, F. Sciarrino, G. Corrielli, A. Crespi, R. Ramponi, R. Osellame, and P. Mataloni, Light: Science & Applications 5, e16064 (2016).
  * Majumdar (2020) M. G. Majumdar, Quantum Information Processing 19, 1 (2020).
  * Shi et al. (2012) J. Shi, S.-J. Yun, Y.-F. Bai, P. Xu, and S.-N. Zhu, Optics Communications 285, 5549 (2012).
  * Prabhu et al. (2021) A. V. Prabhu, B. Suri, and C. Chandrashekar, Physical Review A 103, 052608 (2021).
  * Du et al. (2016) F.-F. Du, F.-G. Deng, and G.-L. Long, Scientific reports 6, 1 (2016).
  * Chapman et al. (2019) J. C. Chapman, C. C. Lim, and P. G. Kwiat, arXiv preprint arXiv:1908.09018 (2019).
  * Wang et al. (2018) X.-L. Wang, Y.-H. Luo, H.-L. Huang, M.-C. Chen, Z.-E. Su, C. Liu, C. Chen, W. Li, Y.-Q. Fang, X. Jiang, et al., Physical review letters 120, 260502 (2018).
  * Xu et al. (2012) H.-B. Xu, K. Du, and C.-F. Qiao, Journal of Modern Optics 59, 1265 (2012).
  * Zhou and Sheng (2021) L. Zhou and Y.-B. Sheng, arXiv preprint arXiv:2101.09006 (2021).
  * Wang et al. (2015) T.-J. Wang, L.-L. Liu, R. Zhang, C. Cao, and C. Wang, Optics express 23, 9284 (2015).
  * Walborn et al. (2003) S. Walborn, S. Pádua, and C. Monken, Physical Review A 68, 042313 (2003).
  * Song et al. (2013) S. Song, Y. Cao, Y.-B. Sheng, and G.-L. Long, Quantum information processing 12, 381 (2013).
  * Tie-Jun et al. (2011) W. Tie-Jun, L. Tao, D. Fang-Fang, and D. Fu-Guo, Chinese Physics Letters 28, 040305 (2011).
  * Wilde and Uskov (2009) M. M. Wilde and D. B. Uskov, Physical Review A 79, 022305 (2009).
  * Rui-Tong et al. (2012) Z. Rui-Tong, G. Qi, C. Li, W. Hong-Fu, and Z. Shou, Chinese Physics B 21, 080303 (2012).
  * Gu et al. (2012) B. Gu, F. Xu, L. Ding, and Y. Zhang, International Journal of Theoretical Physics 51, 3559 (2012).
  * Smith III (2015) J. F. Smith III, in _Quantum Information and Computation XIII_ (International Society for Optics and Photonics, 2015), vol. 9500, p. 95000E.
  * Childs et al. (2002) A. M. Childs, E. Farhi, and S. Gutmann, Quantum Information Processing 1, 35 (2002).
  * Travaglione and Milburn (2002) B. C. Travaglione and G. J. Milburn, Physical Review A 65, 032310 (2002).
  * Shenvi et al. (2003) N. Shenvi, J. Kempe, and K. B. Whaley, Physical Review A 67, 052307 (2003).
  * Feynman (1985) R. P. Feynman, Optics news 11, 11 (1985).
  * Childs (2009) A. M. Childs, Physical review letters 102, 180501 (2009).
  * Lovett et al. (2010) N. B. Lovett, S. Cooper, M. Everitt, M. Trevers, and V. Kendon, Physical Review A 81, 042330 (2010).
  * Singh et al. (2021a) S. Singh, P. Chawla, A. Sarkar, and C. Chandrashekar, Scientific Reports 11, 1 (2021a).
  * Childs et al. (2013) A. M. Childs, D. Gosset, and Z. Webb, Science 339, 791 (2013).
  * Chandrashekar (2012) C. Chandrashekar, arXiv preprint arXiv:1212.5984 (2012).
  * Liu et al. (2020) F. Liu, X. Zhang, P.-A. Xu, Z.-X. He, and H.-Y. Ma, International Journal of Theoretical Physics 59, 3491 (2020).
  * Yasir and Chandrashekar (2021) P. Yasir and C. Chandrashekar, arXiv preprint arXiv:2109.11158 (2021).
  * Ramelow et al. (2012) S. Ramelow, A. Fedrizzi, A. Poppe, N. K. Langford, and A. Zeilinger, Physical Review A 85, 013845 (2012).
  * Wang et al. (2005) C. Wang, F.-G. Deng, Y.-S. Li, X.-S. Liu, and G. L. Long, Physical Review A 71, 044305 (2005).
  * Yang et al. (2021) M.-Y. Yang, P. Zhao, L. Zhou, W. Zhong, and Y.-B. Sheng, Laser Physics Letters 18, 075204 (2021).
  * Wang et al. (2021a) M. Wang, F. Yan, and T. Gao, arXiv preprint arXiv:2103.07671 (2021a).
  * Lukens and Lougovski (2017) J. M. Lukens and P. Lougovski, Optica 4, 8 (2017).
  * Minet et al. (2020) Y. Minet, L. Reis, J. Szabados, C. S. Werner, H. Zappe, K. Buse, and I. Breunig, Optics express 28, 2939 (2020).
  * Preble et al. (2007) S. F. Preble, Q. Xu, and M. Lipson, Nature Photonics 1, 293 (2007).
  * Wright et al. (2017) L. J. Wright, M. Karpiński, C. Söller, and B. J. Smith, Physical review letters 118, 023601 (2017).
  * Hu et al. (2021) Y. Hu, M. Yu, D. Zhu, N. Sinclair, A. Shams-Ansari, L. Shao, J. Holzgrafe, E. Puma, M. Zhang, and M. Lončar, Nature 599, 587 (2021).
  * Meyer (1996) D. A. Meyer, Journal of Statistical Physics 85, 551 (1996).
  * Ambainis et al. (2001) A. Ambainis, E. Bach, A. Nayak, A. Vishwanath, and J. Watrous, in _Proceedings of the thirty-third annual ACM symposium on Theory of computing_ (2001), pp. 37–49.
  * Chandrashekar et al. (2008) C. M. Chandrashekar, R. Srikanth, and R. Laflamme, Physical review A 77, 032326 (2008).
  * Singh et al. (2021b) S. Singh, C. H. Alderete, R. Balu, C. Monroe, N. M. Linke, and C. M. Chandrashekar, Physical review A 104, 062401 (2021b).
  * Hoyer and Meyer (2009) S. Hoyer and D. A. Meyer, Phys. Rev. A 79 (2009).
  * Chandrashekar and Busch (2014) C. M. Chandrashekar and T. Busch, Scientific Reports 4 (2014).
  * Fan et al. (2016) L. Fan, C.-L. Zou, M. Poot, R. Cheng, X. Guo, X. Han, and H. X. Tang, Nature Photonics 10, 766 (2016).
  * Chen et al. (2021) C. Chen, J. E. Heyes, J. H. Shapiro, and F. N. Wong, Scientific Reports 11, 1 (2021).
  * Schilt et al. (2011) S. Schilt, N. Bucalovic, L. Tombez, V. Dolgovskiy, C. Schori, G. Di Domenico, M. Zaffalon, and P. Thomann, Review of Scientific Instruments 82, 123116 (2011).
  * Hartnett et al. (2001) J. G. Hartnett, M. E. Tobar, and E. N. Ivanov, IEEE transactions on ultrasonics, ferroelectrics, and frequency control 48, 743 (2001).
  * Choondaragh and Masoumi (2014) S. E. Choondaragh and N. Masoumi, in _7’th International Symposium on Telecommunications (IST’2014)_ (IEEE, 2014), pp. 198–202.
  * Stec et al. (2006) B. Stec, C. Recko, and W. Susek, in _2006 International Conference on Microwaves, Radar & Wireless Communications_ (IEEE, 2006), pp. 240–242.
  * Gruchala et al. (2002) H. Gruchala, T. Rapacki, A. Rutkowski, and B. Stec, in _14th International Conference on Microwaves, Radar and Wireless Communications. MIKON-2002. Conference Proceedings (IEEE Cat. No. 02EX562)_ (IEEE, 2002), vol. 1, pp. 63–66.
  * Rahimpour and Masoumi (2018) H. Rahimpour and N. Masoumi, IEEE Transactions on Instrumentation and Measurement 67, 2373 (2018).
  * Tsui and McCormick (1990) J. B. Tsui and W. S. McCormick, _Instantaneous frequency measurement (ifm) receiver with only two delay lines_ (1990), uS Patent 4,963,816.
  * Chatterjee et al. (2015) D. Chatterjee, K. Blau, and M. A. Hein, in _2015 European Microwave Conference (EuMC)_ (IEEE, 2015), pp. 889–892.
  * de Moura Barbosa and dos Santos (2005) G. de Moura Barbosa and J. C. A. dos Santos, in _SBMO/IEEE MTT-S International Conference on Microwave and Optoelectronics, 2005._ (IEEE, 2005), pp. 381–385.
  * Lurz et al. (2018) F. Lurz, B. Scheiner, S. Linz, R. Weigel, and A. Koelpin, in _2018 22nd International Microwave and Radar Conference (MIKON)_ (IEEE, 2018), pp. 87–90.
  * Yingjiao and Wenfang (2018) R. Yingjiao and L. Wenfang, in _2018 IEEE 4th International Conference on Computer and Communications (ICCC)_ (IEEE, 2018), pp. 1008–1012.
  * Wang et al. (2021b) Z. Wang, C. Yang, E. Zhu, and W. Xu, IEEE Transactions on Microwave Theory and Techniques (2021b).
  * Kadum et al. (2021) J. E. Kadum, A. Misra, R. Das, and T. Schneider, in _Signal Processing in Photonic Communications_ (Optical Society of America, 2021), pp. SpW1F–7.
  * Pelusi et al. (2009) M. Pelusi, F. Luan, T. D. Vo, M. R. Lamont, S. J. Madden, D. A. Bulla, D.-Y. Choi, B. Luther-Davies, and B. J. Eggleton, nature photonics 3, 139 (2009).


# Repetitive Activity Counting by Sight and Sound

Yunhua Zhang1 Ling Shao2 Cees G. M. Snoek1
1University of Amsterdam 2Inception Institute of Artificial Intelligence

###### Abstract

This paper strives for repetitive activity counting in videos. Different from
existing works, which all analyze the visual video content only, we
incorporate for the first time the corresponding sound into the repetition
counting process. This benefits accuracy in challenging vision conditions such
as occlusion, dramatic camera view changes, low resolution, etc. We propose a
model that starts with analyzing the sight and sound streams separately. Then
an audiovisual temporal stride decision module and a reliability estimation
module are introduced to exploit cross-modal temporal interaction. For
learning and evaluation, an existing dataset is repurposed and reorganized to
allow for repetition counting with sight and sound. We also introduce a
variant of this dataset for repetition counting under challenging vision
conditions. Experiments demonstrate the benefit of sound, as well as the other
introduced modules, for repetition counting. Our sight-only model already
outperforms the state-of-the-art by itself, when we add sound, results improve
notably, especially under harsh vision conditions. The code and datasets are
available at https://github.com/xiaobai1217/RepetitionCounting.

Figure 1: From sight and sound, as well as their cross-modal interaction, we
predict the number of repetitions for an (unknown) activity happening in a
video. This is especially beneficial in challenging vision conditions with
occlusions and low illumination.

##  1 Introduction

The goal of this paper is to count in video the repetitions of (unknown)
activities, like bouncing on a trampoline, slicing an onion or playing ping
pong. The computer vision solutions to this challenging problem have a long
tradition. Early work emphasized on repetitive motion estimation by Fourier
analysis, , [24, 4, 10, 36], and more recently by a continuous wavelet
transform [28, 29]. State-of-the-art solutions rely on convolutional neural
networks [21, 42, 11] and large-scale count-annotated datasets [42, 11] to
learn to predict the number of repetitions in a video. Albeit successful, all
existing works focus exclusively on the visual modality, and could fail in
poor sight conditions such as low illumination, occlusion, camera view
changes, . Different from existing works, we propose in this paper the first
repetitive activity counting method based on sight and sound.

Analyzing sound has recently proven advantageous in a variety of computer
vision challenges, such as representation learning by audio-visual
synchronization [19, 1, 3, 23], video captioning [39, 34, 25], sound source
localization [30, 27], to name a few. Correspondingly, several mechanisms for
fusing both modalities have been introduced. In works for action recognition
by previewing the audio track [20] and talking-face generation [38, 31], the
audio network usually works independently and the predictions guide the
inference process of the visual counterpart. In contrast, feature
multiplication and concatenation operations, as well as cross-modal attention
mechanisms, are widely adopted for fusion in tasks like audio-visual
synchronization [19, 27, 2] and video captioning [39, 34, 25]. We also combine
sight and sound, but observe that for some activities, like playing ping pong,
humans can count the number of repetitions by just listening. This gives us an
incentive that sound could be an important cue by itself. Hence, an
intelligent repetition counting system should be able to judge when the sight
condition is poor and therefore utilize complementary information from sound.

The first and foremost contribution of this paper is addressing video
repetition estimation from a new perspective based on not only the sight but
also the sound signals. As a second contribution, we propose an audiovisual
model with a sight and a sound stream, where each stream facilitates each
modality to predict the number of repetitions. As the repetition cycle lengths
may vary in different videos, we further propose a temporal stride decision
module to select the best sample rate for each video based on both visual and
audio features. Our reliability estimation module finally exploits cross-modal
temporal interaction to decide which modality-specific prediction is more
reliable. Since existing works focus on visual repetition counting only, our
third contribution entails two sight and sound datasets that we derive from
Countix [11] and VGGsound [8]. One of our datasets is for supervised learning
and evaluation and the other for assessing audiovisual counting in various
challenging vision conditions. Finally, our experiments demonstrate the
benefit of sound, as well as the other introduced network modules, for
repetition counting. Our sight-only model already outperforms the state-of-
the-art by itself, and when we add sound, the results improve further,
especially under harsh vision conditions. Before detailing our model, as
summarized in Figure 1, we first discuss related work.

##  2 Related Work

Repetitive activity counting. Existing approaches for repetition estimation in
video rely on visual content only. Early works [24, 4, 10, 36] compress the
motion field of video into one-dimensional signals and count repetitive
activities by Fourier analysis [24, 4, 10, 36], peak detection [33] or
singular value decomposition [9]. Burghouts and Geusebroek [5] propose a
spatiotemporal filter bank, which works online but needs manual adjustment.
Levy and Wolf [21] design a classification network able to learn from
synthetic data. Their network is designed to extract features from an input
video with a predefined sampling-rate, which cannot handle repetitions with
various period lengths. The synthetic dataset is also less suitable for usage
in the wild. All of the above methods assume the repetitions are periodic, so
they can cope with stationary situations only.

Recently, algorithms for non-stationary repetitive action counting have been
proposed. Runia _et al._ [28, 29] are the first to address non-stationary
situations. They leverage the wavelet transform based on the flow field and
collect a dataset containing $100$ videos including non-stationary
repetitions, but the videos do not contain an audio track. Zhang _et al._ [42]
propose a context-aware framework based on a 3D convolution network, and
introduce a new activity repetition counting dataset based on UCF101 [32].
While effective, the temporal length of every two repetitions is predicted by
an iterative refinement, making the approach less appealing from a
computational perspective. Dwibedi _et al._ [11] collect a large-scale dataset
from YouTube, named Countix, containing more than 6,000 videos with activity
repetition counts. Their method utilizes temporal self-similarity between
video frames for repetition estimation. It chooses the frame rate sampling the
input video by picking the one with the maximum periodicity classification
score. While appealing, such a rate selection scheme is not optimal for
accurate counting, as it is prone to select high frame rates leading to
omissions.

Different from all these existing methods, we propose to address repetitive
activity counting by sight and sound. Our network contains a temporal stride
decision module able to choose the most suitable frame rate for counting,
based on features from both modalities. To facilitate our investigation, we
reorganize and supplement the Countix [11] dataset, to arrive at two
audiovisual datasets for repetitive activity counting by sight and sound.

Learning by sight and sound. Many have demonstrated the benefit of audio
signals for various computer vision challenges, , action recognition [20, 14],
audiovisual event localization [40] and self-supervised learning [19, 1, 3,
23]. As processing audio signals is much faster than video frames, both Korbar
_et al._ [20] and Gao _et al._ [14] reduce the computational cost by
previewing the audio track for video analysis. However, the Kinetics-Sound [1]
used for training and evaluation in [1, 20, 14] is simply formed by all the
videos in the Kinetics dataset [6] covering 34 human action classes, which are
potentially manifested visually and aurally. As a result, the audio track of
many videos is full of background music, which introduces noise for training
and fair evaluation. Recent talking-face generation works exploit sound for
creating photo-realistic videos [38, 31]. While audio features are used to
generate expression parameters in [31], Wang  _et al._ [38] to map the audio
to lip movements. There are also numerous works [18, 37, 19, 7, 30, 41] that
consider the interaction between both modalities. Some simply integrate
features by concatenation for tasks like saliency detection [37] and self-
supervised learning [19, 27, 2]. Cartas _et al._ [7] combine multi-modal
predictions by averaging or training a fully connected layer independently,
for egocentric action recognition. Works for sound source localization [30,
27] and separation [41, 13, 12, 43] also commonly generate cross-modal
attention maps. Hu _et al._ [17] use audio features to modulate the visual
features for more accurate crowd counting.

Exploiting sound for activity repetition counting is still unexplored and to
obtain audiovisual datasets facilitating our research, we also select videos
with usable audio track from a large scale visual-only dataset manually to
reduce label noise. To cope with various ‘in the wild’ conditions, we further
introduce a novel scheme to explicitly estimate the reliability of the
predictions from sight and sound.

Figure 2: Proposed class-agnostic activity repetition counting model. Our
model contains four components (1) sight stream, (2) sound stream, (3)
temporal stride decision module and (4) reliability estimation module. Both
streams contain a backbone network that outputs a modality-specific counting
prediction. The temporal stride decision module takes audio and visual
features as inputs and outputs the frame sample rate for the input video.
Finally, the reliability estimation module decides what prediction from which
modality to use.

##  3 Model

Given a video, containing a visual stream and its corresponding audio stream,
our goal is to count the number of repetitions of (unknown) activities
happening in the content. To achieve this, we propose a model that contains
four modules. (i) The sight stream adopts a 3D convolutional network as the
backbone. It takes video clips as inputs and outputs the counting result for
each clip. (ii) For the sound stream, we rely on a 2D convolutional network,
which takes the sound spectrogram generated by the short-time Fourier
transform as input and outputs the counting result in the same way as the
sight stream. (iii) A temporal stride decision module is designed to select
the best temporal stride per video for the sight stream based on both visual
and audio features. (iv) Finally, the reliability estimation module decides
what prediction to use. The overall model is summarized in Figure 2 and
detailed per module next.

###  3.1 Repetition Counting by Sight

The sight stream uses an S3D [35] architecture and the final classification
layer is replaced by two separate fully connected layers, as shown in Figure
2. Given a video clip $V_{i}$ of size $T{\times}H{\times}W{\times}3$, visual
features are extracted with the following equation:

$\mathbf{v}_{i,\textit{feat}}=\mathcal{V}_{\textit{CNN}}(V_{i}),$ (1)

where $\mathbf{v}_{i,\textit{feat}}\in\mathbb{R}^{512}$. Intuitively, a fully
connected layer with one output unit could suffice to output the counting
result. However, this setting leads to inferior repetition counts since
different types of movements should not be counted in the same way, and each
action class cannot be simply regarded as one unique repetition class. For
example, different videos of doing aerobics contain various repetitive
motions, while bouncing on a bouncy castle or a trampoline contains similar
movements despite belonging to different action classes. Therefore, in our
work, two fully connected layers work in tandem, with one $f_{v}^{1}$
outputting the counting result of each repetition class and the other one
$f_{v}^{2}$ classifying which repetition class the input belongs to:

$\begin{split}&\mathbf{C}_{i,v}^{{}^{\prime}}=f_{v}^{1}(\mathbf{v}_{i,\textit{feat}}),\mathbf{C}_{i,v}^{{}^{\prime}}\in\mathbb{R}^{P},\\\
&\mathbf{T}_{i,v}=\textit{softmax}(f_{v}^{2}(\mathbf{v}_{i,\textit{feat}})),\mathbf{T}_{i,v}\in\mathbb{R}^{P},\end{split}$
(2)

where $P$ is the number of repetition classes,
$\mathbf{C}_{i,v}^{{}^{\prime}}$ is the counting result of each class, and
$\mathbf{T}_{i,v}$ is the classification result by the softmax operation.
Here, we assume that there are roughly $P$ classes of repetitive motion
patterns, and the network learns to classify the training videos into those
$P$ classes automatically during training. Then the final counting result
$\mathbf{C}_{i,v}$ from the visual content is obtained by:

$C_{i,v}=\sum_{k=1}^{P}\mathbf{C}_{i,v}^{{}^{\prime}}(k)\mathbf{T}_{i,v}(k).$
(3)

For training the repetition counting, we define the loss function as follows:

$L^{{}^{\prime}}=\frac{1}{N}\sum_{i=1}^{N}L_{2}(C_{i,v},l_{i})+\lambda_{1}^{v}\frac{|C_{i,v}-l_{i}|}{l_{i}},$
(4)

where $N$ is the batch size, $L_{2}$ is the L2 loss [26], $l_{i}$ is the
groundtruth count label of the $i$th sample and $\lambda_{1}^{v}$ is a
hyperparameter for the second term. Note that when using the L2 loss only, the
model tends to predict samples with groundtruth counts of large values
accurately, due to higher losses, while for videos with a few repetitions, the
predicted counts tend to be unreliable. Therefore, we add a second term here
to let the model pay more attention to such data.

Besides, we expect the output units of $f_{v}^{2}$ to focus on different
repetition classes given various videos. However, without constraint,
$f_{v}^{2}$ could simply output a high response via the same unit. To avoid
such degenerated cases, we add a diversity loss [22] based on the cosine
similarity:

$L_{i,v}^{\textit{div}}=\sum_{q=1}^{P-1}\sum_{j=q+1}^{P}\frac{T_{i,v}^{q}\cdot
T_{i,v}^{j}}{||T_{i,v}^{q}||||T_{i,v}^{j}||},$ (5)

where $T_{v}^{q}$ and $T_{v}^{j}$ are the $q$th and $j$th units of the
classification outputs. By minimizing such a diversity loss, the output
$T_{v}$ in the same batch are encouraged to produce different activations on
different types of repetitive motions. Then the total loss function is:

$L_{v}=\frac{1}{N}\sum_{i=1}^{N}L_{2}(C_{i,v},l_{i})+\lambda_{1}^{v}\frac{|C_{i,v}-l_{i}|}{l_{i}}+\lambda_{2}^{v}L_{i,v}^{\textit{div}},$
(6)

where $\lambda_{2}^{v}$ is a hyperparameter.

###  3.2 Repetition Counting by Sound

The sound stream adopts a ResNet-18 [15] as the backbone. Following [16, 8],
we first transform the raw audio clip into a spectrogram and then divide it
into a series of $257{\times}500$ spectrograms, which become the inputs to our
network. Similar to the sight stream, we also replace the final classification
layer by two separate fully connected layers, with one classifying the input
and the other one outputting the corresponding counting result of each
repetition class. We use the same loss function as the sight stream:

$L_{a}=\frac{1}{N}\sum_{i=1}^{N}L_{2}(C_{i,a},l_{i})+\lambda_{1}^{a}\frac{|C_{i,a}-l_{i}|}{l_{i}}+\lambda_{2}^{a}L_{i,a}^{\textit{div}},$
(7)

where $C_{i,a}$ is the counting result from the audio track, and
$\lambda_{1}^{a}$ and $\lambda_{2}^{a}$ are hyperparameters.

###  3.3 Temporal Stride Decision

Repetitions have various period lengths for different videos. For the sound
stream, we can simply resize the spectrogram along the time dimension to
ensure each $257{\times}500$ segment to have at least two repetitions.
However, for the sight stream, we cannot roughly resize the video frames along
the time dimension. Therefore, for each video, we need to use a specific
temporal stride (frame rate) to form video clips of $T$ frames as the inputs.
This is important as video clips with small temporal strides may fail to
include at least two repetitions, while too large temporal strides lead the
network to ignore some repetitions. Therefore, we add an additional temporal
stride decision module to select the best temporal stride for each video. It
has two parallel residual blocks, processing visual and audio features from
the third residual block of the two streams, with the same structure as those
of the backbones. Then we concat the output features (as shown in Figure 2)
and send them into a fully connected layer, which outputs a single unit
representing the score of the current temporal stride. We use a max-margin
ranking loss for training this module:

$L_{s}=\frac{1}{N}\sum_{i=1}^{N}\textit{max}(0,s_{i}^{-}-s_{i}^{+}+m),$ (8)

where $m$ is the margin, $s_{i}^{-}$ and $s_{i}^{+}$ are the scores from
negative and positive strides. During inference, we send a series of clips
from the same video with different strides into the network, and select the
stride with the highest score.

Training details. For each training video, the trained visual model predicts
the counting result with a series of temporal strides,
$s={1,...,S_{k},...,S_{K}}$, where $S_{K}$ is the maximum stride we use. Then
we can obtain corresponding predictions ${C_{i,v}^{1},...,C_{i,v}^{S_{K}}}$.
First, we select the temporal strides that cover less than two repetitions as
negative strides. Then, we choose the smallest stride that is enough to
contain at least two repetitions as the positive temporal stride $S^{*}$.
Correspondingly, for the remaining strides, we quantitatively compute their
prediction deviations from the prediction of the positive stride by:

$\delta_{n}=\frac{C_{i,v}^{*}-C_{i,v}^{k}}{C_{i,v}^{*}},$ (9)

where $C_{i,v}^{*}$ and $C_{i,v}^{k}$ are the counting predictions from the
best stride and a selected stride, and $\delta_{n}$ is the computed deviation.
Finally, we select strides with $\delta_{n}{>}\theta_{s}$ ($\theta_{s}$ is a
predefined threshold) as negative strides, since for these strides, the
network begins to omit certain repetitions. During training, for each video,
its $S^{*}$ is used to form a positive video clip outputting $s^{+}_{i}$,
while we randomly select one from the negative strides to generate the clip
outputting $s^{-}_{i}$.

###  3.4 Reliability Estimation

Depending on the sensory video recording conditions, the reliability of the
sight and sound predictions may vary. To compensate for this variability, we
introduce a reliability estimation module to decide what prediction from which
modality is more reliable for the current input. As shown in Figure 2, it
contains one residual block for processing the audio feature and one fully
connected layer taking features from both modalities as inputs. The output is
a single unit processed by a sigmoid function and represents the confidence
$\gamma$ of the audio modality. Correspondingly, the confidence of the visual
modality is $1-\gamma$. Then the final counting result is obtained by:

$C_{i}=C_{i,v}*(1-\gamma)+C_{i,a}*\gamma.$ (10)

As $C_{i}$ is expected to be close to the groundtruth counting label, the loss
function we use for training is:

$L_{r}=\frac{1}{N}\sum_{i=1}^{N}\frac{|C_{i}-l_{i}|}{l_{i}}.$ (11)

Training details. During training, for each video, the accuracy of $C_{i,v}$
and $C_{i,a}$ in Eq. 10 is expected to indicate the reliability of the
corresponding modality content. To get $C_{i,v}$ and $C_{i,a}$, one simple
approach is to directly use the predictions from the trained models. However,
we empirically observe that such a manner suffers from severe over-fitting,
since the learned models could overfit to recent training samples with poor
modality content. As a result, the obtained $C_{i,v}$ and $C_{i,a}$ cannot
represent the reliability effectively. However, for one modality of a video,
if the corresponding model predicts inaccurately most of the time during
training, then intuitively the content may be too noisy or poor for learning.
Therefore, instead of $C_{i,v}$ and $C_{i,a}$ from the final models, we use
the average prediction of each stream at different training stages. Here, we
take the sight stream as an example. After each training epoch, if the loss
computed by
$\frac{1}{\mathcal{N}}\sum_{i=1}^{\mathcal{N}}\frac{|C_{i,v}-l_{i}|}{l_{i}}$,
$\mathcal{N}$ is the number of videos, over the validation set is below a
threshold $\theta_{r}^{v}$ (current model parameters have competitive
performance), we record the predictions of the model over the training videos.
Once the training is finished, we can obtain the average prediction (empirical
prediction) of each training video by recordings correspondingly. The
empirical prediction of the sound stream is computed in the same way, with a
threshold $\theta_{r}^{a}$ for the validation loss. Finally, our reliability
estimation module uses those empirical predictions for Eq. 10 during training
and learns to switch between sight and sound models for effective late fusion.

##  4 Experimental Setup

###  4.1 Datasets

Existing datasets for repetition counting [21, 28, 42, 11] focus on counting
by visual content only. Thus, the videos have either no audio information at
all, or at best a few only. Nonetheless, we evaluate our (sight) model on the
two largest existing visual-only datasets, UCFRep and Countix. As we focus on
counting by sight and sound, we also repurpose, reorganize and supplement one
of those two datasets.

UCFRep. The UCFRep dataset by Zhang _et al._ [42] contains 526 videos of 23
categories selected from UCF101 [32], a widely used benchmark for action
recognition, with 420 and 106 videos for training and validation.
Particularly, it has boundary annotations for each repetition along the time
dimension. However, the large majority of videos do not have any associated
audio track.

Vision challenge | Number of videos
---|---
Camera viewpoint changes | 69
Cluttered background | 36
Low illumination | 13
Fast motion | 31
Disappearing activity | 25
Scale variation | 24
Low resolution | 29
Overall | 214
Table 1: Extreme Countix-AV dataset statistics.

Countix. The Countix dataset by Dwibedi _et al._ [11] serves as the largest
dataset for video repetition counting in the wild. It is a subset of the
Kinetics [6] dataset annotated with segments of repeated actions and
corresponding counts. The dataset contains 8,757 videos in total of 45
categories, with 4,588, 1,450 and 2,719 for training, validation and testing.

Countix-AV. We repurpose and reorganize the Countix dataset for our goal of
counting repetitive activities by sight and sound. We first select 19
categories for which the repetitive action has a clear sound, such as
clapping, playing tennis, etc. As several videos contain artificially added
background music or have no audio track at all, they are less suited for
assessing the impact of the sound-stream, and the sight and sound combination.
Therefore, we manually filter out such videos so that the videos preserved are
guaranteed to contain the environmental sound only, be it they may also
include realistic background noise or unclear repetition sounds. This results
in the Countix-AV dataset consisting of 1,863 videos, with 987, 311 and 565
for training, validation and testing. We maintain the original count
annotations from Countix and keep the same split (training, validation, or
testing) for each video. The dataset is detailed in the appendix.

Figure 3: Example videos from the Extreme Countix-AV dataset. From top-row to
bottom-row are videos with scale variation, camera viewpoint change, fast
motion, a disappearing activity and low resolution, with the numbers in
colored boxes indicating counting results and the corresponding groundtruth.

Extreme Countix-AV. For most videos in Countix-AV, the ongoing action is both
visible and audible. As the audio signal is expected to play a vital role when
the visual content is not reliable, we further introduce the Extreme Countix-
AV dataset to quantitatively evaluate the benefits brought by audiovisual
counting under various extreme sight conditions. For data collection, we first
define 7 vision challenges, according to which we select videos. Then, 156
videos from Countix-AV are selected and to enlarge this selection, we choose
and label another 58 videos from the VGGSound dataset by Chen  _et al._ [8].
The overall dataset and challenges are summarized in Table 1, with video
examples depicted in Figure 3. More details are provided in the appendix.

###  4.2 Evaluation Criteria

We adopt the same evaluation metrics as previous works [24, 28, 29, 42, 11],
the mean absolute error (MAE) and off-by-one accuracy (OBO), defined as
follows:

$\textit{MAE}=\frac{1}{\mathcal{N}}\sum_{i=1}^{\mathcal{N}}\frac{|\hat{c_{i}}-l_{i}|}{l_{i}},$
(12)
$\textit{OBO}=\frac{1}{\mathcal{N}}\sum_{i=1}^{\mathcal{N}}[|\hat{c_{i}}-l_{i}|\leq
1],$ (13)

where $\mathcal{N}$ is the total number of videos, $\hat{c_{i}}$ is the model
prediction of the $i$th video and $l_{i}$ is the groundtruth. Specifically,
for the Extreme Countix-AV, we report MAE only, as those videos have more
repetitions than other datasets and OBO cannot evaluate the performance
effectively.

###  4.3 Implementation Details

We implement our method using PyTorch with two NVIDIA GTX1080Ti GPUs. We
provide training details below and inference procedures in the appendix.

Sight and sound models. For the sight stream, all input video frames are
resized to $112{\times}112$, and we form each clip of $64$ frames with its
temporal stride $S^{*}$ defined in Section 3.3. We initialize the backbone
with weights from a Kinetics [6] pre-trained checkpoint. The training of the
sight model is on the original Countix training set [11] and takes 8 epochs by
SGD with a fixed learning rate of $10^{-4}$ and batch size of $8$.
$\lambda^{1}_{v}$, $\lambda_{v}^{2}$, $\lambda_{a}^{1}$ and $\lambda_{a}^{2}$
are all set to $10$. The sound model is trained with the same setting as the
sight stream but using our Countix-AV training set for 20 epochs.

Temporal stride decision module. The training takes $5$ epochs with a learning
rate of $10^{-3}$ after obtaining the negative strides of each video. Here, we
provide two options. First, it can be trained with the visual modality only,
without the audio feature, using the original Countix [11] dataset so that the
sight model can work independently. The other option is our full setting (as
shown in Figure 2) trained on Countix-AV with the audio modality. Margin $m$
in Eq. 8 is set to 2.9 and $S_{K}$ is set to 8. In experiments, we find the
value of $\theta_{s}$ does not influence results too much, and
$\theta_{s}{=}0.29$ works best (see appendix for ablation).

Reliability estimation module. We first collect the empirical predictions
before training, and $\theta_{r}^{v}$ and $\theta_{r}^{a}$ are set to 0.36 and
0.40. Then, this module is trained on Countix-AV for 20 epochs with a learning
rate of $10^{-4}$ and batch size of 8.

##  5 Results

Benefit of model components. Our model consists of four main components: the
sight and sound counting models, the temporal stride decision module and the
reliability estimation module. We evaluate the performance of several network
variants on Countix-AV to validate the efficacy of each component. The results
are shown in Table 2. Note that for “Sight stream” in the first row, its
temporal stride decision module takes the visual modality only as input. In
isolation, the sight stream performs better than the sound stream. When we
incorporate audio features into the temporal stride decision module, denoted
as “Sight with temporal stride”, the MAE of the sight stream is further
reduced from 0.331 to 0.314. This demonstrates the audio signals provide
useful temporal information. Simply averaging the predictions from both
modalities results in higher accuracy than either modality alone. However,
when we further reweigh the predictions by our reliability estimation module,
we obtain the best result with an MAE of 0.291 and an OBO of 0.479.

Model components | MAE $\downarrow$ | OBO $\uparrow$
---|---|---
Sight stream | 0.331 | 0.431
Sound stream | 0.375 | 0.377
Sight with temporal stride | 0.314 | 0.459
Averaging predictions | 0.300 | 0.439
Full sight and sound model | 0.291 | 0.479
Table 2: Benefit of model components on Countix-AV. All modules matter and
reliability estimation is preferred over simple averaging of sight and sound
predictions.

Influence of loss function terms. The loss function used for training the
visual and audio models consists of three terms. We perform an ablation on
different term combinations to further understand their contributions. Results
in Table 3 indicate both $L_{\textit{div}}$ and $L_{\textit{mae}}$ reduce the
counting error, especially on the sound stream. We observe adding
$L_{\textit{div}}$ contributes to performance improvements because it allows
the units in the classification layer to affect each other during training. It
prevents this layer from converging to a degenerated solution, in which all
videos are assigned to the same repetition class. Combining all loss terms
during training produces best results for both modalities.

| Sight | Sound
---|---|---
Loss term | MAE $\downarrow$ | OBO $\uparrow$ | MAE $\downarrow$ | OBO $\uparrow$
$L_{2}$ | 0.371 | 0.424 | 0.471 | 0.338
$L_{2}+L_{\textit{div}}$ | 0.324 | 0.478 | 0.410 | 0.343
$L_{\textit{div}}+L_{\textit{mae}}$ | 0.356 | 0.446 | 0.447 | 0.310
$L_{2}+L_{\textit{mae}}$ | 0.370 | 0.421 | 0.426 | 0.340
$L_{2}+L_{\textit{div}}+L_{\textit{mae}}$ | 0.314 | 0.498 | 0.375 | 0.377

Table 3: Influence of loss function terms on Countix-AV. Each term lowers the
counting error, while $L_{\textit{div}}$ is indispensable.  Figure 4: Effect
of repetition classes. The performances of both streams fluctuates slightly
when $P$ is large enough, and achieves the best result when $P{=}41$ (sight)
and $P{=}43$ (sound).

Effect of repetition classes. As detailed in Eq. 2 and 3, our counting models
for both modalities involve a parameter $P$, the number of repetition classes.
We evaluate its effect on both the sight and sound models. To this end, we fix
the backbone architectures and train them by setting $P$ from $1$ to $50$ with
the same setting described in Section 4.3. Here, we do not include cross-modal
interactions, the sight and sound models are trained and evaluated on Countix
and our Countix-AV datasets, separately. The results are shown in Figure 4.
The performances of both models are inferior when $P$ has a low value,
demonstrating the need to model various types of repetitions. The performance
fluctuates only slightly when $P$ is between 20 and 50. We observe that there
is a spike when $P{=}36$ for both streams, at where the networks converge to
local minimum and this can be eliminated by more training epochs. In
particular, the sight and sound models obtain their best results at $P{=}41$
and $P{=}43$, the values we use for all other experiments, while Countix [11]
and our Countix-AV cover $45$ and $19$ action categories. Thus, the repetition
types do not simply correspond to the number of action categories. For
instance, in Figure 3, the second and the last rows have similar repetition
classification distributions, while dissimilar to the fourth row. We show more
examples in the supplementary material.

| UCFRep | Countix | Countix-AV | Extreme Countix-AV
---|---|---|---|---
| MAE $\downarrow$ | OBO $\uparrow$ | MAE $\downarrow$ | OBO $\uparrow$ | MAE $\downarrow$ | OBO $\uparrow$ | MAE $\downarrow$
Baseline† | 0.474 | 0.371 | 0.525 | 0.289 | 0.503 | 0.269 | 0.620
Dwibedi _et al._ [11] | - | - | 0.364 | 0.697 | - | - | -
Levy and Wolf* [21] | 0.286 | 0.680 | - | - | - | - | -
Zhang  _et al._ [42] | 0.147 | 0.790 | - | - | - | - | -
This paper: Sight | 0.143 | 0.800 | 0.314 | 0.498 | 0.331 | 0.431 | 0.392
This paper: Sound | - | - | 0.793 | 0.331 | 0.375 | 0.377 | 0.351
This paper: Sight & Sound | - | - | 0.307 | 0.511 | 0.291 | 0.479 | 0.329

† Sight-only model, pre-trained on Countix, publicly released by authors of
[11].
* Sight-only model [21], reproduced and pre-trained on UCFRep by authors of [42].

Table 5: Comparison with state-of-the-art. Our sight model outperforms two
recent state-of-the-art repetition counting algorithms in terms of MAE, while
combining sight and sound shows further benefit in reducing counting error,
especially in visually challenging settings.

Vision challenge | Sight | Sound | Sight & Sound
---|---|---|---
Camera viewpoint changes | 0.384 | 0.376 | 0.331
Cluttered background | 0.342 | 0.337 | 0.307
Low illumination | 0.325 | 0.269 | 0.310
Fast motion | 0.528 | 0.311 | 0.383
Disappearing activity | 0.413 | 0.373 | 0.339
Scale variation | 0.332 | 0.386 | 0.308
Low resolution | 0.348 | 0.303 | 0.294
Overall | 0.392 | 0.351 | 0.329

Table 4: MAE metric on hard cases in repetition counting. Sound tends to have
lower MAE than sight. Combining sight and sound always outperforms sight only.

Effectiveness of temporal stride module.  We also considered fixed temporal
strides for the sight-stream on Countix. MAE varies from 0.607 to 0.378 (see
appendix). Our temporal stride decision module obtains a better 0.314 MAE.

Hard cases in repetition counting. To quantitatively evaluate the contribution
of sound information and how sensitive the sight stream is under different
visually challenging environment, we test the sight, sound and full sight and
sound model separately on the Extreme Countix-AV dataset. The results are
listed in Table 4. Compared to the performance on Countix-AV dataset, which is
dominated by videos with normal sight conditions, the MAE of the sight stream
increases considerably. In contrast, the sound stream performs stably and is
superior under visually challenging circumstances as expected, except for the
scale variation challenge. This means that changes in image quality can easily
affect the sight stream. Especially when activities are moving fast or
disappearing due to occlusions, the value of the sound stream is prevalent.
Combining sight and sound is always better than sight only, resulting in
considerable MAE reductions on videos with camera view changes, disappearing
activities, scale variation and cluttered background. For scale variation, the
sound stream does not perform competitively compared to the sight stream,
while the fused results do improve over the sight stream. This again indicates
the effectiveness of our reliability estimation module. For low illumination
and fast motion, the sight stream performs poor compared to the sound stream,
and the combination cannot improve over the sound stream only. Overall, the
integration of sight and sound is better than unimodal models and more stable
when the imaging quality varies.

Comparison with state-of-the-art.  We compare our method with two recent
state-of-the-art (vision-only) repetition counting models [11, 42] and one
early work by Levy and Wolf [21], as shown in Table 5. As the complete code of
[11] is unavailable, we also report the performance of their released (vision-
only) model as a baseline. Our sight-only stream already outperforms Dwibedi
_et al._ [11] on their original Countix dataset with respect to the MAE
metric, and achieves competitive performance on UCFRep [42]. Note the work by
Zhang  _et al._ [42] needs the training videos to have boundary annotations
for each repetition, which are not provided for Countix [11]. As Countix is
dominated by “silent” repetitions, our sound-only model performs inferior
compared to the sight-only model. Nevertheless, our full sight and sound model
sets a new state-of-the-art on all three Countix datasets in MAE and surpasses
the released model of [11] by a large margin. Therefore, we conclude that when
the original sound track of the video is available, audiovisual repetition
counting is superior to sight-only models.

##  6 Conclusion

We propose to count repetitive activities in video by sight and sound using a
novel audiovisual model. To facilitate further progress, we repurpose and
reorganize an existing counting dataset for sight and sound analysis.
Experiments show that sound can play a vital role, and combining both sight
and sound with cross-modal temporal interaction is beneficial. Using sight
only we already outperform the state-of-the-art in terms of MAE. When adding
sound, results improve further, especially under harsh vision conditions.

## References

  * [1] Relja Arandjelović and Andrew Zisserman.  Look, listen and learn.  In ICCV, 2017.
  * [2] Relja Arandjelović and Andrew Zisserman.  Objects that sound.  In ECCV, 2018.
  * [3] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  In NeurIPS, 2016.
  * [4] Ousman Azy and Narendra Ahuja.  Segmentation of periodically moving objects.  In ICPR, 2008.
  * [5] Gertjan J Burghouts and Jan-Mark Geusebroek.  Quasi-periodic spatiotemporal filtering.  TIP, 15(6):1572–1582, 2006.
  * [6] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, 2017.
  * [7] Alejandro Cartas, Jordi Luque, Petia Radeva, Carlos Segura, and Mariella Dimiccoli.  Seeing and hearing egocentric actions: How much can we learn?  In ICCVW, 2019.
  * [8] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman.  VGGSound: a large-scale audio-visual dataset.  In ICASSP, 2020.
  * [9] Dmitry Chetverikov and Sándor Fazekas.  On motion periodicity of dynamic textures.  In BMVC, 2006.
  * [10] Ross Cutler and Larry S. Davis.  Robust real-time periodic motion detection, analysis, and applications.  TPAMI, 22(8):781–796, 2000.
  * [11] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  Counting out time: Class agnostic video repetition counting in the wild.  In CVPR, 2020.
  * [12] Ruohan Gao, Rogerio Feris, and Kristen Grauman.  Learning to separate object sounds by watching unlabeled video.  In ECCV, 2018.
  * [13] Ruohan Gao and Kristen Grauman.  Co-separating sounds of visual objects.  In ICCV, 2019.
  * [14] Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani.  Listen to look: Action recognition by previewing audio.  In CVPR, 2020.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [16] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, Malcolm Slaney, Ron J Weiss, and Kevin Wilson.  CNN architectures for large-scale audio classification.  In ICASSP, 2017.
  * [17] Di Hu, Lichao Mou, Qingzhong Wang, Junyu Gao, Yuansheng Hua, Dejing Dou, and Xiao Xiang Zhu.  Ambient sound helps: Audiovisual crowd counting in extreme conditions.  arXiv preprint arXiv:2005.07097, 2020.
  * [18] Hamid Reza Vaezi Joze, Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida.  Mmtm: multimodal transfer module for cnn fusion.  In CVPR, 2020.
  * [19] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  In NeurIPS, 2018.
  * [20] Bruno Korbar, Du Tran, and Lorenzo Torresani.  SCSampler: Sampling salient clips from video for efficient action recognition.  In ICCV, 2019.
  * [21] Ofir Levy and Lior Wolf.  Live repetition counting.  In ICCV, 2015.
  * [22] Daochang Liu, Tingting Jiang, and Yizhou Wang.  Completeness modeling and context separation for weakly supervised temporal action localization.  In CVPR, 2019.
  * [23] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba.  Ambient sound provides supervision for visual learning.  In ECCV, 2016.
  * [24] Erik Pogalin, Arnold WM Smeulders, and Andrew HC Thean.  Visual quasi-periodicity.  In CVPR, 2008.
  * [25] Tanzila Rahman, Bicheng Xu, and Leonid Sigal.  Watch, listen and tell: Multi-modal weakly supervised dense event captioning.  In ICCV, 2019.
  * [26] Rasmus Rothe, Radu Timofte, and Luc Van Gool.  Deep expectation of real and apparent age from a single image without facial landmarks.  International Journal of Computer Vision, 126(2):144–157, 2018\.
  * [27] Andrew Rouditchenko, Hang Zhao, Chuang Gan, Josh McDermott, and Antonio Torralba.  Self-supervised audio-visual co-segmentation.  In ICASSP, 2019.
  * [28] Tom FH Runia, Cees GM Snoek, and Arnold WM Smeulders.  Real-world repetition estimation by div, grad and curl.  In CVPR, 2018.
  * [29] Tom FH Runia, Cees GM Snoek, and Arnold WM Smeulders.  Repetition estimation.  IJCV, 127(9):1361–1383, 2019.
  * [30] Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon.  Learning to localize sound sources in visual scenes: Analysis and applications.  arXiv preprint arXiv:1911.09649, 2019.
  * [31] Linsen Song, Wayne Wu, Chen Qian, Ran He, and Chen Change Loy.  Everybody’s talkin’: Let me talk as you want.  arXiv preprint arXiv:2001.05201, 2020.
  * [32] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  UCF101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [33] Ashwin Thangali and Stan Sclaroff.  Periodic motion detection and estimation via space-time sampling.  In WACV, 2005.
  * [34] Yapeng Tian, Chenxiao Guan, Justin Goodman, Marc Moore, and Chenliang Xu.  An attempt towards interpretable audio-visual video captioning.  In ICCV, 2019.
  * [35] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In CVPR, 2018.
  * [36] Ping-Sing Tsai, Mubarak Shah, Katharine Keiter, and Takis Kasparis.  Cyclic motion detection for motion based recognition.  Pattern recognition, 27(12):1591–1603, 1994.
  * [37] Antigoni Tsiami, Petros Koutras, and Petros Maragos.  STAViS: Spatio-temporal audiovisual saliency network.  In CVPR, 2020.
  * [38] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian, Ran He, Yu Qiao, and Chen Change Loy.  MEAD: A large-scale audio-visual dataset for emotional talking-face generation.  In ECCV, 2020.
  * [39] Xin Wang, Yuan-Fang Wang, and William Yang Wang.  Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning.  arXiv preprint arXiv:1804.05448, 2018.
  * [40] Yu Wu, Linchao Zhu, Yan Yan, and Yi Yang.  Dual attention matching for audio-visual event localization.  In ICCV, 2019.
  * [41] Xudong Xu, Bo Dai, and Dahua Lin.  Recursive visual sound separation using minus-plus net.  In ICCV, 2019.
  * [42] Huaidong Zhang, Xuemiao Xu, Guoqiang Han, and Shengfeng He.  Context-aware and scale-insensitive temporal repetition counting.  In CVPR, 2020.
  * [43] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba.  The sound of motions.  In ICCV, 2019.

## Overview of Appendix

| Number of videos | Average count groundtruth
---|---|---
Action class | Train | Validation | Test | Total | Train | Validation | Test | Average
battle rope training | 57 | 17 | 41 | 115 | 14 | 10 | 6 | 11
bouncing ball (not juggling) | 63 | 25 | 41 | 129 | 7 | 9 | 7 | 7
bouncing on trampoline | 22 | 7 | 15 | 44 | 6 | 6 | 5 | 6
clapping | 16 | 14 | 37 | 67 | 7 | 7 | 9 | 8
gymnastics tumbling | 27 | 6 | 15 | 48 | 4 | 3 | 4 | 4
juggling soccer ball | 65 | 23 | 9 | 97 | 11 | 11 | 9 | 11
jumping jacks | 31 | 16 | 24 | 71 | 7 | 5 | 5 | 6
mountain climber (exercise) | 37 | 12 | 22 | 71 | 10 | 9 | 10 | 10
planing wood | 37 | 16 | 25 | 78 | 5 | 6 | 5 | 5
playing ping pong | 79 | 25 | 34 | 138 | 3 | 3 | 3 | 3
playing tennis | 42 | 11 | 24 | 77 | 3 | 3 | 3 | 3
running on treadmill | 51 | 13 | 24 | 88 | 13 | 13 | 10 | 12
sawing wood | 55 | 12 | 41 | 108 | 9 | 7 | 7 | 8
skipping rope | 62 | 24 | 36 | 122 | 12 | 11 | 9 | 11
slicing onion | 110 | 40 | 66 | 216 | 12 | 13 | 11 | 12
swimming | 80 | 13 | 32 | 125 | 5 | 5 | 6 | 5
tapping pen | 38 | 12 | 24 | 74 | 19 | 25 | 24 | 22
using a wrench | 22 | 3 | 9 | 34 | 5 | 3 | 5 | 5
using a sledge hammer | 93 | 22 | 43 | 158 | 5 | 5 | 5 | 5
Total | 987 | 311 | 562 | 1860 | - | - | - | -
Table 6: Countix-AV dataset statistics. Note our model does not use the action
class labels.

The inference procedures of our model are first illustrated in Appendix A.
Then we present additional details of our Countix-AV and Extreme Countix-AV
datasets in Appendix B and Appendix C. The effect of hyperparameter
$\theta_{s}$ and margin $m$ in our temporal stride decision module is
evaluated in Appendix E, as well as the comparison with fixed temporal
strides. For the reliability estimation module, we also study the effect of
two hyperparameters $\theta_{r}^{v}$ and $\theta_{r}^{a}$ and compare with two
alternatives in Appendix F. In Appendix G, we analyze the performance of
training our model with action class supervision and we demonstrate the
implementation details of our sight model on the UCFRep [42] dataset in
Appendix H. We finally explain the videos provided in the supplementary
material in Appendix I and Appendix J.

##  Appendix A Inference Procedures

For each video, we first divide it into video clips with temporal strides of
$\\{1,2,...,S_{K}\\}$ and their corresponding audio signals, which are sent
into the networks simultaneously. In experiments, we find $S_{K}{=}5$ is
enough for the used datasets, and it can be enlarged for situations where the
action takes place slowly. Then, we choose the stride with the maximum score
outputted by the temporal stride decision module to resample the video and
preserve the estimated reliability score of the selected stride for later
fusion. In the end, after obtaining the counting results from both streams,
the final prediction of our model is computed by Eq. 10.

##  Appendix B Countix-AV Dataset Statistics

The Countix-AV dataset is a subset of Countix [11] and the videos come from
YouTube. It includes a total number of 19 classes for which the repetitive
actions have a clear sound. The statistics per class are summarized in Table
6, including the number of videos per train, val and test fold, as well as the
average count ground truth per class and fold. Some example videos are
included in the supplementary material (“Learned repetition classes/Sound”).

##  Appendix C Extreme Countix-AV Dataset Details

The Extreme Countix-AV contains 214 videos in total, with 156 from Countix-AV
and 58 from the VGGSound dataset [8]. We define 7 vision challenges to collect
videos. First, we manually check every video and choose those that have camera
viewpoint changes, disappearing activity and scale variation based on our
visual observation. Then, for the cluttered background challenge, we also
manually select the videos in which there are multiple persons appearing
simultaneously while only one person is doing the repetitive actions or the
object conducting repetitive activity is too small and hard to be
distinguished (, some videos of bouncing ball). To collect videos captured in
low illumination, we compute the average pixel intensity of each video, and
add those with values below 100 (in the range of 0 and 255) into our dataset.
For the fast motion challenge, we compute the average period length of each
video according to the counting annotations, and find the videos with the
average period length shorter than 3 frames. Finally, together the videos of
those 7 challenges form our Extreme Countix-AV dataset.

Figure 5: Performance on videos with more repetitions. Figure 6: Performance
on videos with a few repetitions.

##  Appendix D Effect of L1 and L2 Loss Terms

In Eq. 6 and Eq. 7, both L1 and L2 terms are used in the loss functions of the
sight and the sound streams to balance accuracy on small and large counts. To
illustrate their effectiveness, we perform an ablation on different term
combinations and report the sight-only model performance on Countix [11]
towards videos of various groundtruth counts. As shown in Figure 5, for videos
with many repetitions, results do not degrade due to the L1 loss, with 0.356
MAE for videos with more than 20 cycles compared to 0.387 (w/o L1 loss) and
0.553 (w/o L2 loss). However, for videos with few repetitions, results get
worse without L1 loss as shown in Figure 6. The MAE for the sight stream
increases from 0.217 to 0.348 on videos having only 2 repetitions. Therefore,
the combination of L1 and L2 terms results in the best overall performance.

##  Appendix E Sight Stream Results

Here, we study the effect of hyperparameter $\theta_{s}$ and margin $m$ in our
temporal stride decision module. All the experiments are based on the sight
stream with visual modality only and the original Countix [11] dataset. Note
that despite the performance varies under different settings, all the results
by our sight stream outperform the state-of-the-art by Dwibedi _et al._ [11]
considerably.

#### Effect of $\theta_{s}$.

As illustrated in Section 3.3, $\theta_{s}$ is used to select the negative
strides for training. With a higher $\theta_{s}$, the chosen negative strides
are larger and lead the sight stream to have more omissions. In contrast, a
small $\theta_{s}$ makes the selection rule strict and thus results in over-
fit issues. We study the effect of $\theta_{s}$ by setting it in the range of
[0.1, 0.33), and the results are shown in Figure 7. We can conclude that the
performance is not very sensitive to $\theta_{s}$, and empirically
$\theta_{s}{=}0.29$ represents the best trade-off. We also observe that the
average MAE error increases when $\theta_{s}\geq 0.33$, since the trained
sight stream tends to select larger temporal strides and omit certain
repetitions.

Figure 7: Effect of $\theta_{s}$. The performance of the sight stream is not
very sensitive to $\theta_{s}$ and the best result is achieved at
$\theta_{s}{=}0.29$.  Figure 8: Effect of margin $m$ for the max-margin
ranking loss. The value of $m$ does not influence the performance much and
$m{=}2.9$ results in the lowest MAE error.

#### Effect of margin $m$.

As detailed in Section 3.3, the max-margin ranking loss is adopted for
training. In Figure 8, we show the performance of the sight stream when $m$
varies from 1.0 to 4.0. We can see that the MAE error fluctuates between 0.314
and 0.330, so the value of margin $m$ does not affect the results much.

#### Effectiveness of temporal stride module.

We report sight-stream results for fixed temporal strides on Countix in Table
7. Our temporal stride decision module obtains a much better 0.314 MAE.

Temporal stride | 1 | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---|---
MAE $\downarrow$ | 0.607 | 0.378 | 0.387 | 0.427 | 0.467 | 0.475

Table 7: Effectiveness of temporal stride module. Using a fixed temporal
stride results in inferior performance compared to 0.314 MAE by our temporal
stride module.

##  Appendix F Reliability Estimation Module Results

#### Effect of $\theta_{r}^{v}$ and $\theta_{r}^{a}$.

In Section 3.4, we use two thresholds $\theta_{r}^{v}$ and $\theta_{r}^{a}$ to
collect predictions from both modalities for training the reliability
estimation module. To study the effect of these two thresholds, we set them to
different values and the results are shown in Table 8. It is clear that the
MAE fluctuates slightly between 0.291 and 0.296 under various settings, and
the best performance is achieved when $\theta_{r}^{v}{=}0.360$ and
$\theta_{r}^{a}{=}0.400$. In particular, the reliability estimation module is
always superior to simply averaging predictions.

$\theta_{r}^{v}$ | $\theta_{r}^{a}$ | MAE $\downarrow$
---|---|---
0.365 | 0.420 | 0.294
0.365 | 0.400 | 0.291
0.365 | 0.390 | 0.292
0.365 | 0.380 | 0.294
0.360 | 0.420 | 0.292
0.360 | 0.400 | 0.291
0.360 | 0.390 | 0.292
0.360 | 0.380 | 0.295
0.355 | 0.420 | 0.294
0.355 | 0.400 | 0.293
0.355 | 0.390 | 0.293
0.355 | 0.380 | 0.296
0.350 | 0.420 | 0.292
0.350 | 0.400 | 0.293
0.350 | 0.390 | 0.292
0.350 | 0.380 | 0.295
Table 8: Effect of thresholds $\theta_{r}^{v}$ and $\theta_{r}^{a}$ in the reliability estimation module. The performance of the full sight and sound model varies slightly with different thresholds, but is always better than simply averaging the predictions from both streams.  Model components | MAE $\downarrow$ | OBO $\uparrow$
---|---|---
Predictions from the final models | 0.297 | 0.436
Fully connected layer | 0.301 | 0.421
Full sight and sound model | 0.291 | 0.479
Table 9: Comparison with other fusion schemes on Countix-AV. Using empirical
predictions is better than the predictions from final models and our fusion
scheme by predicting reliability score performs superior to two additional
fully connected layers for feature integration and counting prediction.

#### Comparison with other fusion methods.

To illustrate the superiority of our proposed scheme, which uses empirical
predictions for training, we compare our approach with two alternatives. One
is to directly use the predictions from the final trained models over the
training videos for learning. The other is similar to the fusion method
described in [7] that trains two additional fully connected layers working in
tandem built upon the penultimate layers of both sight and sound streams,
which take the concatenated features from both modalities as inputs. As our
original counting model, one fully connected layer outputs the repetition
classification results and the other predicts the counting result of each
class. The loss function is the same as Eq. 6 and Eq. 7 with the same
hyperparameters but $P$ is set to $41$ for the best result.

The results are shown in Table 9. We observe empirical predictions perform
better than directly adopting predictions from the final models, and our
reliability estimation module outputting the reliability score outperforms the
counterpart that uses fully connected layers for feature fusion as well as
counting prediction. Therefore, our proposed reliability estimation scheme
effectively integrates information from both modalities for more accurate
counting prediction.

##  Appendix G Counting with Action Class Supervision

To verify whether the action class labels can improve the counting accuracy
further, we replace the $L_{div}$ in Eq. 2 and Eq. 3 with a cross-entropy loss
using action class labels for supervision to train the repetition
classification branch. The results are shown in Table 10, while the
performances of our original models can be found in Table 5. We observe that
action class supervision can only improve the counting accuracy of the sight
stream by a small margin, while degrade the performance of the sound stream
and the full sight and sound model. The results demonstrate that repetition
classes cannot be simply regarded as action classes. Although action class
supervision can guide the network to count the correct repetitive movements
inside each video, each action class may contain various repetition classes
(repetitive motions) in different videos, which should not be counted in the
same way. For instance, for the sight stream, videos of doing aerobics contain
different movements that needed to be counted. In contrast, the arm shows
similar motions in some videos belong to action classes of playing table
tennis and playing tennis. Similar phenomenon can also be found in the sound
stream. On one hand, some videos of “Slicing onion” and “Tapping pen” contain
similar sound patterns and thus can be counted in the same way. On the other
hand, the sound stream needs to focus on various tones in different videos
that belong to the action class “skipping rope”. In some videos, it is easy
and reliable to count the repetitions by hearing how many times the feet of
the person touch the ground. However, in some other videos, only the sound of
rope is clear and usable. We provide example videos of learned repetition
classes in the folder “Learned Repetition Classes” of the supplement for both
sight and sound streams with illustration in Appendix I. Therefore, we can
conclude that for temporal repetition counting, our automatically learned
repetition classification layer is superior to its counterpart that uses
action class supervision.

| Countix | Countix-AV
---|---|---
| MAE $\downarrow$ | OBO $\uparrow$ | MAE $\uparrow$ | OBO $\uparrow$
Sight | 0.309 | 0.490 | 0.330 | 0.407
Sound | - | - | 0.400 | 0.301
Sight & Sound | - | - | 0.316 | 0.424

Table 10: Counting with action class supervision. Only the sight stream can
benefit from the action class supervision marginally, while the performances
of the sound stream and the full sight and sound model degrade. Therefore,
action class supervision cannot effectively guide the learning of repetition
counting models.

##  Appendix H Implementation Details for UCFRep

Here, we illustrate the training details of our sight-only model on the UCFRep
[42] dataset. Similar to the training on the Countix [11] dataset, we also
initialize the weights of the model from a Kinetics pretrained checkpoint.
Hyperparameters, like $\lambda_{v}^{1}$, $\lambda_{v}^{2}$, margin $m$, batch
size, learning rate, etc, remain the same, as described in Section 4.3. The
only difference is the number of repetition classes $P$, which is adjusted by
greedy search, and we find $P{=}24$ works best.

##  Appendix I Learned Repetition Classes

As our model learns to classify the input videos into different repetition
classes automatically during training, here we visualize these learned classes
by example videos in the supplementary material. The sight and sound streams
are illustrated separately.

#### Learned repetition classes of the sight stream.

In the folder “Learned Repetition Classes/Sight” of the supplementary
material, we prepare 4 groups, named from “1.mp4” to “4.mp4”, for the
illustration of videos which have similar repetition class distributions from
the repetition classification layer but belong to different action classes.
Similar movements can be discovered in videos of each group despite there are
significant variations in appearance. In “Doing aerobics.mp4”, we can see that
the video segments contain different repetitive motion patterns and thus they
are classified into different repetition classes for counting by our sight
model. However, in the field of action recognition, these segments belong to
the same action class “doing aerobics”.

#### Learned repetition classes of the sound stream.

Similar to the sight stream, in the folder “Learned Repetition Classes/Sound”
of the supplementary material, we prepare 4 groups named from “1.mp4” to
“4.mp4”, in which videos of each group have similar class distributions by the
repetition classification layer. It is clear that the audio tracks in each
group have similar sound patterns but belong to various action classes. We
also present some videos belong to the the same action class (skipping rope)
but are treated as different repetition classes due to various types of sound
in “skipping_rope.mp4”.

##  Appendix J Example Videos

In “demo_video.mp4” of the supplementary material, we show some example videos
of our Extreme Countix-AV dataset with corresponding challenges as well as the
predictions from the sight and the sound stream, our full sight and sound
model and groundtruth.