# Probing Magnetic Moment Operators in $H\gamma$ Production and
$H\to\tau^{+}\tau^{-}\gamma$ Rare Decay

Qing-Hong Cao<EMAIL_ADDRESS>School of Physics and State Key
Laboratory of Nuclear Physics and Technology, Peking University, Beijing
100871, China  Collaborative Innovation Center of Quantum Matter, Beijing
100871, China  Center for High Energy Physics, Peking University, Beijing
100871, China  Hao-Ran Jiang<EMAIL_ADDRESS>School of Physics and State
Key Laboratory of Nuclear Physics and Technology, Peking University, Beijing
100871, China  Bin Li<EMAIL_ADDRESS>School of Physics and State Key
Laboratory of Nuclear Physics and Technology, Peking University, Beijing
100871, China  Yandong Liu<EMAIL_ADDRESS>Key Laboratory of Beam Technology
of Ministry of Education, College of Nuclear Science and Technology, Beijing
Normal University, Beijing 100875, China  Beijing Radiation Center, Beijing
100875, China  Guojin Zeng<EMAIL_ADDRESS>School of Physics and State
Key Laboratory of Nuclear Physics and Technology, Peking University, Beijing
100871, China

###### Abstract

The magnetic moment ($a_{\gamma}$) and weak magnetic moment ($a_{W}$) of
charged leptons and quarks are sensitive to quantum effects of new physics
heavy resonances. In effective field theory $a_{\gamma}$ and $a_{W}$ are
induced by two independent operators, therefore, one has to measure both the
$a_{\gamma}$ and $a_{W}$ to shed lights on new physics. The $a_{W}$’s of the
SM fermions are measured at the LEP. In this work, we analyze the
contributions from magnetic and weak magnetic moment operators in the
processes of $pp\to H\gamma$ and $gg\to H\to\tau^{+}\tau^{-}\gamma$ at the
High-Luminosity Large Hadron Collider. We demonstrate that the two processes
could cover most of the parameter space that cannot be probed at the LEP.

1\. Introduction.

Searching for new physics (NP) beyond the Standard Model (SM) is the key
mission of particle physics. Although no heavy resonances have been discovered
at the Large Hadron Collider (LHC), one can probe the quantum effects of those
heavy resonances through the measurement of magnetic moment ($a_{\gamma}$) and
weak magnetic moment ($a_{W}$) of the SM fermions Miller et al. (2007);
Lindner et al. (2018); Stockinger (2007); Jegerlehner and Nyffeler (2009).
When NP resonances are too heavy to be directly probed at the current
colliders, one could describe the unknown NP effects through high-dimensional
operators constructed with the SM fields at the NP scale $\Lambda$, obeying
the well-established gauge structure of the SM, i.e. $SU(2)_{W}\otimes
U(1)_{Y}$. The Lagrangian of effective field theory (EFT) is

$\mathcal{L}_{\rm
EFT}=\mathcal{L}_{SM}+\frac{1}{\Lambda^{2}}\sum_{i}\left(C_{i}{O}_{i}+h.c.\right)+\mathcal{O}\left(1/\Lambda^{3}\right),$
(1)

where $C_{i}$’s are the Wilson coefficients. In the Warsaw basis the
dimension-6 operators $O_{fW}$’s and $O_{fB}$’s that generate $a_{\gamma}$ and
$a_{W}$ are given by Grzadkowski et al. (2010); Dedes et al. (2017)

$\displaystyle O_{e_{i}W}$
$\displaystyle=(\bar{L}_{i}\sigma^{\mu\nu}\tau^{I}e_{i})\phi W^{I}_{\mu\nu},$
$\displaystyle O_{e_{i}B}$
$\displaystyle=(\bar{L}_{i}\sigma^{\mu\nu}e_{i})\phi B_{\mu\nu},$
$\displaystyle O_{u_{i}W}$
$\displaystyle=(\bar{Q}_{i}\sigma^{\mu\nu}\tau^{I}u_{i})\tilde{\phi}W^{I}_{\mu\nu},$
$\displaystyle O_{u_{i}B}$
$\displaystyle=(\bar{Q}_{i}\sigma^{\mu\nu}u_{i})\tilde{\phi}B_{\mu\nu},$
$\displaystyle O_{d_{i}W}$
$\displaystyle=(\bar{Q}_{i}\sigma^{\mu\nu}\tau^{I}d_{i})\phi W^{I}_{\mu\nu},$
$\displaystyle O_{d_{i}B}$
$\displaystyle=(\bar{Q}_{i}\sigma^{\mu\nu}d_{i})\phi B_{\mu\nu},$ (2)

where $L_{i}$ and $Q_{i}$ denotes the left-handed weak doublet of the $i$-th
generation in the SM while $e_{i}(u_{i},d_{i})$ the right-handed weak singlet
of charged leptons (up-type and down-type quarks), respectively. Figure 1
shows Feynman diagrams of the dimension-6 operators; see (a) and (b). After
spontaneously symmetry breaking the operators yield $f\bar{f}V$ and
$f\bar{f}VV$ anomalous couplings; see (c) and (d). The $f\bar{f}V$ anomalous
couplings give rise to the magnetic moment ($a_{\gamma}^{f}$) and the weak
magnetic moment ($a_{W}^{f}$) of the fermion $f$ as follows:

$\displaystyle\delta
a_{\gamma}^{f=u,c,t}=-2\sqrt{2}\frac{m_{f}v}{\Lambda^{2}}\frac{1}{eQ_{f}}(c_{W}~{}C_{fB}+s_{W}~{}C_{fW}),$
$\displaystyle\delta
a_{W}^{f=u,c,t}=+2\sqrt{2}\frac{m_{f}v}{\Lambda^{2}}\frac{1}{eQ_{f}}(s_{W}~{}C_{fB}-c_{W}~{}C_{fW}),$
$\displaystyle\delta
a_{\gamma}^{f=\ell,d,s,b}=-2\sqrt{2}\frac{m_{f}v}{\Lambda^{2}}\frac{1}{eQ_{f}}(c_{W}~{}C_{fB}-s_{W}~{}C_{fW}),$
$\displaystyle\delta
a_{W}^{f=\ell,d,s,b}=+2\sqrt{2}\frac{m_{f}v}{\Lambda^{2}}\frac{1}{eQ_{f}}(s_{W}~{}C_{fB}+c_{W}~{}C_{fW}),$
(3)

where $Q_{f}$ and $m_{f}$ denotes the charge and mass of the fermion $f$,
$v=246~{}{\rm GeV}$ is the vacuum expectation value of the Higgs doublet after
symmetry breaking, $s_{W}$ and $c_{W}$ denotes sine and cosine of the Weinberg
angle, respectively. As the $a_{\gamma}^{f}$ and $a_{W}^{f}$ are orthogonal in
the parameter space of $C_{fB}$ and $C_{fW}$, one has to measure both the
$a_{\gamma}^{f}$ and $a_{W}^{f}$ to probe the NP effects.

Figure 1: Feynman diagrams of the dimension-6 operator before and after the
symmetry breaking.

The magnetic moments of up-quarks and down-quarks $a_{\gamma}^{u,d}$ and
$a_{W}^{u,d}$ (corresponding to the operators $O_{uW/dW}$ and $O_{uB/dB}$) are
tightly constrained through Drell-Yan processes, $VV$ pair production and $VH$
associated production at the LHC da Silva Almeida et al. (2019); Aad et al.
(2021). The operators $O_{tW}$ and $O_{tB}$ of top quarks could be examined in
single-top productions or top-quark decays Zhang and Willenbrock (2011); Cao
et al. (2021); Olive et al. (2014); Cirigliano et al. (2016); Goldouzian et
al. (2020). The $a_{W}^{s,c,b}$ is bounded by the precise measurements at the
LEP Abdallah et al. (2004); Rizzo (1995); Alcaraz et al. (2006); Escribano and
Masso (1994), which yield

$\displaystyle\Big{|}s_{W}C_{s(b)B}+c_{W}C_{s(b)W}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq
4.2,$
$\displaystyle\Big{|}s_{W}C_{cB}-c_{W}C_{cW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq
4.2~{}.$ (4)

In this work we show that the $a_{\gamma}^{s,c,b}$ induced by the two
operators can be tested in the process of $pp\to H\gamma$ at the LHC with an
integrated luminosity of $3000~{}{\rm fb}^{-1}$ (HL-LHC).

The magnetic moments of electrons and muons are severely constrained by the
$Z$-boson width measurement at the LEP Escribano and Masso (1994) and the
measurements of magnetic moment Parker et al. (2018); Hanneke et al. (2008,
2011); Kinoshita and Nio (2006); Bennett et al. (2006); Davier et al. (2017);
Keshavarzi et al. (2018); Abi et al. (2021); therefore, we do not consider the
electron and muon in this work. The LEP constraint on the $a_{W}^{\tau}$ of
the $\tau$-lepton reads as Abdallah et al. (2004); Rizzo (1995); Alcaraz et
al. (2006); Escribano and Masso (1994); Heister et al. (2003); Gonzalez-
Sprinberg et al. (2000)

$\Big{|}s_{W}C_{\tau B}+c_{W}C_{\tau
W}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq 0.28~{},$ (5)

while the constraint on the $a_{\gamma}^{\tau}$ as

$-12.8\leq\big{(}c_{W}C_{\tau B}-s_{W}C_{\tau
W}\big{)}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq 3.2~{}.$ (6)

We demonstrate that the $a_{\gamma}^{\tau}$ can be measured in the process of
$gg\to H\to\tau^{+}\tau^{-}\gamma$ at the HL-LHC.

2\. The $H\gamma$ associated production.

Figure 2: Feynman diagrams of the $H\gamma$ production.

In this section we examine the effects of magnetic-moment operators in the
$pp\to H\gamma$ process at the HL-LHC, which has been studied extensively in
the literature CMS (2018); Aaboud et al. (2018); Aad et al. (2020); Shi et al.
(2019); Khanpour et al. (2017). We consider one flavor at a time throughout
this work. Figure 2(a) and (b, c) display the Feynman diagrams of the
$H\gamma$ production induced by the operators $O_{fB}$ and $O_{fW}$. The SM
process is shown in Fig. 2(d, e). There are non-zero interference effects
between the operator-induced diagrams and the SM diagrams, therefore, we also
treat the interference effect as the signal. However, as explained below, only
the first diagram contributes after applying hard kinematic cuts and the
interference effects are negligible.

In our simulation the Higgs bosons are required to decay into a pair of bottom
quarks, the predominant decay mode of the Higgs boson. The event topology of
the signal process is two bottom quarks plus a photon. The SM backgrounds are

$\displaystyle pp\to\gamma+\text{jets},$ $\displaystyle pp\to t\bar{t}\gamma,$
$\displaystyle pp\to Z\gamma,$ $\displaystyle pp\to
t\gamma/\bar{t}\gamma+\text{jets},$ (7)

with the $Z$ boson and top quark hadronic decay. It is noted that background
$\gamma+\text{jets}$ consists of $\gamma+b\bar{b}$, $\gamma+c\bar{c}$,
$\gamma~{}+$ light flavor jets and etc.

We generate signal and background events utilizing MadEvent Alwall et al.
(2014), and then pass those to Pythia Sjostrand et al. (2008) and Delphes de
Favereau et al. (2014) for parton showering, hadronization and collider
simulation. In order to avoid collinear and soft divergences in the process of
$pp\to\gamma+\text{jets}$ we apply the kinematic cuts at the generator level
as follows:

$\displaystyle p_{\text{T}}^{\gamma}\geq 30~{}\mathrm{GeV},~{}$
$\displaystyle|\eta_{\gamma}|\leq 2.5,$ $\displaystyle p_{\text{T}}^{j}\geq
30~{}\mathrm{GeV},~{}$ $\displaystyle|\eta_{j}|\leq 2.5,$ $\displaystyle\Delta
R_{jj}>0.4,~{}$ $\displaystyle\Delta R_{j\gamma}>0.4,$ (8)

where $p_{\rm T}^{\gamma/j}$ and $\eta^{\gamma/j}$ is the transverse momentum
and pseudo-rapidity of $\gamma$ and jet, respectively, and $\Delta
R_{ij}\equiv\sqrt{(\phi_{i}-\phi_{j})^{2}+(\eta_{i}-\eta_{j})^{2}}$ is the
angular distance between the objects $i$ and $j$ in the azimuthal angle
($\phi$)-pseudurapidity ($\eta$) plane. At the detector level two $b$-jets are
demanded in the final state to suppress the SM backgrounds. In the simulation
we utilize $b$-tagging technology Chatrchyan et al. (2013); Aad et al. (2016)
to distinguish the jet flavor. The $b$-tagging efficiency is chosen as $70\%$,
the mistagging rate of $c$-quarks and light flavor quarks is $10\%$ and $1\%$,
respectively. We require at least one photon in the final state, i.e.,

$n^{b-\rm{jet}}=2,~{}n^{\gamma}\geq 1.$ (9)

Figure 3: The normalized distributions of $P_{\text{T}}^{\gamma}$ (a),
$\eta^{\gamma}$ (b), $M_{b\bar{b}}$ (c) and $M_{b\bar{b}\gamma}$ (d). The
black (red) solid curve denotes the distributions of the signal events induced
by the operator $O_{bB}$ ($O_{cB}$), respectively. The operator $O_{bW}$
($O_{cW}$) yields exactly the same normalized distributions as the operator
$O_{bB}$ ($O_{cB}$), respectively. The dashed curves label the SM backgrounds
from the $\gamma+{\rm jets}$ (blue), $Z\gamma$ (magenta) and $t\bar{t}\gamma$
(green), respectively.

Figure 3 displays the normalized distributions of $p_{\text{T}}^{\gamma}$ (a),
$\eta^{\gamma}$ (b), $M_{b\bar{b}}$ (c) and $M_{b\bar{b}\gamma}$ (d) after
imposing the basic cuts given in Eq. 8 and Eq. 9. For demonstration, we plot
the distributions of the signal events induced by the operator $O_{cB}$ and
$O_{bB}$, respectively. It shows that the photons in the signal events exhibit
a hard $p_{\text{T}}$ and mainly appear in the central region of the detector;
see the black-solid and the red-solid curves. The reason can be understood as
follows. In the signal events the two fermions in the initial state are in the
different chirality states and thus are in the $s$-wave state. In order to
respect the angular momentum conservation, the particles in the final state
are in the $p$-wave state such that the matrix element is proportional to
$p_{\rm T}^{\gamma}$. As a result, the matrix element of the signal process is
proportional to $\sin\theta$ where $\theta$ is the polar angle of the photon
with respect to the beam line in the center of mass frame. On the other hand,
the photons in the background mainly arise from the QED radiation and tend to
be soft.

For illustration, we present the leading contributions of the squared matrix
elements of the $f\bar{f}\to H\gamma$ process as follows:

$\displaystyle|M_{a}|^{2}=\frac{4N_{c}c^{2}_{W}(s-m_{H}^{2})^{2}}{\Lambda^{4}}\sin^{2}\theta,$
$\displaystyle|M_{b}+M_{c}|^{2}=\frac{48N_{c}c^{2}_{W}m_{f}^{2}s}{\Lambda^{4}},$
$\displaystyle|M_{d}+M_{e}|^{2}$ $\displaystyle=$
$\displaystyle\frac{8N_{c}Q_{f}^{2}e^{2}m_{f}^{2}\big{[}3s^{2}-2sm_{H}^{2}-\cos\theta(s-m_{H})^{2}+3m_{H}^{4}\big{]}}{v^{2}(s-m_{H})^{2}\sin^{2}\theta},$
$\displaystyle{\rm Re}(M^{\dagger}_{a}(M_{d}+M_{e}))$ $\displaystyle=$
$\displaystyle\frac{\sqrt{2}N_{c}Q_{f}ec_{W}m_{f}\big{[}3s+\cos\theta(s-m_{H}^{2})-7m_{H}^{2}\big{]}}{\Lambda^{2}v},$
(10)

where $N_{c}=3$ is the color factor, the subscript of the matrix element
denotes the corresponding Feynman diagram in Fig. 2. In the region of large
colliding energy, only $|M_{a}|^{2}$ contributes while the others
contributions are negligible. Indeed, the $|M_{a}|^{2}$ is proportional to
$\sin^{2}\theta$.

Taking advantage of the hard photon in the signal events, we impose a hard
$p_{T}$ cut on the photon with the largest $p_{T}$ as following:

$p_{\text{T}}^{\gamma}\geq 300~{}\mathrm{GeV},$ (11)

to suppress the SM backgrounds.

Figure 3(c) displays the normalized distributions of the invariant mass of two
$b$-jets ($M_{b\bar{b}}$). The two $b$-jets in the signal event originate from
the Higgs boson decay, therefore, their invariant mass is around $m_{H}$; see
the black-solid and the red-solid curves. Similarly, there is a peak around
$m_{Z}$ in the $Z\gamma$ backgrounds. The two $b$-jets in other SM backgrounds
are not from a resonance decay and yield a flat $M_{b\bar{b}}$ distribution.
We impose a mass window cut on the two $b$-jets,

$\left|M_{b\bar{b}}-m_{H}\right|\leq 15~{}\mathrm{GeV},$ (12)

to suppress the SM backgrounds.

Figure 3(d) displays the normalized distributions of the invariant mass of two
$b$-jets and a photon ($M_{b\bar{b}\gamma}$). The signal events tend to have a
large invariant mass while the background events prefer the small invariant
mass region. In order to suppress the SM backgrounds, we further impose a hard
cut on $M_{b\bar{b}\gamma}$ as following:

$M_{b\bar{b}\gamma}\geq 700~{}\mathrm{GeV}.$ (13)

Table 1 shows the numbers of the signal events and the background events after
the basic cuts and the optimal cuts (i.e., the $p_{\text{T}}^{\gamma}$,
$M_{b\bar{b}}$ and $M_{b\bar{b}\gamma}$ cuts) at the HL-LHC. Note that the NP
scale $\Lambda$ is set to be $1~{}{\rm TeV}$. The major SM background comes
from the $\gamma+{\rm jets}$ process in which the $\gamma b\bar{b}$ channel
dominates. As both the $O_{fB}$ and $O_{fW}$ operators contribute to the
signal process through the same $f\bar{f}VH$ vertex, they generate the same
differential distributions and therefore have the same cut efficiencies. The
$O_{fW}$ and $O_{fB}$ operators differ in the cross section by a total factor
$\tan^{2}\theta_{W}$.

Table 1: The number of the signal and background events at the HL-LHC for $\Lambda=1~{}{\rm TeV}$. Signal processes | Basic cuts | Optimal cuts
---|---|---
$C_{bB}=1$, $C_{bW}=0$ | $9.38\times 10^{3}$ | $815$
$C_{bB}=0$, $C_{bW}=1$ | $2.82\times 10^{3}$ | $245$
$C_{cB}=1$, $C_{cW}=0$ | $2.78\times 10^{4}$ | $3.17\times 10^{3}$
$C_{cB}=0$, $C_{cW}=1$ | $8.37\times 10^{3}$ | $953$
$C_{sB}=1$, $C_{sW}=0$ | $3.86\times 10^{4}$ | $4.79\times 10^{3}$
$C_{sB}=0$, $C_{sW}=1$ | $1.16\times 10^{4}$ | $1.44\times 10^{3}$
Background processes | Basic Cuts | Optimal cuts
$\gamma$+jets | $3.32\times 10^{8}$ | $9.90\times 10^{4}$
$Z\gamma$ | $5.80\times 10^{5}$ | $828$
$t\bar{t}\gamma$ | $4.85\times 10^{5}$ | $1.25\times 10^{3}$
$t\gamma/\bar{t}\gamma+\text{jets}$ | $2.40\times 10^{5}$ | $106$

Figure 4: The yellow-meshed bands denote the $5\sigma$ discovery region of
$C_{fB}$ and $C_{fW}$ for the $s$-quark (a), the $c$-quark (b) and $b$-quark
(c) in the $H\gamma$ production with $\Lambda=1~{}{\rm TeV}$ at the HL-LHC,
respectively, while the yellow bands denote the allowed regions by the
$2\sigma$ bounds at the LEP. The gray band denotes the allowed region at the
$2\sigma$ significance if no NP effects are observed in the $H\gamma$
production. The black line denotes the $2\sigma$ constraint of the $b\to
s\gamma$ measurement.

Equipped with the optimal cuts shown above, we vary the Wilson coefficients to
obtain a 5 standard deviations ($\sigma$) statistical significance using

$\sqrt{-2\left[(n_{b}+n_{s})\log\frac{n_{b}}{n_{s}+n_{b}}+n_{s}\right]}=5,$
(14)

where $n_{b}$ and $n_{s}$ represents the numbers of the signal and background
events, respectively. The number of the signal events in Table 1 is calculated
with the choice of $C_{fW}=1$ or $C_{fB}=1$ ($f=c,s,b$), and $\Lambda=1~{}{\rm
TeV}$. Denote the number of the signal events in the last column of Table 1
after all the cuts as $n_{s}^{0}$. The $n_{s}^{f}$ for a general choice of
$C_{fW}$, $O_{fB}$ and $\Lambda$ can be obtained as follows:

$\displaystyle
n_{s}^{s}=n_{s}^{0}\Big{|}_{C_{sB}=1}\times\frac{(c_{W}C_{sB}-s_{W}C_{sW})^{2}}{c_{W}^{2}}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{4},$
$\displaystyle
n_{s}^{c}=n_{s}^{0}\Big{|}_{C_{cB}=1}\times\frac{(c_{W}C_{cB}+s_{W}C_{cW})^{2}}{c_{W}^{2}}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{4},$
$\displaystyle
n_{s}^{b}=n_{s}^{0}\Big{|}_{C_{bB}=1}\times\frac{(c_{W}C_{bB}-s_{W}C_{bW})^{2}}{c_{W}^{2}}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{4}.$
(15)

Using Eq. 14, we obtain that a $5\sigma$ discovery significance requires

$\displaystyle\Big{|}c_{W}C_{sB}-s_{W}C_{sW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\geq
0.51,$
$\displaystyle\Big{|}c_{W}C_{cB}+s_{W}C_{cW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\geq
0.62,$
$\displaystyle\Big{|}c_{W}C_{bB}-s_{W}C_{bW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\geq
1.23~{}.$ (16)

Figure 4 displays the $5\sigma$ discovery region in the plane of $C_{fW}$ and
$C_{fB}$ for the strange-quark (a), the charm-quark (c) and the bottom-quark
(c) in the $H\gamma$ production at the HL-LHC (yellow-meshed band). The yellow
band denotes the allowed $2\sigma$ parameter space by the $Z$-width
measurement at the LEP. We consider one flavor at a time. The HL-LHC has a
great potential of probing the $O_{s(c,b)B}$ and $O_{s(c,b)W}$ in comparison
with the LEP. The operator $O_{bW}$ is highly constrained by the $b\to
s\gamma$ measurement Olive et al. (2014); Cirigliano et al. (2016), i.e.,

$-0.008\leq C_{bW}\leq 0.011~{}.$ (17)

We plot the constraint of $b\to s\gamma$ on the $O_{bW}$ at the $95\%$
confidence level in Fig. 4(c); see the black line. Obviously, the constraints
from the $b\to s\gamma$ measurement is much more stringent; however, it can
constrain the $O_{bW}$ but not the $O_{bB}$.

If no deviation is observed in the $H\gamma$ production, we can set an upper
limit on the Wilson coefficients at the $2\sigma$ confidence level in terms of

$\sqrt{-2\left(n_{b}\ln\frac{n_{s}+n_{b}}{n_{b}}-n_{s}\right)}=2.0,$ (18)

which yields bounds on the Wilson coefficients at the $95\%$ confidence level
as follows:

$\displaystyle\Big{|}c_{W}C_{sB}-s_{W}C_{sW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq
0.32,$
$\displaystyle\Big{|}c_{W}C_{cB}+s_{W}C_{cW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq
0.39,$
$\displaystyle\Big{|}c_{W}C_{bB}-s_{W}C_{bW}\Big{|}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq
0.78;$ (19)

see the gray bands in Fig. 4. The slope of the gray bands is $-\cot\theta_{W}$
for the up-type quarks and $+\cot\theta_{W}$ for the down-type quarks,
respectively. The gray bands are perpendicular to the yellow bands owing to
the mixing of the weak and hypercharge fields; see Eq. 3.

3\. The rare decay of $H\to\tau^{+}\tau^{-}\gamma$.

Table 2: Decay width and branching ratio of $H\to\tau^{+}\tau^{-}\gamma$ with the choice of $C_{\tau B}=1$ or $C_{\tau W}=1$ for $\Lambda=1~{}{\rm TeV}$ after demanding $p_{\text{T}}^{\gamma}\geq 10~{}{\rm GeV}$. The SM contribution, the pure NP contribution (square) and the interference between the SM and NP effects are listed separately.  Operators | Processes | Width (GeV) | BR
---|---|---|---
$O_{\tau B}$ | Interference | $2.51\times 10^{-6}$ | $6.27\times 10^{-4}$
Square | $9.71\times 10^{-7}$ | $2.43\times 10^{-4}$
$O_{\tau W}$ | Interference | $-1.35\times 10^{-6}$ | $-3.38\times 10^{-4}$
Square | $2.81\times 10^{-7}$ | $7.04\times 10^{-5}$
SM | $H\to\tau^{\pm}\tau^{\mp*}\to\tau^{+}\tau^{-}\gamma$ | $8.68\times 10^{-6}$ | $2.17\times 10^{-3}$

In the section we explore the HL-LHC potential of searching for the $O_{\tau
B}$ and $O_{\tau W}$ through the rare decay of $H\to\tau^{+}\tau^{-}\gamma$ in
the single Higgs-boson production process of $gg\to H$.

The partial decay width of $H\to\tau^{+}\tau^{-}\gamma$ in the SM is very
tiny, $\sim 8.68\times 10^{-6}~{}\rm{GeV}$ Kovalchuk and Korchin (2017) in
comparison with the full width of the Higgs boson, $\sim 4~{}{\rm MeV}$
Andersen et al. (2013). Table 2 shows the partial decay width and the
branching ratio of the rare decay of $H\to\tau^{+}\tau^{-}\gamma$ induced by
the operators $O_{\tau B}$ and $O_{\tau W}$ after the cut of
$p_{\text{T}}^{\gamma}\geq 10~{}{\rm GeV}$, which is used to trigger the
signal event and suppress the SM backgrounds. To be more specific, we present
the interference effect and the pure NP contribution (square) separately. The
NP operator effects are comparable to the SM contribution.

In the collider simulation we demand the $\tau^{\pm}$-leptons decaying
hadronically; therefore, the signal process of $pp\to
H\to\tau^{+}\tau^{-}\gamma$ yields a collider signature of two $\tau$-jets
plus a hard photon. Figure 5 displays the representative Feynman diagrams for
the signal and background processes. The irreduciable backgrounds in the SM
are

$\displaystyle pp\to Z\gamma\to\tau^{+}\tau^{-}\gamma,$ $\displaystyle pp\to
H\to\tau^{+}\tau^{-}\gamma,$ (20)

and the reducible QCD backgrounds are

$pp\to\gamma+\text{jets},$ (21)

when at least two of the QCD jets are mistagged as $\tau$-jets. The jet in the
event is reconstructed with anti-$k_{\text{T}}$ jet algorithm Cacciari et al.
(2008) with $R=0.4$. The $\tau$-tagging efficiency of the hadronic decay
$\tau^{\pm}$ is chosen to be 60% with the mis-tagging rate
$\epsilon(j\to\tau)=1\%$.

Figure 5: Feynman diagrams of the signal process of $gg\to
H\to\tau^{+}\tau^{-}\gamma$ (a) and the representative diagrams of the SM
backgrounds (b, c, d).

Figure 6: Normalized distributions of the $p_{\text{T}}^{\gamma}$ (a), the
$p_{\text{T}}^{j1}$ (b) and the invariant mass $M_{\tau^{+}\tau^{-}\gamma}$
(c) after the basic cuts given in Eq. 22. The black-solid and red-solid curve
denotes the pure NP contribution (squared) and the interference effect
(interference) in the signal event, respectively. The blue-dashed, magenta-
dashed and the green-dashed curves represent the SM background processes.

To tigger the signal event, we demand a set of basic cuts as follows:

$\displaystyle p_{\text{T}}^{\gamma}\geq 10~{}{\rm GeV},$
$\displaystyle|\eta_{\gamma}|\leq 2.5,$ $\displaystyle p_{\text{T}}^{j}\geq
20~{}{\rm GeV},$ $\displaystyle|\eta_{j}|\leq 2.5,$ $\displaystyle\Delta
R_{\gamma j/jj}\geq 0.4~{}.$ (22)

Denote $j_{1}$ as the $\tau$-jet with a larger $p_{\text{T}}$. Figure 6
displays the normalized distributions of $p_{\text{T}}^{\gamma}$ (a),
$p_{\text{T}}^{j_{1}}$ (b) and $M_{\tau^{+}\tau^{-}\gamma}$ (c). In the signal
event the photon exhibits a $p_{\text{T}}$ distribution harder than the
$\tau$-jet. On the other hand, the photons in the background processes tend to
be soft as they arise predominantly from the radiation of the charged leptons.
In order to further suppress the SM backgrounds, we demand hard cuts on the
$p_{\text{T}}$ of the photons and $\tau$-jets as follows:

$\displaystyle p_{\text{T}}^{\gamma}\geq
30~{}\mathrm{GeV},~{}~{}p_{\text{T}}^{j}\geq 30~{}\mathrm{GeV}.$ (23)

Figure 6(c) shows the normalized distributions of the invariant mass
$M_{\tau^{+}\tau^{-}\gamma}$ in which the signal process peaks around $m_{H}$
and one of the background process of $pp\to Z\gamma$ peaks around $m_{Z}$. The
$\tau$-jets in the QCD background mainly arise from the faked $\tau$-tagging
and do not exhibit any resonance effect. Therefore, we require

$\left|M_{\tau^{+}\tau^{-}\gamma}-m_{H}\right|\leq 15~{}\mathrm{GeV}$ (24)

to suppress the SM background from the process of $pp\to Z\gamma$.

Table 3: The numbers of the signal and background events after the hard cuts and the $M_{\tau^{+}\tau^{-}\gamma}$ mass window cut at the HL-LHC for $\Lambda=1~{}{\rm TeV}$.  Operators | Process | Hard cuts | $M_{\tau^{+}\tau^{-}\gamma}$
---|---|---|---
$C_{\tau B}=1$, $C_{\tau W}=0$ | Interference | $1.16\times 10^{3}$ | $331$
Square | $470$ | $139$
$C_{\tau B}=0$, $C_{\tau W}=1$ | Interference | $-624$ | $-178$
Square | $136$ | $40.2$
Backgrounds | $pp\to H\to\tau^{+}\tau^{-}\gamma$ | $5.75\times 10^{3}$ | $1.63\times 10^{3}$
$pp\to Z\gamma\to\tau^{+}\tau^{-}\gamma$ | $4.40\times 10^{4}$ | $1.01\times 10^{4}$
$pp\to\gamma+$jets | $8.11\times 10^{6}$ | $1.85\times 10^{5}$

Table 3 shows the numbers of the signal and the background events at the HL-
LHC after the hard cuts and the mass window cut of
$M_{\tau^{+}\tau^{-}\gamma}$. We separate the signal contribution into the
pure NP effect (square) and the interference effect. Again, the $O_{\tau B}$
and $O_{\tau W}$ yield exactly the same cut efficiencies. As a result, the
number of the signal events for a general choice of $C_{\tau B}$ and $C_{\tau
W}$ can be expressed as follows:

$\displaystyle n_{s}$ $\displaystyle=$ $\displaystyle
n_{s}^{\text{sqr}}\times\frac{(c_{W}C_{\tau B}-s_{W}C_{\tau
W})^{2}}{c_{W}^{2}}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{4}$ (25)
$\displaystyle+$ $\displaystyle n_{s}^{\text{int}}\times\frac{(c_{W}C_{\tau
B}-s_{W}C_{\tau W})}{c_{W}}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2},$

where $n_{s}^{\text{sqr}}$ is the signal event number from the pure NP
contribution (square) and $n_{s}^{\text{int}}$ is the signal event number from
the interference effect (interference) for $C_{\tau B}=1$ and $C_{\tau W}=0$.

Using Eq. 14, we obtain that a $5\sigma$ discovery significance in the process
of $gg\to H\to\tau^{+}\tau^{-}\gamma$ requires

$\displaystyle\Big{(}c_{W}C_{\tau B}-s_{W}C_{\tau
W}\Big{)}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq-4.7,$ (26)

or

$\displaystyle\Big{(}c_{W}C_{\tau B}-s_{W}C_{\tau
W}\Big{)}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\geq 2.6~{}.$ (27)

Figure 7 shows the parameter space of $5\sigma$ discovery in the plane of
$C_{\tau B}$ and $C_{\tau W}$ obtained from the process of $gg\to
H\to\tau^{+}\tau^{-}\gamma$ at the HL-LHC (yellow-meshed band) with
$\Lambda=1~{}{\rm TeV}$. The yellow band denotes the $2\sigma$ parameter space
allowed by the LEP measurement Escribano and Masso (1994). The HL-LHC can
cover the most of the parameter space that cannot be accessed at the LEP.

Figure 7: The yellow-meshed bands denote the $5\sigma$ discovery region of
$C_{\tau B}$ and $C_{\tau W}$ in the the process of $gg\to
H\to\tau^{+}\tau^{-}\gamma$ at the HL-LHC with $\Lambda=1~{}{\rm TeV}$. The
yellow bands denote the $2\sigma$ parameter space allowed by the $Z$-pole
measurement at the LEP. The region between the two dashed lines is allowed at
the $2\sigma$ significance if no NP effects are observed at the HL-LHC, where
the overlapped gray band satisfies both the LEP and the HL-LHC bounds.

If no deviation is found in the process of $gg\to H\to\tau^{+}\tau^{-}\gamma$,
we obtain a bound on the Wilson coefficients at the $95\%$ confidence level as

$-3.5\leq\Big{(}c_{W}C_{\tau B}-s_{W}C_{\tau
W}\Big{)}\left(\frac{1~{}\text{TeV}}{\Lambda}\right)^{2}\leq 1.4;$ (28)

see the region between the two dashed lines. The overlapped gray band
satisfies both the LEP and the HL-LHC bounds.

4\. Conclusion.

The magnetic moment and weak magnetic moment of the SM fermions are sensitive
to quantum effects of new physics resonances. For each fermion $f$ there are
two independent operators that generate the magnetic moment and weak magnetic
moment; one is $O_{fB}$ involving the hypercharge field, the other is $O_{fW}$
involving the weak field. After symmetry breaking the magnetic moment and the
weak magnetic moment depend on the orthogonal combinations of the two
operators. Therefore, at least two independent experiments are needed to probe
the $O_{fB}$ and $O_{fW}$.

The weak magnetic moment of the strange-quark ($s$), charm-quark ($c$) or
bottom-quark ($b$) is bounded by the width measurement of the $Z$-boson at the
LEP, but the magnetic moments of the three quarks are less constrained. In
this paper, we explore the potential of the HL-LHC on probing the operators
$O_{fB}$ and $O_{fW}$ ($f=s,c,b$) in the process of $pp\to H\gamma$, in which
the magnetic moment of the quarks dominates. We consider one flavor of quarks
at a time. We showed that, in most of the parameter space that cannot be
accessed at the LEP, a $5\sigma$ significance discovery can be reached in the
$H\gamma$ production at the HL-LHC.

The magnetic moment and weak magnetic moment of the electron or muon have been
accurately measured and thus severely constrained. In analogue to the electron
and muon leptons the weak magnetic moment of the $\tau$ lepton is also bounded
at the LEP; however, the magnetic moment of the $\tau$ lepton is less bounded.
In this work we consider the rare decay of $H\to\tau^{+}\tau^{-}\gamma$
induced by the $O_{\tau B}$ and $O_{\tau W}$ and examined the potential of
probing the two operators in the process of $gg\to
H\to\tau^{+}\tau^{-}\gamma$. Similar to the case of quark operators, in most
of the parameter space that cannot be accessed at the LEP, a $5\sigma$
significance discovery can be reached in the process of $gg\to
H\to\tau^{+}\tau^{-}\gamma$ at the HL-LHC.

In summary, one can probe the magnetic moment of the $s(c,b)$ quarks in the
process of $pp\to H\gamma$ and the magnetic moment of the $\tau^{\pm}$ lepton
in the $gg\to H\to\tau^{+}\tau^{-}\gamma$ at the HL-LHC.

Acknowledgement.

The work is supported in part by the National Science Foundation of China
under Grant Nos. 11725520, 11675002, 11635001, 11805013, 12075257 and the
Fundamental Research Funds for the Central Universities under Grant No.
2018NTST09.

## References

  * Miller et al. (2007) J. P. Miller, E. de Rafael, and B. L. Roberts, Rept. Prog. Phys. 70, 795 (2007), eprint hep-ph/0703049.
  * Lindner et al. (2018) M. Lindner, M. Platscher, and F. S. Queiroz, Phys. Rept. 731, 1 (2018), eprint 1610.06587.
  * Stockinger (2007) D. Stockinger, J. Phys. G 34, R45 (2007), eprint hep-ph/0609168.
  * Jegerlehner and Nyffeler (2009) F. Jegerlehner and A. Nyffeler, Phys. Rept. 477, 1 (2009), eprint 0902.3360.
  * Grzadkowski et al. (2010) B. Grzadkowski, M. Iskrzynski, M. Misiak, and J. Rosiek, JHEP 10, 085 (2010), eprint 1008.4884.
  * Dedes et al. (2017) A. Dedes, W. Materkowska, M. Paraskevas, J. Rosiek, and K. Suxho, JHEP 06, 143 (2017), eprint 1704.03888.
  * da Silva Almeida et al. (2019) E. da Silva Almeida, N. Rosa-Agostinho, O. J. P. Éboli, and M. C. Gonzalez-Garcia, Phys. Rev. D 100, 013003 (2019), eprint 1905.05187.
  * Aad et al. (2021) G. Aad et al. (ATLAS), Eur. Phys. J. C 81, 178 (2021), eprint 2007.02873.
  * Zhang and Willenbrock (2011) C. Zhang and S. Willenbrock, Phys. Rev. D 83, 034006 (2011), eprint 1008.3869.
  * Cao et al. (2021) Q.-H. Cao, H.-r. Jiang, and G. Zeng (2021), eprint 2105.04464.
  * Olive et al. (2014) K. A. Olive et al. (Particle Data Group), Chin. Phys. C 38, 090001 (2014).
  * Cirigliano et al. (2016) V. Cirigliano, W. Dekens, J. de Vries, and E. Mereghetti, Phys. Rev. D 94, 034031 (2016), eprint 1605.04311.
  * Goldouzian et al. (2020) R. Goldouzian, J. H. Kim, K. Lannon, A. Martin, K. Mohrman, and A. Wightman (2020), eprint 2012.06872.
  * Abdallah et al. (2004) J. Abdallah et al. (DELPHI), Eur. Phys. J. C 35, 159 (2004), eprint hep-ex/0406010.
  * Rizzo (1995) T. G. Rizzo, Phys. Rev. D 51, 3811 (1995), eprint hep-ph/9409460.
  * Alcaraz et al. (2006) J. Alcaraz et al. (ALEPH, DELPHI, L3, OPAL, LEP Electroweak Working Group) (2006), eprint hep-ex/0612034.
  * Escribano and Masso (1994) R. Escribano and E. Masso, Nucl. Phys. B 429, 19 (1994), eprint hep-ph/9403304.
  * Parker et al. (2018) R. H. Parker, C. Yu, W. Zhong, B. Estey, and H. Müller, Science 360, 191–195 (2018), ISSN 1095-9203, URL http://dx.doi.org/10.1126/science.aap7706.
  * Hanneke et al. (2008) D. Hanneke, S. Fogwell, and G. Gabrielse, Physical Review Letters 100 (2008), ISSN 1079-7114, URL http://dx.doi.org/10.1103/PhysRevLett.100.120801.
  * Hanneke et al. (2011) D. Hanneke, S. Fogwell Hoogerheide, and G. Gabrielse, Physical Review A 83 (2011), ISSN 1094-1622, URL http://dx.doi.org/10.1103/PhysRevA.83.052122.
  * Kinoshita and Nio (2006) T. Kinoshita and M. Nio, Phys. Rev. D 73, 013003 (2006), URL https://link.aps.org/doi/10.1103/PhysRevD.73.013003.
  * Bennett et al. (2006) G. W. Bennett, B. Bousquet, H. N. Brown, G. Bunce, R. M. Carey, P. Cushman, G. T. Danby, P. T. Debevec, M. Deile, H. Deng, et al., Physical Review D 73 (2006), ISSN 1550-2368, URL http://dx.doi.org/10.1103/PhysRevD.73.072003.
  * Davier et al. (2017) M. Davier, A. Hoecker, B. Malaescu, and Z. Zhang, The European Physical Journal C 77 (2017), ISSN 1434-6052, URL http://dx.doi.org/10.1140/epjc/s10052-017-5161-6.
  * Keshavarzi et al. (2018) A. Keshavarzi, D. Nomura, and T. Teubner, Phys. Rev. D 97, 114025 (2018), URL https://link.aps.org/doi/10.1103/PhysRevD.97.114025.
  * Abi et al. (2021) B. Abi et al. (Muon g-2), Phys. Rev. Lett. 126, 141801 (2021), eprint 2104.03281.
  * Heister et al. (2003) A. Heister et al. (ALEPH), Eur. Phys. J. C 30, 291 (2003), eprint hep-ex/0209066.
  * Gonzalez-Sprinberg et al. (2000) G. A. Gonzalez-Sprinberg, A. Santamaria, and J. Vidal, Nucl. Phys. B 582, 3 (2000), eprint hep-ph/0002203.
  * CMS (2018) CMS (2018), eprint CMS-PAS-EXO-17-019.
  * Aaboud et al. (2018) M. Aaboud et al. (ATLAS), Phys. Rev. D 98, 032015 (2018), eprint 1805.01908.
  * Aad et al. (2020) G. Aad et al. (ATLAS), Phys. Rev. Lett. 125, 251802 (2020), eprint 2008.05928.
  * Shi et al. (2019) L. Shi, Z. Liang, B. Liu, and Z. He, Chin. Phys. C 43, 043001 (2019), eprint 1811.02261.
  * Khanpour et al. (2017) H. Khanpour, S. Khatibi, and M. Mohammadi Najafabadi, Phys. Lett. B 773, 462 (2017), eprint 1702.05753.
  * Alwall et al. (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), eprint 1405.0301.
  * Sjostrand et al. (2008) T. Sjostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun. 178, 852 (2008), eprint 0710.3820.
  * de Favereau et al. (2014) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP 02, 057 (2014), eprint 1307.6346.
  * Chatrchyan et al. (2013) S. Chatrchyan et al. (CMS), JINST 8, P04013 (2013), eprint 1211.4462.
  * Aad et al. (2016) G. Aad et al. (ATLAS), JINST 11, P04008 (2016), eprint 1512.01094.
  * Kovalchuk and Korchin (2017) V. A. Kovalchuk and A. Y. Korchin, Ukr. J. Phys. 62, 557 (2017), eprint 2006.09672.
  * Andersen et al. (2013) J. R. Andersen et al. (LHC Higgs Cross Section Working Group) (2013), eprint 1307.1347.
  * Cacciari et al. (2008) M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008), eprint 0802.1189.


# Motion Aware Self-Supervision for Generic Event Boundary Detection

Ayush K. Rai, Tarun Krishna, Julia Dietlmeier, Kevin McGuinness∗, Alan F.
Smeaton∗, Noel E. O’Connor∗
Insight SFI Centre for Data Analytics, Dublin City University (DCU)
<EMAIL_ADDRESS>
###### Abstract

The task of Generic Event Boundary Detection (GEBD) aims to detect moments in
videos that are naturally perceived by humans as generic and taxonomy-free
event boundaries. Modeling the dynamically evolving temporal and spatial
changes in a video makes GEBD a difficult problem to solve. Existing
approaches involve very complex and sophisticated pipelines in terms of
architectural design choices, hence creating a need for more straightforward
and simplified approaches. In this work, we address this issue by revisiting a
simple and effective self-supervised method and augment it with a
differentiable motion feature learning module to tackle the spatial and
temporal diversities in the GEBD task. We perform extensive experiments on the
challenging Kinetics-GEBD and TAPOS datasets to demonstrate the efficacy of
the proposed approach compared to the other self-supervised state-of-the-art
methods. We also show that this simple self-supervised approach learns motion
features without any explicit motion-specific pretext task. Our results can be
reproduced on github.

**footnotetext: Equal supervision Figure 1: The overall architecture consists
of two stages: a) Stage 1 involves the pre-training of the modified ResNet50
encoder (augmented with a MotionSqueeze layer) with four pretext tasks using a
contrastive learning based objective; b) Stage 2 consists of fine tuning of
the encoder on the downstream GEBD task. Refer to Table 4 in Supplementary
material for encoder details.

##  1 Introduction

Modeling videos using deep learning methods in order to learn effective global
and local video representations is an extremely challenging task. Current
state-of-the-art video models [18] are built upon a limited set of predefined
action classes and usually process short clips followed by a pooling operation
to generate global video-level predictions. Other mainstream computer vision
tasks for video processing have been mainly focused on action anticipation
[56, 1], temporal action detection [7, 22], temporal action segmentation [43,
41] and temporal action parsing [62, 67]. However, only limited attention has
been given to understanding long form videos. Cognitive scientists [74] have
observed that humans perceive videos by breaking them down into shorter
temporal units, each carrying a semantic meaning and can also reason about
them. This creates a need to investigate research problems to detect temporal
boundaries in videos that is consistent with their semantic validity and
interpretability from a cognitive point of view.

To this end, the GEBD task was recently introduced in [68]111LOVEU@CVPR2021,
LOVEU@CVPR2022 with an objective to study the long form video understanding
problem through the lens of a human perception mechanism. GEBD aims at
identifying changes in content, independent of changes in action, brightness,
object, etc., i.e. generic event boundaries, making it different to tasks such
as video localization [77]. Video events could indicate completion of goals or
sub-goals, or occasions where it becomes difficult for humans to predict what
will happen next. The recently released Kinetics-GEBD dataset [68] is the
first dataset specific to the GEBD task. It is annotated by 5 different event
boundary annotators, thereby capturing the subtlety involved in human
perception and making it the dataset with the greatest number of temporal
boundaries ($8\times$ EPIC-Kitchen-100 [11] and $32\times$ ActivityNet [16]).
The primary challenge in the GEBD task is to effectively model generic spatial
and temporal diversity as described in DDM-Net [72]. Spatial diversity is
primarily the result of both low-level changes, e.g. changes in brightness or
appearance, and high-level changes, e.g., changes in camera angle, or
appearance and disappearance of the dominant subject. Temporal diversity, on
the other hand, can be attributed to changes in action or changes by the
object of interaction with different speeds and duration, depending on the
subject. These spatio-temporal diversities make GEBD a difficult problem to
address.

In this work, to address the biased nature of video models trained over
predefined classes in a supervised setting, and the spatial diversity in GEBD,
we leverage the power of self-supervised models. Self-supervised techniques
like TCLR [12] and CCL [38] have achieved breakthrough results on various
downstream tasks for video understanding. The representations learned using
self-supervised learning (SSL) methods are not biased towards any predefined
action class making SSL methods an ideal candidate for the GEBD task. In
addition, in order to characterize temporal diversity in GEBD, learning motion
information is essential to capture the fine-grained temporal variations that
occur during the change of action scenarios. Previous methods in video
modeling learn temporal motion cues by pre-computing the optical flow [53, 52,
54] between consecutive frames, which is done externally and requires
substantial computation. Alternatively, methods such as those described in
[32, 21] estimate optical flow internally by learning visual correspondences
between images. The motion features learnt on-the-fly can also be used for
downstream applications such as action recognition as illustrated in [82, 42].

This presents an interesting research question: how can we develop a SSL
framework for video understanding that accounts for both appearance and motion
features? Do we need an explicit motion-specific training objective or can
this be implicitly achieved? We answer these questions by rethinking SSL by
reformulating the training objective proposed in VCLR [40] at clip-level and
further integrating it with a differentiable motion estimation layers using
the MotionSqueeze (MS) module introduced in [42] to jointly learn appearance
and motion features for videos. To summarise, the main contributions of our
work are as follows:

  * •

We revisit a simple self-supervised method VCLR [40] with a noticeable change
by modifying its pretext tasks by splitting them into frame-level and clip-
level to learn effective video representations (cVCLR) . We further augment
the encoder with a differentiable motion feature learning module for GEBD.

  * •

We conduct exhaustive evaluation on the Kinetics-GEBD and TAPOS datasets and
show that our approach achieves comparable performance to the self-supervised
state-of-the-art methods without using enhancements like model ensembles,
pseudo-labeling or the need for other modality features (e.g. audio).

  * •

We show that the model can learn motion features under self-supervision even
without having any explicit motion specific pretext task.

##  2 Related Work

###  2.1 Generic Event Boundary Detection.

The task of GEBD [68] is similar in nature to the Temporal Action Localization
(TAL) task, where the goal is to localize the start and end points of an
action occurrence along with the action category. Initial attempts to address
GEBD were inspired from popular TAL solvers including the boundary matching
networks (BMN) [52] and BMN-StartEnd [68], which generates proposals with
precise temporal boundaries along with reliable confidence scores. Shou et al.
[68] introduced a supervised baseline Pairwise Classifier (PC), which
considers GEBD as a framewise binary classification problem (boundary or not)
by having a simple linear classifier that uses concatenated average features
around the neighbourhood of a candidate frame. However, since GEBD is a new
task, most of the current methods are an extension of state-of-the-art video
understanding tasks, which overlook the subtle differentiating characteristics
of GEBD. Hence there is a necessity for GEBD specialized solutions.

DDM-Net [72] applied progressive attention on multi-level dense difference
maps (DDM) to characterize motion patterns and jointly learn motion with
appearance cues in a supervised setting. However, we learn generic motion
features by augmenting the encoder with a MS module in a self-supervised
setting. Hong et al. [29] used a cascaded temporal attention network for GEBD,
while Rai et al. [64] explored the use of spatio-temporal features using two-
stream networks. Li et al. [49] designed an end-to-end spatial-channel
compressed encoder and temporal contrastive module to determine event
boundaries. Recently, SC-Transformer [48] introduced a structured partition of
sequences (SPoS) mechanism to learn structured context using a transformer
based architecture for GEBD and augmented it with the computation of group
similarity to learn distinctive features for boundary detection. One advantage
of SC-Transformer is that it is independent of video length and predicts all
boundaries in a single forward pass by feeding in 100 frames, however it
requires substantial memory and computational resources.

Regarding unsupervised GEBD approaches, a shot detector library222
https://github.com/Breakthrough/PySceneDetect and PredictAbility (PA) have
been investigated in [68]. The authors of UBoCo [36, 35] proposed a novel
supervised/unsupervised method that applies contrastive learning to a TSM333
Temporal Self-Similarity Matrix based intermediary representation of videos to
learn discriminatory boundary features. UBoCo’s recursive TSM3 parsing
algorithm exploits generic patterns and detects very precise boundaries.
However, they pre-process all the videos in the dataset to have the same
frames per second (fps) value of 24, which adds a computational overhead.
Furthermore, like the SC-Transformer, UBoCo inputs the frames representing the
whole video at once, whereas in our work we use raw video signals for pre-
training and only the context around the candidate boundary as input to the
GEBD task. TeG [63] proposed a generic self-supervised model for video
understanding for learning persistent and more fine-grained features and
evaluated it on the GEBD task. The main difference between TeG and our work is
that TeG uses a 3D-ResNet-50 encoder as their backbone, which makes the
training computationally expensive, whereas we use 2D-ResNet-50 model and
modify it by adding temporal shift module (TSM444Temporal Shift Module) [51]
to achieve the same effect as 3D convolution while keeping the complexity of a
2D CNN.

GEBD can be used as a preliminary step in a larger downstream application,
e.g. video summarization, video captioning [76], or ad cue-point detection
[8]. It is, therefore, important that the GEBD model not add excessive
computational overhead to the overall pipeline, unlike many of the examples of
related work presented here.

###  2.2 SSL for video representation learning.

Self-supervision has become the new norm for learning representations given
its ability to exploit unlabelled data [59, 23, 15, 2, 5, 81, 4, 9, 60, 39,
14]. Recent approaches devised for video understanding can be divided into two
categories based on the SSL objective, namely pretext task based and
contrastive learning based.

Pretext task based. The key idea here is to design a pretext task for which
labels are generated in an online fashion, referred to as pseudo labels,
without any human annotation. Examples include: predicting correct temporal
order [58], Video Rot-Net [34] for video rotation prediction, clip order
prediction [78], odd-one-out networks [20], sorting sequences [45], and pace
prediction [75]555leverages contrastive learning as an additional objective as
well.. All these approaches exploit raw spatio-temporal signals from videos in
different ways based on pretext tasks and consequently learn representations
suitable for varied downstream tasks.

Contrastive learning based. Contrastive learning approaches bring semantically
similar objects, clips, etc., close together in the embedding space while
contrasting them with negative samples, using objectives based on some variant
of Noise Contrastive Estimation (NCE) [24]. The Contrastive Predictive Coding
(CPC) approach [60] for images was extended to videos in DPC [26] and MemDPC
[27], which augments DPC with the notion of compressed memory. Li et al. [73]
extends the contrasting mutli-view framework for inter-intra style video
representation, while Kong et al. [38] combine ideas from cycle-consistency
with contrastive learning to propose cycle-contrast. Likewise, Yang et al.
[79] exploits visual tempo in a contrastive framework to learn spatio-temporal
features. Similarly, [12, 3] use temporal cues with contrastive learning. VCLR
[40] formulates a video-level contrastive objective to capture global context.
In the work presented here, we exploit VCLR as our backbone objective.
However, different to pretext tasks in VCLR, which perform computation only on
frame level, we modify those pretext tasks to not only operate on frame-level
but also on clip-level thereby leading to better modeling of the spatio-
temporal features in videos. See [66] for a more extensive review of SSL
methods for video understanding.

###  2.3 Motion estimation and Learning visual correspondences for video
understanding.

Motion estimation.  Two-stream architectures [19, 69] have exhibited promising
performance on the action recognition task by using pre-computed optical flow,
although such approaches reduce the efficiency of video processing. Several
other methods [17, 33, 47, 61, 71] have proposed architectures that learn
motion internally in an end-to-end fashion. The work presented in [50, 80]
introduced a motion-specific contrastive learning task to learn motion
features in a self-supervised setting.

Learning visual correspondences.  Many recent works have proposed to learn
visual correspondences between images using neural networks [21, 25, 46, 57,
65, 70]. Regarding learning correspondences for video understanding, CPNet
[55] introduced a network that learns representations of videos by mixing
appearance and long-range motion features from an RGB input only. Zhao et al.
[82] proposed a method that learns a disentangled representation of a video,
namely static appearance, apparent motion and appearance change from RGB input
only. MotionSqueeze (MS) [42] introduced an end-to-end trainable, model-
agnostic and lightweight module to extract motion features that does not
require any correspondence supervision for learning.

##  3 Method

In order to apply a contrastive learning framework to videos specifically for
generic event boundary detection, we follow the framework proposed by VCLR
[40] and make noticeable modifications to it. For simplicity the notations are
kept similar to [40] unless otherwise explicitly stated.

###  3.1 SSL for Video Representation Learning

1: Contrastive encoder. Our processing backbone is a ResNet-50 based encoder
equipped with four pretext tasks, as defined in VCLR [40], trained following a
contrastive objective as defined in MoCo-V2 [10].

Let $\mathbf{x}_{p}=\mathcal{T}(\mathbf{x}_{q}$) be an augmented view of an
anchor image $\mathbf{x}_{q}$ with $\mathcal{T}\sim\mathcal{P}$
($\mathcal{P}=$ {random scaling, color-jitter, random grayscale, random
Gaussian blur, and random horizontal flip} being set of augmentations) and
$\mathcal{N}^{-}$ negative samples. $\mathbf{x}_{q}$ and $\mathbf{x}_{p}$ are
processed through query ($f_{q}(\mathbf{x}_{q})$) and a key
$(f_{k}(\mathbf{x}_{p}))$ encoder respectively. In addition, these encoders
are appended with projection heads (MLP layers) to get low dimensional
representations of inputs i.e.
$q=g_{q}(f_{q}(\mathbf{x}_{q})),p=g_{k}(f_{k}(\mathbf{x}_{p}))$. The overall
objective can be optimized by InfoNCE loss [60]:

$\mathcal{L}_{\text{NCE}}(q,p,\mathcal{N^{-}})=-\log\Bigg{(}\frac{e^{\text{sim}({q,p})}}{e^{\text{sim}({q,p})}+\sum_{j=1}^{\text{N}}e^{\text{sim}({q,n_{j}})}}\Bigg{)},$
(1)

where, $\mathrm{sim}(\cdot,\cdot)$ is a similarity function. We note that
$g_{q}$ and $g_{k}$ can be thought of as a task-specific projection heads with
further details below666subscript $q$, $k$ in $f$ and $g$ represents query and
key.

2: Pre-text setup. In order to capture different generic subtle nuances
(spatial variations, temporal coherency, long range dependencies) for video
understanding, the pretext tasks defined in VCLR [40] are a good candidate for
pre-training as they serve the purpose of capturing such semantics for
powerful video representation from raw video signals. We alter the pretext
task setup in VCLR to ensure that Intra and Inter instance discrimination (ID)
tasks operate at frame-level while computation of the video segment ID and
temporal order regularization tasks occurs at clip-level. Below we elaborate
on the intuition behind this notion.

For frame-level pretext tasks, consider three randomly selected frames from a
video, $v_{1}$, $v_{2}$ and $v_{3}$. $v_{1}$ undergoes different augmentations
to generate $v_{1}^{a}$ and $v_{1}^{+}$. $\mathcal{N}^{-}$ represents negative
samples from other videos. $v_{1}^{a}$ is processed through query encoder
$f_{q}(v_{1}^{a})$, while ($v_{1}^{+}$, $v_{2}$, $v_{3}$) is processed through
key encoder $f_{k}(\cdot)$. While depending upon the pretext, the projection
head varies across the tasks.

For clip-level pretext tasks, a video $V$ is divided into $K$ (set to 3)
segments $\\{S_{1},S_{2},...,S_{K}\\}$ of equal duration. Two tuples,
comprising of 4 frame long clip, are randomly and independently sampled from
each of these segments to form an anchor tuple and a positive tuple. For
instance, let $c_{k}=\\{u_{1},u_{2},u_{3},u_{4}\\}$ where $c_{k}$ denote the
ordered clip sampled from the $k^{\text{th}}$ segment while $u_{1}.....u_{4}$
represent the frames in that clip. Similarly the anchor and positive tuple are
given by $t^{a}=\\{c_{1}^{a},c_{2}^{a},\ldots c_{K}^{a}\\}$ and
$t^{+}=\\{c_{1}^{+},c_{2}^{+},\dots c_{K}^{+}\\}$ respectively.

a. Intra-frame ID task. In order to model spatial diversity for the GEBD task,
we adopt the intra-frame instance discrimination task proposed in VCLR [40] to
model inherent spatial changes across frames. For this task only $v_{1}^{+}$
is considered as a positive example while $v_{2}$ and $v_{3}$ represent
negative examples. MLP heads are given by $g_{q}^{r}$ and $g_{k}^{r}$ while
anchor embedding $q_{1}^{a}=g_{q}^{r}(f_{q}(v_{1}^{a}))$, positive embedding
as $p_{1}^{+}=g_{k}^{r}(f_{k}(v_{1}^{+}))$ and the negative sample777Note:
Negative samples comes from the same video i.e. two samples as shown in Eq.
(2). embeddings $p_{2}=g_{k}^{r}(f_{k}(v_{2}))$,
$p_{3}=g_{k}^{r}(f_{k}(v_{3}))$. The loss objective is given by:

$\mathcal{L}_{\text{Intra}}=\mathcal{L}_{\text{NCE}}(q_{1}^{a},p_{1}^{+},\\{p_{2},p_{3}\\}).$
(2)

b. Inter-frame ID task. Detecting generic event boundaries requires encoding
fine-grained temporal structures from a coherent action, which are consistent
with each other. To model this, inter-frame instance discrimination task
considers $v_{1}^{a}$ as an anchor frame and ($v_{1}^{+}$, $v_{2}$, $v_{3}$)
as positive samples while $\mathcal{N}^{-}$ as negative samples. $g_{q}^{e}$
and $g_{k}^{e}$ are MLP projection heads which output the anchor embedding
$q_{1}^{a}=g_{q}^{e}(f_{q}(v_{1}^{a}))$ and positive embeddings as
$p_{1}^{+}=g_{k}^{e}(f_{k}(v_{1}^{+}))$, $p_{2}=g_{k}^{e}(f_{k}(v_{2}))$,
$p_{3}=g_{k}^{e}(f_{k}(v_{3}))$. Let
$p^{\prime}\in\\{p_{1}^{+},p_{2},p_{3}\\}$, hence the inter objective becomes:

$\displaystyle\mathcal{L}_{\text{Inter}}=\frac{1}{3}\sum_{p^{\prime}}\mathcal{L}_{\text{NCE}}(q_{1}^{a},p^{\prime},\mathcal{N^{-}}).$
(3)

c. Video segment based ID task. Learning long range temporal diversity in a
video is also crucial for the GEBD task. For capturing the evolving semantics
in the temporal dimension we need to incorporate global video level
information. The contrastive loss objective is chosen in a way that each clip
in the clip anchor and clip positive tuples i.e. $t^{a}$ and $t^{+}$ learn a
video-level embedding through consensus operation (denoted by $\mathcal{C}$)
e.g. average. Mathematically this can be represented as:

$q_{t}^{a}=g_{p}^{s}(\mathcal{C}\mid\mathcal{C}(f_{q}(c_{1}^{a})),\mathcal{C}(f_{q}(c_{2}^{a})),\ldots,\mathcal{C}(f_{q}(c_{K}^{a}))),$
(4)
$p_{t}^{+}=g_{k}^{s}(\mathcal{C}\mid\mathcal{C}(f_{k}(c_{1}^{+})),\mathcal{C}(f_{k}(c_{2}^{+})),\ldots,\mathcal{C}(f_{k}(c_{K}^{+}))),$
(5)
$\mathcal{L}_{\text{Segment}}=\mathcal{L}_{\text{NCE}}(q_{t}^{a},p_{t}^{+},\mathcal{N}^{-}).$
(6)

Here, $g_{q}^{s}$ and $g_{k}^{s}$ represent the MLP heads,
$\mathcal{C}(f_{q}(c_{k}^{a}))$ indicates the average over the encoder
representation of the individual frame in the $k^{\text{th}}$ clip while
$q_{t}^{a}$, $p_{t}^{+}$ denotes the final embeddings for anchor and positive
clip tuples. The video-level contrastive loss is given by
$\mathcal{L}_{\text{Segment}}$.

d. Temporal order regularization task. In order to enforce inherent sequential
structure on videos for signalling supervision in self-supervised video
representation learning, we need a pretext task to learn the correct temporal
order of the video data. This can also be attained through pretext tasks
proposed in [20, 75]. However, in this work we restrict ourselves to use the
temporal ordering as a regularization term (denoted by
$\mathcal{L}_{\text{Order}}$) within the contrastive framework as explained in
Section 3.3 in [40] though we reformulate it to include clip-level
computation.

The modifications made to video segment based ID task and temporal order
regularization task in VCLR to incorporate clip-level computation is referred
to as cVCLR (clip-VCLR).

###  3.2 Motion Estimation

For learning motion features we use the MotionSqueeze (MS) module presented in
[42], a learnable motion feature extractor that can be inserted into any video
understanding architecture to learn motion features and replace the external
computation of optical flow. The motion features are learned in three steps:

1: Correlation computation. Consider $F^{(t)}$ and $F^{(t+1)}$ represent two
adjacent input feature maps of spatial resolution $H\times W$ and channel
dimension $C$. The correlation tensor $\mathtt{S}^{(t)}$ is computed by
calculating correlation score of every spatial position $\mathtt{x}$ with
respect to displacement $\mathtt{p}$ following the correlation layer
implementation in FlowNet [21]. The correlation for position $\mathtt{x}$ is
only computed in neighborhood size $\mathtt{P}=2l+1$ by restricting a maximum
displacement $\mathtt{p}\in[-l,l]^{2}$ and the value of $\mathtt{P}$ is set to
$15$.

2: Displacement estimation. The next step involves estimating the displacement
map of size $H\times W\times 2$ from the correlation tensor
$\mathtt{S}^{(t)}$. To get the best matching displacement for position
$\mathtt{x}$, kernal-soft-argmax [46] is used. In addition, a motion
confidence map (of size $H\times W\times 1$) of correlation as auxiliary
motion information is obtained by pooling the highest correlation on each
position $\mathtt{x}$ as described in [42]. The motion confidence map helps in
identifying displacement outliers and learn informative motion features. The
displacement map is then concatenated with the motion confidence map to create
a displacement tensor $\mathbf{D}^{(t)}$ of size $H\times W\times 3$.

3: Feature transformation.  In order to convert displacement tensor
$\mathbf{D}^{(t)}$ to a relevant motion feature $\mathbf{M}^{(t)}$ (with same
channel dimension $C$ as input $\mathbf{F}^{(t)}$) , $\mathbf{D}^{(t)}$ is
passed through four depth-wise separable convolutions [30] similar to [42].
Contrary to [42], in our work, we apply this feature transformation in a self-
supervision setting to learn displacement tensor and motion confidence map
(generic motion features). Finally, the motion features $\mathbf{M}^{(t)}$ are
added to the input of the next layer using an element-wise addition operation
: $\mathbf{F}^{\prime(t)}=\mathbf{F}^{(t)}+\mathbf{M}^{(t)}$. The resulting
fused feature $\mathbf{F}^{\prime(t)}$ is passed as input to the next layer.
For more details we refer the readers to [42].

###  3.3 Optimisation

The overall contrastive loss objective is given by:

$\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{Inter}}+\mathcal{L}_{\text{Intra}}+\mathcal{L}_{\text{Segment}}+\mathcal{L}_{\text{Order}}.$
(7)

Our encoder is augmented with a MS module (introduced after conv3_x888notation
as in [28]) to jointly learn appearance and motion features. More precisely, a
ResNet-50 [28] model is adopted as the CNN encoder and we insert a TSM4 [51]
for each residual block of ResNet. Each of the four losses contribute equally
to $\mathcal{L}_{\text{total}}$ although weighing them appropriately might
boost the performance. The overall framework is illustrated in Figure 1.

Table 1: F1 scores on the Kinetics-GEBD validation set with Rel. Dis threshold
ranging from 0.05 to 0.5 with step of 0.05. $\ddagger$: soft-labels,
$\dagger$: hard-labels. ∗ is pretrained on Kinetics-400 [37] dataset.

Rel. Dis Threshold | Finetuning | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | avg
---|---|---|---|---|---|---|---|---|---|---|---|---
| BMN† [52] | ✗ | 0.186 | 0.204 | 0.213 | 0.220 | 0.226 | 0.230 | 0.233 | 0.237 | 0.239 | 0.241 | 0.223
| BMN-SE [52]† | ✗ | 0.491 | 0.589 | 0.627 | 0.648 | 0.660 | 0.668 | 0.674 | 0.678 | 0.681 | 0.683 | 0.640
| TCN-TAPOS [44]† | ✓ | 0.464 | 0.560 | 0.602 | 0.628 | 0.645 | 0.659 | 0.669 | 0.676 | 0.682 | 0.687 | 0.627
| TCN [44]† | ✗ | 0.588 | 0.657 | 0.679 | 0.691 | 0.698 | 0.703 | 0.706 | 0.708 | 0.710 | 0.712 | 0.685
Supervised | PC [68]† | ✓ | 0.625 | 0.758 | 0.804 | 0.829 | 0.844 | 0.853 | 0.859 | 0.864 | 0.867 | 0.870 | 0.817
| SBoCo-Res50 [36]† | ✗ | 0.732 | - | - | - | - | - | - | - | - | - | 0.866
| SBoCo-TSN [36]†,∗ | ✗ | 0.787 | - | - | - | - | - | - | - | - | - | 0.892
| DDM-Net [72]† | ✗ | 0.764 | 0.843 | 0.866 | 0.880 | 0.887 | 0.892 | 0.895 | 0.898 | 0.900 | 0.902 | 0.873
| Li et.al. [49]‡ | ✗ | 0.743 | 0.830 | 0.857 | 0.872 | 0.880 | 0.886 | 0.890 | 0.893 | 0.896 | 0.898 | 0.865
|  SC-Transformer [48]‡ | ✗ | 0.777 | 0.849 | 0.873 | 0.886 | 0.895 | 0.900 | 0.904 | 0.907 | 0.909 | 0.911 | 0.881
| SceneDetect [6] | ✗ | 0.275 | 0.300 | 0.312 | 0.319 | 0.324 | 0.327 | 0.330 | 0.332 | 0.334 | 0.335 | 0.318
| PA - Random [68] | ✗ | 0.336 | 0.435 | 0.484 | 0.512 | 0.529 | 0.541 | 0.548 | 0.554 | 0.558 | 0.561 | 0.506
| PA [68] | ✗ | 0.396 | 0.488 | 0.520 | 0.534 | 0.544 | 0.550 | 0.555 | 0.558 | 0.561 | 0.564 | 0.527
Un-supervised | UBoCo-Res50 [36] | ✗ | 0.703 | - | - | - | - | - | - | - | - | - | 0.866
| UBoCo-TSN [36]∗ | ✗ | 0.702 | - | - | - | - | - | - | - | - | - | 0.892
| TeG-PS [63]† | ✓ | 0.699 | - | - | - | - | - | - | - | - | - | -
(Self-supervised) | TeG-FG [63]† | ✓ | 0.714 | - | - | - | - | - | - | - | - | - | -
| Ours† | ✓ | 0.680 | 0.779 | 0.806 | 0.818 | 0.825 | 0.830 | 0.834 | 0.837 | 0.839 | 0.841 | 0.809
| Ours‡ | ✓ | 0.711 | 0.777 | 0.791 | 0.795 | 0.798 | 0.799 | 0.801 | 0.802 | 0.802 | 0.803 | 0.788

##  4 Experimental Setup

###  4.1 Implementation Details.

Stage 1: Pre-training. We closely follow VCLR [40] to train the encoder. The
model was pre-trained end-to-end with the objective as defined in Eq. (7) on 2
NVIDIA GeForce RTX-2080Ti GPUs with an effective batch size ($\mathcal{B}$) of
8 distributed across the GPUs (4 each) with temperature set to 0.01 across all
pretext tasks. The input to the frame level and clip level pretext task is
($\mathcal{B},3,3,224,224$) and ($\mathcal{B},K,4,3,224,224$) respectively
with $K=3$. TSM4 and MotionSqueeze is only applied on clip level task. The
encoder is initialised to MoCo-v2 [10] weights with negative samples
$\mathcal{N}^{-}$ (queue size) set to 8192 and is trained with SGD for 400
epochs with a warm-start of 5 epochs following a cosine decay with base
learning rate of 0.01. Pre-training is only performed on the Kinetics-GEBD
[68] dataset.

Stage 2: Finetuning. Input to the encoder is based on the temporal window
($W$=5) which defines a context over a candidate frame (before and after) with
a stride $m=3$999selecting one frame out of every 3 consecutive frames
resulting into a 4D tensor $(10,3,224,224)$ as input. We fine-tune the model
end-to-end with a binary cross entropy (BCE) (boundary is 0/1) as the
objective augmented with Gaussian smoothing ($\sigma=3$) for soft labeling as
in [48]. The learning rate set to 7.5$e^{-4}$ for Kinetics-GEBD, while for
TAPOS it was set to 1$e^{-4}$. Balance sampling is applied to each batch
during training to avoid class imbalance. We finetune the model for 8 epochs
and use early stopping to find the best model. More details in supplementary
material.

To select the final boundary predictions for the video, we apply post-
processing scheme on the obtained boundary proposals. First, proposals should
be greater then a threshold of 0.5. Second, we aggregate all the proposals
within a 1 second time window.

Table 2: F1 scores on the TAPOS validation set with Rel. Dis threshold ranging
from 0.05 to 0.5 with step of 0.05. $\ddagger$: soft-labels, $\dagger$: hard-
labels. (-) : Not clear.

Rel. Dis Threshold | Finetuning | 0.05 | 0.1 | 0.15 | 0.2 | 0.25 | 0.3 | 0.35 | 0.4 | 0.45 | 0.5 | avg
---|---|---|---|---|---|---|---|---|---|---|---|---
| ISBA [13] | - | 0.106 | 0.170 | 0.227 | 0.265 | 0.298 | 0.326 | 0.348 | 0.369 | 0.376 | 0.384 | 0.314
| TCN [44] | ✗ | 0.237 | 0.312 | 0.331 | 0.339 | 0.342 | 0.344 | 0.347 | 0.348 | 0.348 | 0.348 | 0.330
| CTM [31] | - | 0.244 | 0.312 | 0.336 | 0.351 | 0.361 | 0.369 | 0.374 | 0.381 | 0.383 | 0.385 | 0.350
Supervised | Transparser [67] | - | 0.289 | 0.381 | 0.435 | 0.475 | 0.500 | 0.514 | 0.527 | 0.534 | 0.540 | 0.545 | 0.474
| PC [68]† | ✓ | 0.522 | 0.595 | 0.628 | 0.647 | 0.660 | 0.666 | 0.672 | 0.676 | 0.680 | 0.684 | 0.643
| DDM-Net [72]† | ✗ | 0.604 | 0.681 | 0.715 | 0.735 | 0.747 | 0.753 | 0.757 | 0.760 | 0.763 | 0.767 | 0.728
|  SC-Transformer [48]‡ | ✗ | 0.618 | 0.694 | 0.728 | 0.749 | 0.761 | 0.767 | 0.771 | 0.774 | 0.777 | 0.780 | 0.742
| SceneDetect [6] | ✗ | 0.035 | 0.045 | 0.047 | 0.051 | 0.053 | 0.054 | 0.055 | 0.056 | 0.057 | 0.058 | 0.051
Un-supervised | PA - Random [68] | ✗ | 0.158 | 0.233 | 0.273 | 0.310 | 0.331 | 0.347 | 0.357 | 0.369 | 0.376 | 0.384 | 0.314
| PA [68] | ✗ | 0.360 | 0.459 | 0.507 | 0.543 | 0.567 | 0.579 | 0.592 | 0.601 | 0.609 | 0.615 | 0.543
(Self-supervised) | Ours† | ✓ | 0.573 | 0.614 | 0.639 | 0.656 | 0.669 | 0.679 | 0.687 | 0.693 | 0.700 | 0.704 | 0.661
| Ours‡ | ✓ | 0.586 | 0.624 | 0.648 | 0.663 | 0.675 | 0.685 | 0.692 | 0.697 | 0.704 | 0.708 | 0.668

Figure 2: Qualitative Analysis I: visualization of some detected boundaries on
the validation set of Kinetics-GEBD. Compared with baseline PC [68], our
method produces more precise boundaries that are consistent with the ground
truth.

###  4.2 Results

We perform extensive quantitative and qualitative studies on the given
datasets. For evaluation protocol we refer the reader to supp. material. In
Tables 1 and 2, we report F1 scores for different thresholds ranging from 0.05
to 0.5 with a step of 0.05, for the Kinetics-GEBD and TAPOS datasets
respectively. On the Kinetics-GEBD dataset, our model outperforms the
supervised baseline PC [68] with Rel. Dis threshold 0.05 and is also
comparable with other state-of-the-art unsupervised/self-supervised GEBD
models like UBoCo [36] and TeG [63] in terms of performance. Table 1
illustrates the result on the Kinetics-GEBD dataset. On the TAPOS dataset,
which consists of Olympic sport videos with 21 action classes, we have a
similar observation. Our model outperforms the supervised baseline PC [68] at
the Rel. Dis threshold of 0.05 and is comparable on other thresholds. Other
state-of-the-art methods on TAPOS like DDM-Net [72] and SC-Transformer [48]
fall in the supervised category and cannot be directly compared with our
results. We were unable to find other state-of-the-art un/self-supervised
models for GEBD to directly compare our results with, on the TAPOS dataset
shown in Table 2.

We also perform a qualitative analysis of boundaries detected by our method
and compare them with the supervised baseline PC [68] and the ground truth
annotation in Figure 2. Figure 3 shows a visualization of the motion
confidence map learned by the MS module during pre-training. We observe that
motion confidence generalizes well to the TAPOS dataset too, which was not
used for pre-training. This validates that MS module learns general motion
features even in a self-supervised setting without any explicit motion
specific pretext task. In addition our model’s event boundary detection
results on the TAPOS dataset further justifies that our model is a generic
event boundary detector, as after fine-tuning it generalizes beyond Kinetics-
GEBD dataset to the TAPOS benchmark. We also found that linear evaluation
(freezing the encoder) on the downstream GEBD task resulted in poor
performance.

Note: For GEBD challenge specific reports, we refer the reader to challenge
page for 2021, 2022.

Figure 3: Qualitative Analysis II: visualization of the learned motion confidence map. The first two blocks (categories: jumping on trampoline and situp respectively) are taken from the Kinetics-GEBD dataset, while the bottom block (category: uneven bar) is derived from TAPOS. In each block, the first row shows the RGB frames while the second depicts the motion confidence map learnt by the model. Note: the model is only pre-trained on Kinetics-GEBD but it generalizes to the TAPOS dataset as well. Table 3: Ablation study on validation set of TAPOS and Kinetics-GEBD for F1 score at Rel. Dis threshold $0.05$ Method | TAPOS | Kinetics-GEBD
---|---|---
Vanilla VCLR | 0.496 | 0.596
cVCLR | 0.502 ($\uparrow$) | 0.605 ($\uparrow$)
\+ MotionSqueeze | 0.573 ($\uparrow$) | 0.680 ($\mathbf{\uparrow}$)
\+ Soft labels | 0.586 ($\uparrow$) | 0.711 ($\uparrow$)

###  4.3 Ablation Studies

1: Does MotionSqueeze through self-supervision helps? As shown in Figure 3,
the MS module shows high confidence in regions of images that are more
dynamic. The module learns optical flow in an online fashion even under self-
supervision. Intuitively, temporal order regularization and video segment
instance discrimination pre-text tasks implicitly complements the MS module to
learn generic motion features. From Table 3, we observe that by incorporating
the MS module, the<EMAIL_ADDRESS>score on the GEBD task increases by 7.5% on
Kinetics-GEBD and 7.1% on TAPOS, which is a significant increase.

2: Does soft labelling helps in boosting the performance? Kinetics-GEBD has 5
annotators to capture human perception differences but this introduces
ambiguity. Ideally the neighbouring frames of the candidate boundary frame
should also have a high value of the ground truth label. To tackle this issue,
we use Gaussian smoothing ($\sigma=3$) to create soft labels from hard labels,
which ensures that model avoids making over confident predictions for the
event boundary. As shown in Table 3, soft labels improves the<EMAIL_ADDRESS>score by
3.1% on Kinetics-GEBD and 1.3% on TAPOS dataset.

##  5 Conclusions and Discussion

In this work, we presented a self-supervised model that can be pre-trained for
the generic event boundary detection task. The GEBD task is an ideal problem
for self-supervised learning given that the task aims to learn generic
boundaries and is not biased towards any predefined action categories from
pre-trained state-of-the-art action recognition models. In order to learn
spatial and temporal diversity we reformulate SSL objective at frame-level and
clip-level to learn effective video representations (cVCLR). In addition we
augment our encoder with a MS module and find this indeed compliments the
overall performance on the downstream GEBD task. Furthermore, the motion
features learnt are generic since the model is only pre-trained on Kinetics-
GEBD but generalizes to TAPOS dataset as well. Through our extensive
evaluation, we achieve comparable performance to self-supervised state-of-the-
art methods on the Kinetics-GEBD as shown in Table 1.

However, there are limitations with this work. First, we have not used more
powerful models, e.g. transformers as in [48], or cascaded networks as in
[29]. Second, since MS module is directly applied on feature maps, it learns
global motion features. However, in GEBD the boundaries are generic and every
type of motion may not indicate a boundary, hence a more fine-grained motion
module can boost the performance. Third, due to computational constraints, our
self-supervised model is only pre-trained on the Kinetics-GEBD dataset;
however, pre-training the model on Kinetics-400 could yield even better
performance on the downstream GEBD task. These limitations will be addressed
in our future work.

##  6 Acknowledgement

This work has emanated from research supported by Science Foundation Ireland
(SFI) under Grant Number SFI/12/RC/2289_P2, co-funded by the European Regional
Development Fund and Xperi FotoNation.

## References

  * [1] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what?-anticipating temporal occurrences of activities.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5343–5352, 2018.
  * [2] Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi.  Self-labelling via simultaneous clustering and representation learning.  arXiv preprint arXiv:1911.05371, 2019.
  * [3] Yutong Bai, Haoqi Fan, Ishan Misra, Ganesh Venkatesh, Yongyi Lu, Yuyin Zhou, Qihang Yu, Vikas Chandra, and Alan Yuille.  Can temporal information help with contrastive self-supervised learning?  arXiv preprint arXiv:2011.13046, 2020.
  * [4] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  Advances in Neural Information Processing Systems, 33:9912–9924, 2020.
  * [6] Brandon Catellano.  Pyscenedetect: an intelligent scene cut detection and video splitting tool.  https://github.com/Breakthrough/PySceneDetect, 2014.
  * [7] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A. Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster R-CNN architecture for temporal action localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [8] Shixing Chen, Xiaohan Nie, David Fan, Dongqing Zhang, Vimal Bhat, and Raffay Hamid.  Shot contrastive self-supervised learning for scene boundary detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9796–9805, 2021.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [11] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al.  Scaling egocentric vision: The epic-kitchens dataset.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 720–736, 2018.
  * [12] Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah.  TCLR: Temporal contrastive learning for video representation.  Computer Vision and Image Understanding, 219:103406, 2022.
  * [13] Li Ding and Chenliang Xu.  Weakly-supervised action segmentation with iterative soft boundary assignment.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6508–6516, 2018.
  * [14] Yasser Abdelaziz Dahou Djilali, Tarun Krishna, Kevin McGuinness, and Noel E. O’Connor.  Rethinking 360deg image visual attention modelling with unsupervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 15414–15424, October 2021.
  * [15] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [16] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970, 2015.
  * [17] Lijie Fan, Wenbing Huang, Chuang Gan, Stefano Ermon, Boqing Gong, and Junzhou Huang.  End-to-end learning of motion representation for video understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6016–6025, 2018.
  * [18] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [19] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1933–1941, 2016.
  * [20] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould.  Self-supervised video representation learning with odd-one-out networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636–3645, 2017.
  * [21] Philipp Fischer, Alexey Dosovitskiy, Eddy Ilg, Philip Häusser, Caner Hazırbaş, Vladimir Golkov, Patrick Van der Smagt, Daniel Cremers, and Thomas Brox.  FlowNet: Learning optical flow with convolutional networks.  arXiv preprint arXiv:1504.06852, 2015.
  * [22] Jiyang Gao, Zhenheng Yang, and Ram Nevatia.  Cascaded boundary regression for temporal action detection.  arXiv preprint arXiv:1705.01180, 2017.
  * [23] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [24] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 297–304, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
  * [25] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong, Minsu Cho, Cordelia Schmid, and Jean Ponce.  SCNet: Learning semantic correspondence.  In Proceedings of the IEEE international conference on computer vision, pages 1831–1840, 2017.
  * [26] Tengda Han, Weidi Xie, and Andrew Zisserman.  Video representation learning by dense predictive coding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [27] Tengda Han, Weidi Xie, and Andrew Zisserman.  Memory-augmented dense predictive coding for video representation learning.  In European conference on computer vision, pages 312–329. Springer, 2020.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [29] Dexiang Hong, Congcong Li, Longyin Wen, Xinyao Wang, and Libo Zhang.  Generic event boundary detection challenge at cvpr 2021 technical report: Cascaded temporal attention network (castanet).  arXiv preprint arXiv:2107.00239, 2021.
  * [30] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [31] De-An Huang, Li Fei-Fei, and Juan Carlos Niebles.  Connectionist temporal modeling for weakly supervised action labeling.  In European Conference on Computer Vision, pages 137–153. Springer, 2016.
  * [32] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  FlowNet 2.0: Evolution of optical flow estimation with deep networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462–2470, 2017.
  * [33] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan.  STM: Spatiotemporal and motion encoding for action recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2000–2009, 2019.
  * [34] Longlong Jing, Xiaodong Yang, Jingen Liu, and Yingli Tian.  Self-supervised spatiotemporal feature learning via video rotation prediction.  arXiv preprint arXiv:1811.11387, 2018.
  * [35] Hyolim Kang, Jinwoo Kim, Kyungmin Kim, Taehyun Kim, and Seon Joo Kim.  Winning the CVPR’2021 kinetics-gebd challenge: Contrastive learning approach.  arXiv preprint arXiv:2106.11549, 2021.
  * [36] Hyolim Kang, Jinwoo Kim, Taehyun Kim, and Seon Joo Kim.  UBoCo: Unsupervised boundary contrastive learning for generic event boundary detection.  arXiv preprint arXiv:2111.14799, 2021.
  * [37] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [38] Quan Kong, Wenpeng Wei, Ziwei Deng, Tomoaki Yoshinaga, and Tomokazu Murakami.  Cycle-contrast for self-supervised video representation learning.  Advances in Neural Information Processing Systems, 33:8089–8100, 2020.
  * [39] Tarun Krishna, Kevin McGuinness, and Noel O’Connor.  Evaluating contrastive models for instance-based image retrieval.  In Proceedings of the 2021 International Conference on Multimedia Retrieval, pages 471–475, 2021.
  * [40] Haofei Kuang, Yi Zhu, Zhi Zhang, Xinyu Li, Joseph Tighe, Sören Schwertfeger, Cyrill Stachniss, and Mu Li.  Video contrastive learning with global context.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3195–3204, 2021.
  * [41] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 780–787, 2014.
  * [42] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho.  Motionsqueeze: Neural motion feature learning for video understanding.  In European Conference on Computer Vision, pages 345–362. Springer, 2020.
  * [43] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 36–52, Cham, 2016. Springer International Publishing.
  * [44] Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager.  Segmental spatiotemporal CNNs for fine-grained action segmentation.  In European conference on computer vision, pages 36–52. Springer, 2016.
  * [45] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Unsupervised representation learning by sorting sequences.  In Proceedings of the IEEE international conference on computer vision, pages 667–676, 2017.
  * [46] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham.  SFNet: Learning object-aware semantic correspondence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2278–2287, 2019.
  * [47] Myunggi Lee, Seungeui Lee, Sungjoon Son, Gyutae Park, and Nojun Kwak.  Motion feature network: Fixed motion filter for action recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 387–403, 2018.
  * [48] Congcong Li, Xinyao Wang, Dexiang Hong, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen.  Structured context transformer for generic event boundary detection.  arXiv preprint arXiv:2206.02985, 2022.
  * [49] Congcong Li, Xinyao Wang, Longyin Wen, Dexiang Hong, Tiejian Luo, and Libo Zhang.  End-to-end compressed video representation learning for generic event boundary detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13967–13976, June 2022.
  * [50] Rui Li, Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei.  Motion-focused contrastive learning of video representations.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2105–2114, 2021.
  * [51] Ji Lin, Chuang Gan, and Song Han.  TSM: Temporal shift module for efficient video understanding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083–7093, 2019.
  * [52] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  BMN: Boundary-matching network for temporal action proposal generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3889–3898, 2019.
  * [53] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  BSN: Boundary sensitive network for temporal action proposal generation.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [54] Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao.  Future frame prediction for anomaly detection–a new baseline.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6536–6545, 2018.
  * [55] Xingyu Liu, Joon-Young Lee, and Hailin Jin.  Learning video representations from correspondence proposals.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4273–4281, 2019.
  * [56] Antoine Miech, Ivan Laptev, Josef Sivic, Heng Wang, Lorenzo Torresani, and Du Tran.  Leveraging the present to anticipate the future in videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
  * [57] Juhong Min, Jongmin Lee, Jean Ponce, and Minsu Cho.  Hyperpixel flow: Semantic correspondence with multi-layer neural features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3395–3404, 2019.
  * [58] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  * [59] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European conference on computer vision, pages 69–84. Springer, 2016.
  * [60] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [61] AJ Piergiovanni and Michael S Ryoo.  Representation flow for action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9945–9953, 2019.
  * [62] Hamed Pirsiavash and Deva Ramanan.  Parsing videos of actions with segmental grammars.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [63] Rui Qian, Yeqing Li, Liangzhe Yuan, Boqing Gong, Ting Liu, Matthew Brown, Serge Belongie, Ming-Hsuan Yang, Hartwig Adam, and Yin Cui.  Exploring temporal granularity in self-supervised video representation learning.  arXiv preprint arXiv:2112.04480, 2021.
  * [64] Ayush K Rai, Tarun Krishna, Julia Dietlmeier, Kevin McGuinness, Alan F Smeaton, and Noel E O’Connor.  Discerning generic event boundaries in long-form wild videos.  arXiv preprint arXiv:2106.10090, 2021.
  * [65] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic.  Convolutional neural network architecture for geometric matching.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6148–6157, 2017.
  * [66] Madeline Chantry Schiappa, Yogesh Singh Rawat, and Mubarak Shah.  Self-supervised learning for videos: A survey.  ArXiv, abs/2207.00419, 2022.
  * [67] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin.  Intra- and inter-action understanding via temporal action parsing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [68] Mike Zheng Shou, Stan W Lei, Weiyao Wang, Deepti Ghadiyaram, and Matt Feiszli.  Generic event boundary detection: A benchmark for event segmentation.  arXiv preprint arXiv:2101.10511, 2021.
  * [69] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  Advances in neural information processing systems, 27, 2014.
  * [70] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934–8943, 2018.
  * [71] Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, and Wei Zhang.  Optical flow guided feature: A fast and robust motion representation for video action recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1390–1399, 2018.
  * [72] Jiaqi Tang, Zhaoyang Liu, Chen Qian, Wayne Wu, and Limin Wang.  Progressive attention on multi-level dense difference maps for generic event boundary detection.  arXiv preprint arXiv:2112.04771, 2021.
  * [73] Li Tao, Xueting Wang, and Toshihiko Yamasaki.  Self-supervised video representation learning using inter-intra contrastive framework.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 2193–2201, 2020.
  * [74] Barbara Tversky and Jeffrey M Zacks.  Event perception.  Oxford handbook of cognitive psychology, 1(2):3, 2013.
  * [75] Jiangliu Wang, Jianbo Jiao, and Yun-Hui Liu.  Self-supervised video representation learning by pace prediction.  In European conference on computer vision, pages 504–521. Springer, 2020.
  * [76] Yuxuan Wang, Difei Gao, Licheng Yu, Stan Weixian Lei, Matt Feiszli, and Mike Zheng Shou.  Generic event boundary captioning: A benchmark for status changes understanding.  arXiv preprint arXiv:2204.00486, 2022.
  * [77] Huifen Xia and Yongzhao Zhan.  A survey on temporal action localization.  IEEE Access, 8:70477–70487, 2020.
  * [78] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang.  Self-supervised spatiotemporal learning via video clip order prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10334–10343, 2019.
  * [79] Ceyuan Yang, Yinghao Xu, Bo Dai, and Bolei Zhou.  Video representation learning with visual tempo consistency.  arXiv preprint arXiv:2006.15489, 2020.
  * [80] Xitong Yang, Xiaodong Yang, Sifei Liu, Deqing Sun, Larry Davis, and Jan Kautz.  Hierarchical contrastive motion learning for video action recognition.  arXiv preprint arXiv:2007.10321, 2020.
  * [81] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021.
  * [82] Yue Zhao, Yuanjun Xiong, and Dahua Lin.  Recognize actions by disentangling components of dynamics.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6566–6575, 2018.

##  7 Supplementary : Additional Details

###  7.1 Implementation Details

Stage 2: Finetuning. Input to the encoder is based on the temporal window
$W=5$ and stride $m=3$. $(W,m)$ can be thought of as hyper-parameter, setting
a larger value of each might introduce noise information when two different
boundaries lie close to each other, a smaller value might be unable to capture
the necessary context information for a boundary. Among the 5 annotations
available for Kinetics-GEBD for every video, the ones with highest annotator
F1 consistency score is used for fine-tuning.

###  7.2 Architectural Design Choice

Temporal Shift Module (TSM) [51] is inserted in every residual block of
ResNet50 encoder. MotionSqueeze module is added after the conv3_x layer of the
ResNet50 encoder.

Table 4: Modified ResNet50 Encoder

Layers | ResNet-50 | Modified ResNet-50 | Output size
---|---|---|---
conv1 | $7\times 7,64\text{, stride }2$ | $112\times 112$
| $3\times 3,\text{ max-pool},\text{stride }2$ |
conv2_x | $\begin{bmatrix}1\times 1,64\\\ 3\times 3,64\\\ 1\times 1,256\end{bmatrix}\times 3$ | $\begin{bmatrix}\text{TSM}\\\ 1\times 1,64\\\ 3\times 3,64\\\ 1\times 1,256\end{bmatrix}\times 3$ | $56\times 56$
conv3_x | $\begin{bmatrix}1\times 1,128\\\ 3\times 3,128\\\ 1\times 1,128\end{bmatrix}\times 4$ | $\begin{bmatrix}\text{TSM}\\\ 1\times 1,128\\\ 3\times 3,128\\\ 1\times 1,128\end{bmatrix}\times 4$ | $28\times 28$
MS Module | ✗ | ✓ | $28\times 28$
conv4_x | $\begin{bmatrix}1\times 1,256\\\ 3\times 3,256\\\ 1\times 1,1024\end{bmatrix}\times 6$ | $\begin{bmatrix}\text{TSM}\\\ 1\times 1,256\\\ 3\times 3,256\\\ 1\times 1,1024\end{bmatrix}\times 6$ | $14\times 14$
conv5_x | $\begin{bmatrix}1\times 1,512\\\ 3\times 3,512\\\ 1\times 1,2048\end{bmatrix}\times 3$ | $\begin{bmatrix}\text{TSM}\\\ 1\times 1,512\\\ 3\times 3,512\\\ 1\times 1,2048\end{bmatrix}\times 3$ | $7\times 7$

It should be noted that our encoder definition is consistent with the
architecture design introduced in ResNet [28] and is different from the
encoder in the work of MotionSqueeze in [42] as shown in Table 4.

###  7.3 Evaluation Protocol.

We conduct evaluation on two datasets Kinetics-GEBD [68] and TAPOS [67]. For
evaluation, we follow the standard evaluation protocol explained in [68],
which uses the F1 score as the measurement metric. Rel. Dis (Relative
Distance) is used to decide whether a detected event boundary is correct (if
detection probability $\geq 0.5$) or otherwise incorrect. More formally, Rel.
Dis is defined as the error between detected and ground-truth timestamps,
divided by the length of the whole video. F1 score calculated at Rel. Dis
threshold 0.05 was used as the evaluation metric for the GEBD challenge1. We
compare our detection results with all annotations (5 annotations per video
for Kinetics-GEBD and 1 annotation for TAPOS) in the same video and select the
annotation with the highest F1 score.