# Revealing Topology in Metals using Experimental Protocols Inspired by
$K$-Theory

Wenting Cheng<EMAIL_ADDRESS>Department of Physics, New Jersey Institute of
Technology, Newark, NJ, USA  Alexander Cerjan<EMAIL_ADDRESS>Center for
Integrated Nanotechnologies, Sandia National Laboratories, Albuquerque, New
Mexico 87185, USA  Ssu-Ying Chen<EMAIL_ADDRESS>Department of Physics, New
Jersey Institute of Technology, Newark, NJ, USA  Emil Prodan<EMAIL_ADDRESS>Department of Physics, Yeshiva University, New York, NY, USA  Terry A. Loring
<EMAIL_ADDRESS>Department of Mathematics and Statistics, University of
New Mexico, Albuquerque, New Mexico 87131, USA  Camelia Prodan
<EMAIL_ADDRESS>Department of Physics, New Jersey Institute of Technology,
Newark, NJ, USA

###### Abstract

Topological metals are special conducting materials with gapless band
structures and nontrivial edge-localized resonances, whose discovery has
proved elusive because the traditional topological classification methods do
not apply in this context. Inspired by recent theoretical developments that
leveraged techniques from the field of $C^{*}$-algebras to identify
topological metals [1], here, we directly observe topological phenomena in
gapless acoustic crystals and provide a general experimental technique to
demonstrate their topology. Specifically, we not only observe robust boundary-
localized states in a topological acoustic metal, but also re-interpret a
composite operator, mathematically derived from the K-theory of the problem,
as a new Hamiltonian, whose physical implementation allows us to directly
observe a topological spectral flow and measure the topological invariants.
Our observations and experimental protocols may offer insights for discovering
topological behavior across a wide array of artificial and natural materials
that lack bulk band gaps.

Over the past two decades, immense progress has been made in predicting and
observing topological phases of matter and their associated boundary-localized
states in insulators [2, 3, 4, 5, 6] and semi-metals [7, 8, 9, 10, 11, 12, 13,
14, 15, 16, 17, 18, 19]. These developments have been predicated upon the
spectral isolation of the topological phenomena in these classes of systems;
although semi-metals such as Dirac [7, 8, 9], Weyl [10, 11, 12, 13, 14, 15,
16] and nodal line/ring semimetals [17, 18, 19] generally do not possess
complete gaps in their band structures, the topological phenomena that
manifest in these systems nevertheless appear within these systems’ incomplete
band gaps, allowing these phenomena to be uniquely identified at some energy
and quasimomentum. In contrast, the lack of band gaps (or, more generally,
mobility gaps) in metals and other gapless systems has largely prohibited the
study of topological metals (beyond studies of the topology of a metal’s Fermi
surface [20, 21, 22]). From a theoretical perspective, the absence of spectral
or dynamical gaps essentially precludes the use of topological band theory to
identify the invariants of these systems. Moreover, any boundary-localized
phenomena in gapless systems will generally hybridize with the degenerate bulk
states to create boundary-localized resonances, which complicates their
experimental observation. Thus, despite the enormous advances that have been
made in topological materials, the study of topological metals and gapless
materials has remained almost entirely unexplored.

Figure 1: Topological metals and their acoustic implementation. a, Photograph
of the fully assembled acoustic metalized SSH consisting of block resonators
and coupling bridges. b, Photograph of the inner structure, with all coupling
bridges visible. The 3D-printed resonators are open at the bottom and are
coupled through the 3D printed bridges, and grooves in the black base when
assembled. The acoustic resonators are embedded in the transparent acrylic
plate (top) that sits above the black base (bottom). The white dotted line
indicate how the acoustic resonators dimer is mounted on the black base, so
that the chiral symmetry is well enforced in the system. c, Top view of the
wave propagation domain near the domain boundary, with the relevant
parameters: $t_{\textrm{M}}=7mm$. $t_{\textrm{out}}=5mm$ and
$t_{\textrm{in}}=15mm$. d, Left: the acoustic eigen-pressure field for a
single acoustic block resonator at first elementary resonance mode, together
with the geometry parameters. Middle: a 3D-printed acoustic dimer consisting
of one resonator belonging to the SSH layer and one resonator belonging to the
metal layer, with the pair connected by an acoustic 3D printed bridge. Right:
Another view of the acoustic dimer with the coupling bridge of width
$\textit{t}_{c}=3mm$ clearly visible.

Recently, a general theoretical method for evaluating the topology of metallic
and gapless systems was put forward, opening new opportunities for discovering
topology in this class of systems that could not be previously explored [1].
This theoretical framework is rooted in the system’s spectral localizer, which
makes use of the system’s real-space description and yields local invariants
that are protected by local gaps [23, 24, 25]. The key concept that links
traditional band theoretic approaches to this local understanding of a
system’s topology stems from a dual description of atomic limits, i.e., the
limit where a complete basis of spatially localized Wannier functions exists.
In band theoretic approaches, a filled group of bands is topologically trivial
if they can be continued to an atomic limit without closing the band gap or
breaking a symmetry; any obstruction to this continuation manifests as a non-
trivial invariant [26, 27, 28]. From a real-space operator perspective, an
atomic limit’s complete set of Wannier functions each have both a well-defined
position and energy (in crystals they can be expressed as a flat band [29]).
Thus, a $d$-dimensional atomic limit’s Hamiltonian, $H$, commutes with its
position operators, $X_{j}$, $[H,X_{j}]=0$, for all $j\in 1,\ldots,d$. Using
this mathematical observation, the spectral localizer ascertains a system’s
topology by determining whether there is an obstruction in continuing its $H$
and $X_{j}$ to be commuting (given similar restrictions as before), an
analysis that can be performed using recent developments from the study of
$C^{*}$-algebras [23, 30]; any obstruction to continuing the system to possess
commuting matrices yields a non-trivial invariant. By recasting the
determination of a system’s topology to real space, and not relying upon a
bulk band gap to be the measure of topological protection, the spectral
localizer is equally applicable to insulators and metals—meaningful local gaps
that protect non-trivial topology can be found in both cases [1]. However,
despite the prediction that the local topological invariants of gapless
systems can be robust against system perturbations, robust topological metals
have not been previously observed in any platform.

Here, we theoretically develop and experimentally realize a robust topological
metallic state in an acoustic crystal. Our design is based on coupling a
topologically gapped acoustic crystal to a gapless one, yielding a system that
full-wave simulations show it possesses a gapless resonant spectrum.
Nevertheless, when domain boundaries are introduced, both simulations and
experimental observations reveal that this two-layer system possesses
boundary- and domain-localized states, and the topological origins of these
states can be proven using the spectral localizer. Moreover, to confirm the
topological origin of these localized states, we develop an experimental
protocol that treats the system’s spectral localizer as the Hamiltonian of a
perturbed system, enabling the direct observation of the underlying system’s
spatially resolved $K$-theory, i.e., its local topology. Taken together, these
measurements demonstrate that we have realized a topological metal. Given the
generality of our experimental methodologies, these findings open
opportunities for the discovery of gapless topological phenomena across a
broad range of natural and artificial materials.

Phononic crystals have proven to be an excellent platform for proof-of-
principle demonstrations of topological physics given their relative ease of
fabrication and reconfigurability [13, 14, 31, 15, 16, 32, 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43, 44]. However, enforcing exact symmetries, such as the
chiral symmetry invoked by our present application, requires fine engineering
because the couplings of acoustic cavities generate both hopping and onsite
tight-binding terms. These terms need to be finely tuned in order to ensure
the chiral symmetry (see Supplemental Material). Our acoustic crystals consist
of lattices of acoustic cavities coupled together via channels grooved into
the system’s base and also via direct bridges (see Fig. 1). The geometry of
the cavities is designed so that their fundamental axial pressure modes are
well-separated in frequency from the rest of the resonant spectrum. The
coupling strength between adjacent resonators is controlled through the width
of the channels. The base of the systems are laser-cut while the dimmers of
bridged resonating cavities are fabricated by 3D printing UV-curing resin. For
example, a bipartite single-layer metamaterial consisting of identical
resonators and alternating coupling channels supplies an accurate acoustic
realization of the standard Su-Schrieffer-Heeger (SSH) lattice [45] (see
Suplemental Material). By designing a metamaterial with a domain boundary
between two topologically distinct SSH insulating phases, and trivial outer
edges, both simulation and experimental observations confirm that the system
possesses a single topological resonant mode localized at the domain boundary
and whose frequency is in the middle of the system’s bulk band gap (see Fig.
2a,b).

Figure 2: Experimental demonstration of acoustic topological metal. a,
Numerically calculated eigenfrequencies of the acoustic SSH model, the
structure is shown in Fig. 1, but the metal layer and the connection between
the SSH and the metal layer are removed. The SSH model consists of 41
resonators with a domain boundary in the middle and topologically trivial at
both ends. b, Measured local density of states (LDOS), assembled from
microphone readings on 41 resonators of the acoustic SSH system. A bulk band
gap can be identified and the resonant mode in the bulk band gap is the domain
boundary mode. c, Numerically computed eigenfrequencies of the metallized SSH
system consist of 41 dimers which is consistent with our experiments in Fig.
1. There is evidence of resonant states localized at the domain boundary and
edges. d, Measured local density of states, assembled from microphone readings
on the 41 dimers (82 resonators) of the metallized SSH system. LDOS is
resolved by the dimer index. The data from the SSH resonator and the data from
the metal resonator with the same dimer index are added up. Note that the
dimer index are odd numbers. e, Measured acoustic pressure field distribution
at 4560 Hz for the domain boundary- and edge- localized modes. f, Simulated
acoustic eigenpressure fields of the four domain boundary- and edge- localized
states.

To realize a topological metal, we couple the single-layer SSH lattice to a
second single-layer lattice consisting of the same number of resonators (with
identical resonator geometry), but in which all of the intra-layer couplings,
$t_{\textrm{M}}$, are equal. The couplings between the two layers are also all
uniform, $t_{\textrm{c}}$ (see Figs. 1a-c). Note that the domain-boundary in
the SSH lattice layer is still present, but that this domain boundary does not
appear in the uniform layer. Altogether, this two-layer system possesses four
resonators per unit cell, has real coupling coefficients, and displays the
chiral symmetry (i.e., it is a 1D lattice in class BDI [46, 29, 47]); thus,
this system has the necessary symmetries to protect topological states at the
center of its fundamental axial mode’s spectrum. Full-wave simulations of this
two-layer lattice show that it possesses a gapless spectrum (see Fig. 2c), and
as such, topological band theories are unable to identify any robust topology
in this system, as it does not exhibit a bulk band gap. Nevertheless,
experimental observations (Figs. 2d,e) and numerical simulations (Fig. 2f)
both show that this metamaterial possesses edge- and domain boundary-localized
resonant states.

To prove that the observed localized states are of topological origin, we use
the spectral localizer. In general, a system’s spectral localizer combines its
Hamiltonian and position operators using a non-trivial Clifford
representation, Eq. (S1). However, as our acoustic metamaterial is a 1D chiral
symmetric system in which all of the couplings are real (i.e., its effective
Hamiltonian is real-symmetric), its spectral localizer can be written in a
reduced form as (see Supplemental Material)

$\tilde{L}_{(x,E)}(X,H)=\kappa(X-xI)\Pi+H-iE\Pi.$ (1)

Here, the spectral localizer can be evaluated at any choice of parameters
$x,E\in\mathbb{R}$ (e.g. inside or outside of the system’s spatial and
spectral extent), $\kappa$ is a tuning parameter and ensures that the terms
have compatible units, $\Pi$ is the system’s chiral operator, $H\Pi=-\Pi H$,
and $I$ is the identity matrix. Although the spectral localizer is basis
independent, if $H$ is written in a tight-binding basis, $X$ is simply a
diagonal matrix whose entries correspond to the coordinates of each lattice
site.

The spectral localizer (of appropriate dimension) can be used to both
construct the relevant local topological invariant for a system in any
symmetry class, as well as define the associated local gap. As the two-layer
“metallized SSH model” considered here is in symmetry class BDI [46], its
local invariant is given by [23],

$\nu(x)=\tfrac{1}{2}\textrm{sig}\left(\tilde{L}_{(x,0)}(X,H)\right)\in\mathbb{Z},$
(2)

where sig is the signature of a matrix, i.e., its number of positive
eigenvalues minus its number of negative ones. Note, $\nu(x)$ is only defined
for $E=0$, which reflects the fact that chiral symmetry can only protect
states at the middle of the system’s spectrum. Similarly, the local gap,
$\mu(x,E)$, is given by the smallest singular value of $\tilde{L}_{(x,E)}$,

$\mu(x,E)=\sigma_{\textrm{min}}(\tilde{L}_{(x,E)}(X,H)).$ (3)

Together, these two quantities yield a complete picture of a system’s
topology. Rigorously, $\nu(x)$ is ascertaining whether the matrices $H$ and
$X-xI$ can be continued to commuting while preserving both operators real-
symmetric form and chiral symmetry, and without closing the associated local
gap, i.e., $\mu(x,0)>0$ during the entire continuation process; if $\nu(x)=0$,
such a continuation is possible. Moreover, here we emphasize the local nature
of this picture, different choices of $x$ can yield different invariants—for
$x$ outside of the system’s spatial extent, one expects to see a system with
trivial topology, while $x$ in the bulk of the system may reveal non-trivial
topology. At the domain boundary between these two regions, $x_{0}$, where
$\nu(x_{0})$ changes, the local gap must close $\mu(x_{0},0)=0$, which is a
direct manifestation of bulk-boundary correspondence [30].

To realize the spectral localizer in both full-wave simulations and
experimentally, we develop an experimental protocol based on reinterpreting
our metamaterial’s reduced spectral localizer, Eq. (1), as a Hamiltonian for a
related system. To do so, we first note that at the center of the spectrum
(i.e., $E=0$), $\tilde{L}_{(x,0)}(X,H)$ is Hermitian; thus, we define a new
metamaterial’s Hamiltonian to be

$\tilde{H}_{x}=\tilde{L}_{(x,0)}(X,H)=\kappa(X-xI)\Pi+H.$ (4)

In this new metamaterial, the re-centered chiral position operator,
$\kappa(X-xI)\Pi$, is a modification of the central frequencies of each
resonator (i.e., an onsite perturbation). Thus, by simulating and observing
the spectrum of $\tilde{H}_{x}$, we are directly measuring the spectrum of
$\tilde{L}_{(x,0)}$, which, through Eq. (2), determines the topology of the
underlying system described by $H$. In practice, this reinterpretation
presents a challenge, as $\|X-xI\|$ can become arbitrarily large as the
lattice’s size increases, but it is not possible to alter a resonator’s
geometry to yield arbitrarily large or small resonance frequencies.
Nevertheless, we can circumvent this challenge by using the substitution

$\kappa(X-xI)\Pi\rightarrow\kappa\left[\tanh\left(\frac{X-xI}{\alpha}\right)\right]\Pi,$
(5)

such that

$\tilde{H}_{x}=\kappa\left[\tanh\left(\frac{X-xI}{\alpha}\right)\right]\Pi+H.$
(6)

As this bounded operator is linear in the vicinity of $x=0$ (the range of
approximate linearity is set by $\alpha$), one can prove that it preserves the
necessary information for determining the system’s topology using Eq. (2) (see
Supplementary Materials). Moreover, this choice of alteration to the system’s
resonators be experimentally realized for lattices of any size (see Fig. 3).

Figure 3: Experimental protocol for observing the spectral localizer. a,
Photograph of the fully assembled acoustic metallized SSH built to use the
spectral localizer as its Hamiltonian. The resonators that comprise this
system can be re-assembled to realize different choices of central localizer
position, $x$, in Eq. (6). b, Top view of the wave propagation domain of the
localizer system near the domain boundary, the indices of the dimers are
marked with odd numbers and the chiral sign are marked on each resonator with
red (+) and blue (-) color codes. c, The configurations of the system when the
localizer is centered at $x=-23$, $x=-1$ and $x=17$ respectively (indicated by
red arrows). Figure 4: Observation of topology in acoustic metallic systems
with the spectral localizer a, COMSOL simulated resonant spectrum of the
spectrally localized metamaterial as the position of the localizer’r center,
$x$, is varied, demonstrating the existence of the two bands in the dynamical
localization gap. The lower two panels show the change in the localizer index
and localizer gap as the localizer is moved from left to the right, and the
four orange points in the localizer gap graph indicate where the gap closes.
b, Experimental mapping of the local density of states as the localizer’s
center is moved (constructed from microphone readings on the dimer where the
localizer is located), confirming the existence of two bands in a. $\alpha=5$
and $\kappa=2$ were used in our simulations and experiments. c, Measured local
density of states (constructed from the microphone readings on the bulk
resonators at least 3 resonators away from the localizer). d, Experimentally
measured mode profiles at 4760 Hz and 4860 Hz when $x=-1$.

Using Eq. (6), we confirm the topological behavior of the metallized SSH model
by observing the properties of its spectrally localized counterpart. First,
full-wave simulations of the localized metamaterial, $\tilde{H}_{x}$,
demonstrate that there are four $x$ locations in the underlying system where
the local invariant changes and the local gap closes (Fig. 4a), two locations
at the outer edges of the system, and two next to the domain boundary. Thus,
as locations where $\mu(x,0)=0$ predict the presence of states, the localized
states seen in the underlying metallized SSH metamaterial (Fig. 2f) are a
direct manifestation of bulk-boundary correspondence and are necessarily of
topological origin. These topological states are protected against disorder
due to a secondary gap (see Supplemental Materials), despite the absence of a
bulk band gap in the system. Moreover, we experimentally realize the localized
metamaterial, and directly observe the changes in the system’s local
$K$-theory (Fig. 4b-d). In particular, the simulated resonance spectrum is
reproduced with high fidelity by experimental measurements, confirming the
topological properties of our metallized SSH metamaterial. Although different
choices of $x$ in $\tilde{H}_{x}$ yield distinct physical systems (see Fig.
3c), our acoustic metamaterial is re-configurable, and thus we do not need to
fabricate a new system for each choice of $x$ shown in Fig. 4b. Altogether,
our experimental results, coupled with our full-wave simulations, prove that
the metallized SSH metamaterial is a gapless topological material.

In conclusion, we have demonstrated a topological metal in an acoustic
metamaterial and directly observed its boundary-localized states despite its
lack of a bulk band gap. To do so, we have used the spectral localizer, a
local theory of topological materials that is able to predict topological
phenomena, and a measure of topological protection, even in the absence of a
bulk band gap. Moreover, we have introduced an experimental protocol that uses
a system’s spectral localizer as its Hamiltonian, providing a direct probe of
the underlying system’s local $K$-theory. Here, it is worth emphasizing that
this protocol can be applied to any topological system. Although our specific
demonstration has leveraged the system’s symmetries to yield a real-symmetric
spectral localizer, the spectral localizer for any system is, by definition,
Hermitian, and as such, it can always be adapted to be an observable system.
Thus, the overall methodology that we have introduced may enable the
prediction and observation of topological metals across a broad array of
systems, including materials that exhibit higher-order topology and those
whose topology is determined by its crystalline symmetries. Finally, as the
spectral localizer takes an operator-based, rather than eigenstate-based,
approach to topology, it is potentially broadly applicable to interacting
systems, a class of systems that traditional band theories of topology have
had difficulty gaining traction with.

##  I Methods

###  I.1 Fabrication

Our fabrication process is modular and the acoustic crystals are assembled
from parts that are independently manufactured with different automated
process. This approach enables a high throughput of acoustic crystals, which
can be dis-assembled and stored after use.

One leg of the process is the manufacturing of the supporting bases, which
consist of two layers of 3-mm thick black acrylic plates (Fig. 1bottom) and
one top layer of 1.5-mm thick transparent acrylic plates (Fig. 1b top), of
which the top transparentr layer and middle black layer have through holes,
laser-cut at specific geometries with the Boss Laser-1630 Laser Engraver. The
middle layer with 3mm deep through channels provides coupling channels between
the dimers. The top transparent layer with through square holes holds the
dimers in place.

The resonators were manufactured using an Anycubic Photon 3D printer, which
uses UV resin and has 47 um XY-resolution and 10 um Z-resolution. The
thickness of their walls is 2 mm, to ensure a good quality factor and to
justify rigid boundaries in our numerical simulations. The inner dimensions of
the resonators are supplied in Fig. 1d. For the metallized-SSH system, the
resonators are 3D printed as dimers with identical narrow channels connecting
the resonators. To implement the spectral localizer, the resonators were
printed with different heights according to the algorithm Eq. (6) and were
made ready for the assembling.

The resonators were mounted and coupled through the channels grooved in the
acrylic plates of the base. Let us specify that the resonators are
interchangeable so that they can be move around and acoustic crystals with
different probe positions can be generated, as described in the main text.

###  I.2 Experimental protocols

The protocol for the acoustic measurements reported in Fig. 2 and Fig. 4 was
as follows: Sinusoidal signals of duration 1 s and amplitude of 0.5 V were
produced with a Rigol DG 1022 function generator and applied on a speaker
placed in a porthole opened in a resonator. A dbx RTA-M Reference Microphone
with a Phantom Power was inserted in a porthole opened in the same resonator
where the speaker was inserted and acquired the acoustic signals. The signals
were read by a custom LabVIEW code via National Instruments USB-6122 data
acquisition box and the data was stored on a computer for graphic renderings.

The density of states reported in Fig. 2 and Fig. 4 was obtain by integrating
the local density of states acquired from resonators whose index are the same
as the position of the probe. Same instrumentation was used. The measurements
were repeated with moving the position of the probe. For each measurement, the
frequency was scanned from 4200 Hz to 5200 Hz in 10 Hz steps.

###  I.3 Simulation

The simulations reported in Fig. 2 and Fig. 4 were performed with the COMSOL
Multiphysics pressure acoustic module. The wave propagation domain shown in
Fig. 1 was filled with air with a mass density 1.3 kg/m3 and the sound’s speed
was fixed at 343 m/s, appropriate for room temperature. Because of the huge
acoustic impedance mismatch compared with air, the 3D printing UV resin
material was considered as hard boundary.

## Acknowledgements

T.L. acknowledges support from the National Science Foundation, grant
DMS-2110398. A.C., T.L., and C.P. acknowledge support from the Center for
Integrated Nanotechnologies, an Office of Science User Facility operated for
the U.S. Department of Energy (DOE) Office of Science. A.C. and T.L.
acknowledge support from the Laboratory Directed Research and Development
program at Sandia National Laboratories. Sandia National Laboratories is a
multimission laboratory managed and operated by National Technology &
Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell
International, Inc., for the U.S. DOE’s National Nuclear Security
Administration under contract DE-NA-0003525. The views expressed in the
article do not necessarily represent the views of the U.S. DOE or the United
States Government. C. P. acknowledges additional support from the National
Science Foundation, grant CMMI-2131759. E.P. was supported by the U.S.
National Science Foundation through the grants DMR-1823800 and CMMI-2131760.

## Data and code availability

The data and code that support the findings of this study are available from
the corresponding authors on reasonable request.

## Supplemental Material

The supplemental materials contain detailed mathematical discussions of the
properties of the spectral localizer and its spectrum, as well as a validation
of the chiral symmetry of our acoustic metamaterial. It contains citations to
the references [48, 49, 50].

## References

  * Cerjan and Loring [2022] A. Cerjan and T. A. Loring, Local invariants identify topology in metals and gapless systems, Phys. Rev. B 106, 064109 (2022).
  * Hasan and Kane [2010] M. Z. Hasan and C. L. Kane, Colloquium: Topological insulators, Rev. Mod. Phys. 82, 3045 (2010).
  * Qi and Zhang [2011] X.-L. Qi and S.-C. Zhang, Topological insulators and superconductors, Rev. Mod. Phys. 83, 1057 (2011), publisher: American Physical Society.
  * Bansil  _et al._ [2016] A. Bansil, H. Lin, and T. Das, Colloquium: Topological band theory, Rev. Mod. Phys. 88, 021004 (2016), publisher: American Physical Society.
  * Chiu  _et al._ [2016] C.-K. Chiu, J. C. Y. Teo, A. P. Schnyder, and S. Ryu, Classification of topological quantum matter with symmetries, Rev. Mod. Phys. 88, 035005 (2016), publisher: American Physical Society.
  * Ozawa  _et al._ [2019] T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, Topological photonics, Rev. Mod. Phys. 91, 015006 (2019).
  * Wang  _et al._ [2012] Z. Wang, Y. Sun, X.-Q. Chen, C. Franchini, G. Xu, H. Weng, X. Dai, and Z. Fang, Dirac semimetal and topological phase transitions in $A_{3}$Bi ($A$=Na, K, Rb), Phys. Rev. B 85, 195320 (2012).
  * Liu  _et al._ [2014] Z. K. Liu, B. Zhou, Y. Zhang, Z. J. Wang, H. M. Weng, D. Prabhakaran, S.-K. Mo, Z. X. Shen, Z. Fang, X. Dai, Z. Hussain, and Y. L. Chen, Discovery of a Three-Dimensional Topological Dirac Semimetal, Na3Bi, Science 343, 864 (2014), publisher: American Association for the Advancement of Science.
  * Wang  _et al._ [2013] Z. Wang, H. Weng, Q. Wu, X. Dai, and Z. Fang, Three-dimensional Dirac semimetal and quantum transport in Cd3As2, Phys. Rev. B 88, 125427 (2013).
  * Weng  _et al._ [2015] H. Weng, C. Fang, Z. Fang, B. A. Bernevig, and X. Dai, Weyl Semimetal Phase in Noncentrosymmetric Transition-Metal Monophosphides, Phys. Rev. X 5, 011029 (2015), publisher: American Physical Society.
  * Lv  _et al._ [2015] B. Q. Lv, N. Xu, H. M. Weng, J. Z. Ma, P. Richard, X. C. Huang, L. X. Zhao, G. F. Chen, C. E. Matt, F. Bisti, V. N. Strocov, J. Mesot, Z. Fang, X. Dai, T. Qian, M. Shi, and H. Ding, Observation of Weyl nodes in TaAs, Nat. Phys. 11, 724 (2015).
  * Lu  _et al._ [2015] L. Lu, Z. Wang, D. Ye, L. Ran, L. Fu, J. D. Joannopoulos, and M. Soljačić, Experimental observation of Weyl points, Science 349, 622 (2015).
  * Xiao  _et al._ [2015] M. Xiao, W.-J. Chen, W.-Y. He, and C. T. Chan, Synthetic gauge flux and Weyl points in acoustic systems, Nat. Phys. 11, 920 (2015).
  * Li  _et al._ [2018] F. Li, X. Huang, J. Lu, J. Ma, and Z. Liu, Weyl points and Fermi arcs in a chiral phononic crystal, Nat. Phys. 14, 30 (2018).
  * Xie  _et al._ [2019] B. Xie, H. Liu, H. Cheng, Z. Liu, S. Chen, and J. Tian, Experimental Realization of Type-II Weyl Points and Fermi Arcs in Phononic Crystal, Phys. Rev. Lett. 122, 104302 (2019).
  * Yang  _et al._ [2019] Y. Yang, H.-x. Sun, J.-p. Xia, H. Xue, Z. Gao, Y. Ge, D. Jia, S.-q. Yuan, Y. Chong, and B. Zhang, Topological triply degenerate point with double Fermi arcs, Nat. Phys. 15, 645 (2019).
  * Burkov  _et al._ [2011] A. Burkov, M. Hook, and L. Balents, Topological nodal semimetals, Physical Review B 84, 235126 (2011).
  * Xu  _et al._ [2015] Q. Xu, Z. Song, S. Nie, H. Weng, Z. Fang, and X. Dai, Two-dimensional oxide topological insulator with iron-pnictide superconductor lifeas structure, Physical Review B 92, 205310 (2015).
  * Fu  _et al._ [2019] B.-B. Fu, C.-J. Yi, T.-T. Zhang, M. Caputo, J.-Z. Ma, X. Gao, B. Lv, L.-Y. Kong, Y.-B. Huang, P. Richard, _et al._ , Dirac nodal surfaces and nodal lines in zrsis, Science advances 5, eaau6459 (2019).
  * Alexandradinata  _et al._ [2018] A. Alexandradinata, C. Wang, W. Duan, and L. Glazman, Revealing the Topology of Fermi-Surface Wave Functions from Magnetic Quantum Oscillations, Phys. Rev. X 8, 011027 (2018).
  * Sun  _et al._ [2020] S. Sun, Z. Song, H. Weng, and X. Dai, Topological metals induced by the Zeeman effect, Phys. Rev. B 101, 125118 (2020).
  * Yuan and Fu [2021] N. F. Q. Yuan and L. Fu, Topological metals and finite-momentum superconductors, Proceedings of the National Academy of Sciences 118, e2019063118 (2021).
  * Loring [2015] T. A. Loring, $K$-theory and pseudospectra for topological insulators, Annals of Physics 356, 383 (2015).
  * Loring and Schulz-Baldes [2020] T. A. Loring and H. Schulz-Baldes, The spectral localizer for even index pairings, J. Noncommut. Geom. 14, 1 (2020).
  * Loring and Schulz-Baldes [2017] T. A. Loring and H. Schulz-Baldes, Finite volume calculation of $K$-theory invariants, New York J. Math. 23, 1111 (2017).
  * Bradlyn  _et al._ [2017] B. Bradlyn, L. Elcoro, J. Cano, M. G. Vergniory, Z. Wang, C. Felser, M. I. Aroyo, and B. A. Bernevig, Topological quantum chemistry, Nature 547, 298 (2017).
  * Po  _et al._ [2017] H. C. Po, A. Vishwanath, and H. Watanabe, Symmetry-based indicators of band topology in the 230 space groups, Nat. Commun. 8, 50 (2017).
  * Cano  _et al._ [2018] J. Cano, B. Bradlyn, Z. Wang, L. Elcoro, M. G. Vergniory, C. Felser, M. I. Aroyo, and B. A. Bernevig, Building blocks of topological quantum chemistry: Elementary band representations, Phys. Rev. B 97, 035139 (2018).
  * Kitaev [2009] A. Kitaev, Periodic table for topological insulators and superconductors, AIP Conference Proceedings 1134, 22 (2009).
  * Hastings and Loring [2011] M. B. Hastings and T. A. Loring, Topological insulators and $C^{*}$-algebras: Theory and numerical practice, Ann. Phys. July 2011 Special Issue, 326, 1699 (2011).
  * Serra-Garcia  _et al._ [2018] M. Serra-Garcia, V. Peri, R. Süsstrunk, O. R. Bilal, T. Larsen, L. G. Villanueva, and S. D. Huber, Observation of a phononic quadrupole topological insulator, Nature 555, 342 (2018).
  * Ni  _et al._ [2019a] X. Ni, M. Weiner, A. Alù, and A. B. Khanikaev, Observation of higher-order topological acoustic states protected by generalized chiral symmetry, Nat. Mater. 18, 113 (2019a).
  * Ni  _et al._ [2019b] X. Ni, K. Chen, M. Weiner, D. J. Apigo, C. Prodan, A. Alù, E. Prodan, and A. B. Khanikaev, Observation of Hofstadter butterfly and topological edge states in reconfigurable quasi-periodic acoustic crystals, Commun. Phys. 2, 1 (2019b).
  * Wen  _et al._ [2019] X. Wen, C. Qiu, Y. Qi, L. Ye, M. Ke, F. Zhang, and Z. Liu, Acoustic Landau quantization and quantum-Hall-like edge states, Nat. Phys. 15, 352 (2019).
  * Xue  _et al._ [2019a] H. Xue, Y. Yang, F. Gao, Y. Chong, and B. Zhang, Acoustic higher-order topological insulator on a kagome lattice, Nat. Mater. 18, 108 (2019a).
  * Xue  _et al._ [2019b] H. Xue, Y. Yang, G. Liu, F. Gao, Y. Chong, and B. Zhang, Realization of an Acoustic Third-Order Topological Insulator, Phys. Rev. Lett. 122, 244301 (2019b).
  * Apigo  _et al._ [2019] D. J. Apigo, W. Cheng, K. F. Dobiszewski, E. Prodan, and C. Prodan, Observation of Topological Edge Modes in a Quasiperiodic Acoustic Waveguide, Phys. Rev. Lett. 122, 095501 (2019).
  * Peri  _et al._ [2020] V. Peri, Z.-D. Song, M. Serra-Garcia, P. Engeler, R. Queiroz, X. Huang, W. Deng, Z. Liu, B. A. Bernevig, and S. D. Huber, Experimental characterization of fragile topology in an acoustic metamaterial, Science 367, 797 (2020).
  * Cheng  _et al._ [2020] W. Cheng, E. Prodan, and C. Prodan, Experimental Demonstration of Dynamic Topological Pumping across Incommensurate Bilayered Acoustic Metamaterials, Phys. Rev. Lett. 125, 224301 (2020).
  * Ni  _et al._ [2020] X. Ni, M. Li, M. Weiner, A. Alù, and A. B. Khanikaev, Demonstration of a quantized acoustic octupole topological insulator, Nat. Commun. 11, 2108 (2020).
  * Chen  _et al._ [2021] Z.-G. Chen, R.-Y. Zhang, C. T. Chan, and G. Ma, Classical non-Abelian braiding of acoustic modes, Nat. Phys. , 1 (2021).
  * Xue  _et al._ [2021] H. Xue, D. Jia, Y. Ge, Y.-j. Guan, Q. Wang, S.-q. Yuan, H.-x. Sun, Y. Chong, and B. Zhang, Observation of Dislocation-Induced Topological Modes in a Three-Dimensional Acoustic Topological Insulator, Phys. Rev. Lett. 127, 214301 (2021).
  * Jiang  _et al._ [2021] B. Jiang, A. Bouhon, Z.-K. Lin, X. Zhou, B. Hou, F. Li, R.-J. Slager, and J.-H. Jiang, Experimental observation of non-Abelian topological acoustic semimetals and their phase transitions, Nat. Phys. , 1 (2021).
  * Deng  _et al._ [2022] Y. Deng, W. A. Benalcazar, Z.-G. Chen, M. Oudich, G. Ma, and Y. Jing, Observation of Degenerate Zero-Energy Topological States at Disclinations in an Acoustic Lattice, Phys. Rev. Lett. 128, 174301 (2022).
  * Su  _et al._ [1979] W. Su, J. Schrieffer, and A. J. Heeger, Solitons in polyacetylene, Physical review letters 42, 1698 (1979).
  * Schnyder  _et al._ [2008] A. P. Schnyder, S. Ryu, A. Furusaki, and A. W. W. Ludwig, Classification of topological insulators and superconductors in three spatial dimensions, Phys. Rev. B 78, 195125 (2008).
  * Ryu  _et al._ [2010] S. Ryu, A. P. Schnyder, A. Furusaki, and A. W. W. Ludwig, Topological insulators and superconductors: tenfold way and dimensional hierarchy, New Journal of Physics 12, 065010 (2010).
  * Loring [2019] T. A. Loring, A Guide to the Bott Index and Localizer Index, arXiv:1907.11791 [cond-mat, physics:math-ph] (2019), arXiv: 1907.11791.
  * Cerjan  _et al._ [2022] A. Cerjan, T. A. Loring, and F. Vides, Quadratic pseudospectrum for identifying localized states, arXiv:2204.10450 [cond-mat, physics:math-ph, physics:physics, physics:quant-ph] (2022).
  * Schulz-Baldes and Stoiber [2021] H. Schulz-Baldes and T. Stoiber, Invariants of disordered semimetals via the spectral localizer, EPL 136, 27001 (2021).


ROI

    Region of Interest
FOV

    Field of View
MAE

    Mean Absolute Error

# Table tennis ball spin estimation with an event camera

Thomas Gossard Julian Krismer∗ Andreas Ziegler Jonas Tebbe Andreas Zell
Cognitive Systems Group, University of Tübingen, Germany
<EMAIL_ADDRESS>equal contribution
This research was funded by Sony AI.

###### Abstract

Spin plays a pivotal role in ball-based sports. Estimating spin becomes a key
skill due to its impact on the ball’s trajectory and bouncing behavior. Spin
cannot be observed directly, making it inherently challenging to estimate. In
table tennis, the combination of high velocity and spin renders traditional
low frame rate cameras inadequate for quickly and accurately observing the
ball’s logo to estimate the spin due to the motion blur. Event cameras do not
suffer as much from motion blur, thanks to their high temporal resolution.
Moreover, the sparse nature of the event stream solves communication bandwidth
limitations many frame cameras face. To the best of our knowledge, we present
the first method for table tennis spin estimation using an event camera. We
use ordinal time surfaces to track the ball and then isolate the events
generated by the logo on the ball. Optical flow is then estimated from the
extracted events to infer the ball’s spin. We achieved a spin magnitude mean
error of $10.7\pm 17.3$ rps and a spin axis mean error of $32.9\pm
38.2\degree$ in real time for a flying ball.

##  1 Introduction

Spin is crucial in ball sports, enhancing ball control and adding an element
of unpredictability for the opponent. Whether it is topspin in tennis,
backspin in table tennis, or spin variations in cricket, it adds complexity to
the game. In soccer and baseball, players use spin to curve shots, making it
challenging for opponents to predict the ball’s trajectory. This makes spin
esimation decisive for playing such sports. Thus, it is also a key component
for developing autonomous agents and sports analysis systems. However, in most
cases, the spin is too fast for low frame rate cameras, which results in
severe motion blur. Event-based cameras do not suffer as much from motion blur
as standard frame-based cameras. Indeed, event cameras capture logarithmic
changes in a scene’s brightness with high temporal resolution in the order of
$\mu$s. Unlike traditional frame-based cameras, events are only emitted when
there is a visual change, offering low latency and reduced bandwidth
requirements. This leads to much less information to be transmitted, given a
scene with a static background. As a consequence, we do not need to limit the
Region of Interest (ROI) of the camera to avoid bandwidth issues. This was
indeed a problem in [39], where the full resolution of the camera
($1920$x$1200$ pixels) could not be used while maintaining a high enough frame
rate ($380$ fps). Instead, a ROI of $1920$x$400$ pixels was chosen. The ball
is thus observed for a shorter time, which decreases the likelihood of
observing the ball’s logo for low spin values. The event camera can use its
full resolution because it does not need to transmit redundant information.
The high temporal resolution of event cameras has an additional advantage.
Estimating spin using a frame-based camera is constrained by the Nyquist-
Shannon sampling theorem, which stipulates that the sampling rate must be at
least twice the signal’s bandwidth. In other words, we need a camera with a
certain frame rate to be able to measure high spin: $\text{fps}>|\omega|/2$,
where $\omega$ is the ball spin in $rps$. In Fig. 1, we can see that we only
capture two frames where the logo is visible.

Figure 1:  Comparison between the event stream of an event-based camera and
captured frames from a frame-based camera. Black and blue dots are,
respectively, ON- and OFF-events. The dotted red box highlights the events
generated by the spinning logo.

This makes it extremely difficult to estimate spin accurately. On the other
hand, the event camera could continuously capture the movement of the logo.
Occasionally, the ball rotates in a manner that causes the logo to
consistently appear on the opposite side when captured by a frame-based
camera. This is no issue for event-based cameras, as events are generated
asynchronously. This overcomes the previously mentioned issue as long as the
logo moves in front of the camera. For all the previously mentioned reasons,
event cameras are better suited for table tennis ball spin estimation.

Contributions of this work are as follows:

  * •

Tracking the ball in real-time and extracting events generated by the logo.

  * •

A real-time and accurate method for table tennis ball spin estimation.

  * •

An evaluation using a ball spinner to show the potential of the approach and a
deployment with a ball thrower to show the approach in a real setup.

##  2 Related work

###  2.1 Spin estimation

There are multiple approaches for spin estimation in table tennis. Human
players mostly estimate the spin applied to the ball from the motion of the
opponent’s racket. This can be reproduced by using human pose estimation [31,
26], racket tracking [18], or an inertial measurement unit (IMU) installed in
the racket [6, 5]. However, the estimated spin is only an approximation, as
there is no information on the racket’s bounce dynamics, which are greatly
influenced by its rubber type. For a more quantitative spin estimate, there
are two main approaches: observing the trajectory or directly observing the
ball. Trajectory-based spin estimation [42, 34, 39, 11] makes use of the
Magnus effect, which bends the ball’s trajectory depending on the magnitude
and type of spin applied. Nevertheless, these approaches rely heavily on
precise estimations of ball positions. The Magnus effect correlates directly
with the level of spin imparted on the ball, complicating the differentiation
between the spin-induced curvature of the trajectory and measurement
inaccuracies of the ball positions. While [2] showed that the ball’s
trajectory is insufficient for accurate spin estimation, the estimation of the
spin can be further improved by using the bounce behavior of the ball [42].

However, for the most accurate spin estimation, ball observation methods are
preferred. Using the blur generated by the logo for spin estimation has been
studied in [7] but it does not seem to scale well to higher spin values.
Within the ball observation methods, we distinguish two sub-categories: logo-
based [22, 39, 43] and pattern-based [41, 35, 16, 23, 38, 37]. The patterns
are always visible in contrast to the logo which is barely visible or hidden
from a single camera’s point of view, making previous logo-based approaches
unreliable. For official matches, custom balls are not allowed. This makes
logo-based methods necessary for competitive games. Our proposed approach
leverages event cameras for ball spin estimation.

All of the previously described methods are not exclusive. They can be
combined for better spin estimation. A more recent work [36] combined a logo-
based method with a trajectory-based approach. The ambiguity about the spin
magnitude stemming from the fact that the Nyquist-Shannon sampling theorem
might not be fulfilled was resolved by estimating the ball trajectory for
various spin magnitudes and choosing the spin magnitude for which the current
ball trajectory fits best to the estimated one.

###  2.2 Event cameras

Event cameras offer many advantages, such as a high temporal resolution and a
high dynamic range. However, they also change the processing paradigm. Most
computer vision algorithms developed in the last decades are either ill-
adapted to process events or do not make the most out of said advantages.
Frames can be reconstructed from events to apply state-of-the-art computer
vision algorithms, but this introduces extra latency and does not fully
utilize the event cameras’ speed.

Tracking fast-moving objects is a perfect use case for event cameras, thanks
to their high temporal resolution and less severe motion blur. A lot of
research has focused on fusing the data from frame-based and event-based
cameras [44, 8, 9, 10] to compensate for each other’s weaknesses: frames for
static or slow moving objects and events for fast-moving objects.

Because an event camera generates only ON/OFF events, new algorithms were
developed to distinguish different moving objects in a scene. [15, 14] tackle
this issue by segmenting events generated by the background and events
generated by independently moving objects with a motion-compensated mean time
stamp image. Events are then clustered using DBSCAN. Such methods were used
for either obstacle avoidance for drones [14] or for capturing a ball with a
quadruped robot [15]. Next to the rather specific motion-compensated mean time
stamp image, other common event representations are accumulated event frames
and time-surfaces [17]. However, the same object will not be viewed the same
way depending on its velocity. This was solved with an ordinal time surface
such as TOS [21] and EROS [19]. These ordinal time surfaces capture the edge
of objects without any trail of events, independent of the object’s velocity.
A similar representation could be achieved with accumulated event frames, but
only if the accumulation time was tuned to fit the object’s velocity.

Feature tracking can also benefit from using event cameras, as shown in [3,
12]. However, because of the logo’s small size and rapidly changing shape due
to the curve of the spinning ball, such methods are difficult to apply.

Optical flow is an alternative for estimating the logo’s speed. There are
multiple ways to infer the optical flow. There are Lucas-Kanade [27], plane-
fitting [4], and time surface matching [28], just to name a few. However, the
current state-of-the-art is learning-based [20]. The most common architectures
are U-Net [30], FireNet [32] and RAFT [40]. The optical flow can be learned in
either a supervised [20], semi-supervised [24], or unsupervised [25] fashion.
Although more accurate, learning-based optical flow methods require more
computational power, which makes achieving real-time performance difficult.
For this reason, we favor a model-based approach.

##  3 Method

Our table tennis ball spin estimation pipeline is divided into three phases.
Initially, a ball tracker, outlined in Sec. 3.1, estimates the ball’s
position, velocity, and radius. Next, this information is used to extract
events generated by the logo, as described in Sec. 3.2. Finally, the ball’s
spin is estimated from the extracted events, detailed in Sec. 3.3.

###  3.1 Ball tracker

We set up a static event camera, observing a primarily static background (a
table tennis table). This configuration is beneficial since the flying ball
triggers most of the events.

We use the Exponential Reduced Ordinal Surface (EROS) [19] event
representation for processing the events. EROS works by having new events
decay the values of surrounding pixels by a factor, thus enabling objects to
have sharp edges, whatever their velocity. This representation enables
continuous and asynchronous updates from the event stream, maximizing the
potential of the event camera. The update of EROS depends on a parameter named
$k_{eros}$, which represents the update zone around the event. $k_{eros}$ was
hand-tuned for the best circle detection and set to $10$. A comparison between
the accumulated event frames and EROS can be seen in Fig. 2.

Figure 2:  Pairs of representations for the same events. (left) EROS [19]
event representations with $k_{eros}=10$. (right) Accumulated event frames
with an accumulation time of $t_{acc}=2\text{ms}$.

One issue with the EROS time surface is the accumulation of noise. Because the
noise is spatially and temporally sparse, it stays in the representation. We
thus “clean” the EROS time surface by removing isolated pixels. We do so with
a hit-or-miss transform to detect isolated non-zero pixels. Detected isolated
pixels are then set to zero. This hit-or-miss transform is performed on every
update of the EROS time surface. For more details, we refer to Sec. 9 in the
supplementary material.

We do not use convolutions like in [19] because we also need the radius of the
ball for extracting the events generated by the logo as explained in Sec. 3.3.
Instead, we apply a Hough circle transform to detect the ball and extract the
peak value when the accumulation exceeds a certain threshold.

We added a Kalman filter to improve the ball tracker’s performance. This
allows for smoother tracking as well as ball velocity estimation. We also
estimate the ball’s radius to reduce the search space of the Hough circle
transform for faster processing and for converting distances in pixels to
meters. The state of the Kalman filter is defined with the state
$\bm{\mathrm{x}}=[x_{b},y_{b},\dot{x}_{b},\dot{y}_{b},r]$ where $x_{b}$ and
$y_{b}$ are the coordinates of the center of the ball, $\dot{x}_{b}$ and
$\dot{y}_{b}$ are the ball’s velocity in pixels/s and $r$ is the ball’s radius
in pixels. Estimating the ball velocity and radius is relevant for extracting
the events generated by the logo as explained in Sec. 3.2. The state
transition of the Kalman filter is defined as

$F=\begin{bmatrix}1&0&dt&0&0\\\ 0&1&0&dt&0\\\ 0&0&1&0&0\\\ 0&0&0&1&0\\\
0&0&0&0&1\end{bmatrix},$ (1)

where $dt$ is the time difference between two Kalman filter updates. The
observation vector of the Kalman filter is
$\bm{z}=[x_{b,\text{hough}},y_{b,\text{hough}},r_{\text{hough}}]$ where
$x_{b,\text{hough}}$ and $y_{b,\text{hough}}$ are the coordinates of the ball
center returned by the Hough circle transform and $r_{\text{hough}}$ is the
corresponding radius. The observation model of the Kalman filter is defined as

$H=\begin{bmatrix}1&0&0&0&0\\\ 0&1&0&0&0\\\ 0&0&0&0&1\end{bmatrix}.$ (2)

Though the ball’s velocity is not directly observed, it can implicitly be
estimated from the sequential measurements of ball positions. The initial
variance of the velocities must be set high enough to converge fast enough to
the correct velocity. In Fig. 3, we show an example of the Kalman filter
tracking the ball.

Figure 3:  Tracked states of the Kalman filter: Positon,
$\bm{p}_{b}=[x_{b},y_{b}]$, velocity, $\bm{v}_{b}=[\dot{x}_{b},\dot{y}_{b}]$
and ball radius $r$. The Kalman filter runs with $200$ Hz. The dots indicate
measurements from the hough circle transform. The colored band around the
estimated values represents the standard deviation.

We can observe that the radius of the ball is not constant. This is because
the ball can get closer to the camera after being shot.

###  3.2 Extracting logo events

The ball is tracked accurately using the previously described method. With the
Kalman filter, we have an estimate of the ball’s velocity and radius. The
ball’s velocity and radius can be used to extract the events generated
“inside” the ball, i.e., events generated by the logo. Events that we assume
are generated by the logo have to satisfy

$(y_{b}(t_{e})-y_{e})^{2}+(x_{b}(t_{e})-x_{e})^{2}<(r-pad)^{2},$ (3)

where $t_{e}$ is the event’s time stamp, $x_{e}$ the x-positon of the event,
$y_{e}$ the y-position of the event, $r$ the radius of the ball and $pad$ is a
tolerance to avoid extracting events from the edge of the ball.

For maximal accuracy, the ball’s position is continuously updated to the time
stamp of the selected event, according to

$\begin{split}\bm{p_{b}}(t_{e})&=(x_{b}(t_{e}),y_{b}(t_{e}))\\\
&=\bm{v_{b}}(t_{i})\cdot(t_{e}-t_{i})+\bm{p_{b}}(t_{i}),\end{split}$ (4)

where $t_{i}$ is the previous time the Kalman filter’s state was updated. With
this, we can get the events generated by the logo of a flying ball.

###  3.3 Spin Estimation

Frame-based table tennis spin estimation methods capture successive logo
orientations and then regress the spin. Though this method could be translated
to event-based cameras, it would not fully exploit their advantages.

Generally, optical flow is a good tool for estimating velocities or, in this
case, spin. As mentioned in Sec. 2, there are numerous optical flow methods
for event cameras.

We first explain how we obtain the ball’s spin $\bm{\omega}$ from the optical
flow $v_{f}$ generated by the logo. From the flow, we can calculate the
velocity $\bm{v}$ of the point on the ball’s surface where the flow is
generated

$\bm{v}=\begin{bmatrix}v_{f,x}\\\ v_{f,y}\\\
\frac{-v_{f,x}e_{r,x}-v_{f,y}e_{r,y}}{e_{r,z}}\end{bmatrix}.$ (5)

We can calculate the $\bm{z}$ component of $\bm{v}$ by knowing that the
velocity is tangent to the ball’s surface, $\bm{e_{r}}\cdot\bm{v}=0$. This is
sketched in Fig. 4.

Figure 4: Sketch showing how the spin is calculated from the optical flow.

We can then calculate the rotation vector from the cross-product of the radius
and the velocity.

$\bm{\omega}=\frac{\bm{e_{r}}\times\bm{v}}{||\bm{r}||}.$ (6)

When working with different optical flow methods, we noticed that they are
able to infer flow correctly when the logo is in the center of the ball. But
they fail as soon as the logo is on the edge of the ball. This is due to
multiple factors. First, table tennis ball logos have multiple edges. As such,
events are simultaneously generated at different places of the logo and not in
a straight line. This makes identifying successive events generated by the
same edge difficult. Secondly, because of the geometry of the ball, the logo’s
velocity is not constant. There are several parameters to tune the different
optical flow algorithms. The main one is the accumulation time, $t_{acc}$,
which aggregates events into an event frame used for optical flow. To obtain
the best optical flow results, the $t_{acc}$ should be chosen based on the
logo’s speed to ensure enough change to observe flow but not so much as to
cause motion blur. For a fast, first estimate, we choose $t_{acc}$ as a
constant value, based on the median of the expected spin speeds as
$t_{acc}=\lceil(\omega/10)\rceil$. From testing, we divided each rotation into
$10$ time slices because it ensures that even if the true spin speed is higher
than we estimated, we still observe the flow multiple times per rotation.

For a better estimate of the spin magnitude and therefore $t_{acc}$, we also
consider a second approach based on the variation of the event rate from the
extracted logo events. Indeed, peaks in the event rate indicates when the logo
is periodically the most visible. This approach is inspired by [29] and it
returns results after a longer observation time but with a higher accuracy. We
first apply an exponential moving average (EMA) and subtract the mean event
rate. Then, we apply a low pass filter to smooth the curve. After that, we
locate all the time stamps at which the sign of this curve transitions from
positive to negative values. Since the change in the event rate is periodic
and depends on the ball’s spin, as the logo appears and disappears, it can be
used to estimate the magnitude of the ball’s spin. Figure 5 shows how the sign
transitions of the event rates EMA can be used to calculate the magnitude of
the spin, by taking the time difference between adjacent time stamps,
calculating the mean, and converting this time per rotation into the number of
rotations per second.

Figure 5:  Prediction of the spin magnitude from the event rate. The
histograms of the event rate are displayed. We first apply an exponential
moving average (EMA) and subtract the mean event rate. Then, we apply a low
pass filter to smooth the curve. After that, we locate all the time stamps at
which the sign of this curve transitions from positive to negative values.

We can then apply the various optical flow methods for both methods of
choosing $t_{acc}$. The parameters are chosen for each one depending on
$t_{acc}$ to return the best results.

For each $t_{acc}$, several flows and consequently spins are estimated, with
the resulting value being the mean of all computed $\bm{\omega}$. This process
can be repeated for each $t_{acc}$ to generate additional spin estimations,
which are then averaged collectively.

##  4 Experiments

We start this section by describing our experimental setup in Sec. 4.1. In
Sec. 4.2 we evaluate the ball detector and in Sec. 4.3 we describe our
experiment to evaluate the spin estimation of our proposed approach.

###  4.1 Setup

We use a Prophesee EVK4 event camera with a resolution of $1280$x$720$ pixels
for capturing events. The camera is mounted $\sim 2\text{m}$ above the table
tennis table, equipped with a $8\text{mm}$ lens. This allows us to cover the
whole width of the table, as shown in Fig. 6.

Figure 6: Event camera recording setup

The camera’s focus was adjusted using the protocol provided by Prophesee [1].
Event-based cameras possess particular settings that differ from standard,
frame-based cameras. These were investigated and tuned to generate the most
events related to the ball and logo observation while minimizing noise. For
more information about the bias settings used with quantitative and
qualitative results, please refer to Sec. 6 in the supplementary material. We
use LED panels to light the scene uniformly from the ceiling, leading to a
luminosity of around $1930\;\text{Lux}$ measured on the table. This is to
avoid the flickering light from standard lighting powered with $50\text{Hz}$
AC, which triggers events over the entire scene in the event camera, though
events generated by the flickering light can also be filtered out with a band-
cut filter.

We rely on a custom-made static ball spinner and a ball thrower (Amicus Prime
from Butterfly) to generate recordings with known ground truth. The ground
truth spin can be calculated from the ball spinner by knowing its relative
position to the event camera. We mounted the balls with different logo
orientations on the ball spinner to study its impact on the spin estimation,
as shown in Fig. 7.

Figure 7: Balls with different logo orientations

For the ball thrower, we rely on SpinDOE [23] to generate ground truth spin,
using balls with a custom pattern drawn on them. A frame camera, the
Grasshopper3 GS3-U3-23S6C (1900x400), was installed next to the event camera
for capturing frames of the balls.

###  4.2 Ball detection

To evaluate the ball tracker, we recorded multiple balls flying in front of
the event camera with speeds ranging from $4$ m/s to $12$ m/s. The ground
truth was generated automatically with blob detection applied on accumulated
event frames with a large accumulation time. In Fig. 8, we show our ball
detector’s Mean Absolute Error (MAE) for different ball thrower settings
(arbitrary unit). We can see that we have sub-pixel accuracy, which is
necessary for extracting the events generated by the ball’s logo.

Figure 8:  Evaluation of our ball detector for different ball thrower settings
(integer values), reporting the MAE in pixels. The colored band around the
error represents the standard deviation.

*2c*7cp1.2cm[hvlines] green!103-33-9 green!105-35-9 green!107-37-9 green!109-39-9 & Ball 1 2 3 4 6 7
4-110ms 2-1Back/Top-spin 30.0 $\pm$ 28.7 27.0 $\pm$ 27.5 23.36 $\pm$ 26.4 22.4
$\pm$ 21.9 32.0 $\pm$ 27.3 32.5 $\pm$ 28.2
18.1 $\pm$ 10.4 ° 25.3 $\pm$ 2.0 ° 27.6 $\pm$ 12.7 ° 25.6 $\pm$ 3.9 ° 15.4
$\pm$ 19.6 ° 9.5 $\pm$ 7.2 °
2-1Sidespin - 31.0 $\pm$ 20.6 30.5 $\pm$ 18.8 32.0 $\pm$ 23.4 - -
\- 38.17 $\pm$ 14.6 ° 40.3 $\pm$ 14.3 ° 42.8 $\pm$ 7.9 ° - -
4-1100ms 2-1Back/Top-spin 0.8 $\pm$ 0.1 0.5 $\pm$ 0.1 2.8 $\pm$ 2.7 3.7 $\pm$
3.9 0.6 $\pm$ 0.1 0.5 $\pm$ 0.1
12.5 $\pm$ 4.0 ° 25.1 $\pm$ 1.8 ° 25.7 $\pm$ 1.4 ° 25.5 $\pm$ 1.9 ° 10.7 $\pm$
5.5 ° 6.2 $\pm$ 3.2 °
2-1Sidespin 4.8 $\pm$ 1.8 8.4 $\pm$ 1.5 8.0 $\pm$ 3.5 6.5 $\pm$ 3.2 - -
\- 16.7 $\pm$ 9.4 ° 11.6 $\pm$ 13.5 ° 24.0 $\pm$ 10.5 ° - -

Table 1:  Evaluation of our spin estimation with the ball spinner. Our test
was performed with balls with different logo positions relative to the spin
axis, as shown in Fig. 7 (white rows: MAE for the spin magnitude in rps, green
rows: MAE for the spin axis in degrees). $10$ms and $100$ms indicate the
amount of observation time allowed. Entries with ”-” indicate that we did not
get any results. For ball 1, the sidespin could not be estimated since the
logo only rotates around the spin axis of the ball spinner, and there is no
sidespin. The logo of balls 6 and 7 is a rotated version of ball 1; therefore,
the same restriction applies.  Figure 9:  (top) Spin magnitude (speed) error
with the ball spinner for different ball spins. (bottom) Spin axis estimation
error with the ball spinner for different ball spins. $10$ms and $100$ms
indicate the amount of observation time allowed. Only the events recorded
during that time window can be used for the estimation. Backspin/Topspin was
evaluated with balls 1, 6, and 7, and sidespin was evaluated with 2, 3, and 4,
with an equal number of samples.

###  4.3 Spin estimation

We tested our spin estimation method by two different means: with a ball
spinner and with a ball thrower. The former has the advantage to give us an
extremely accurate and reliable ground truth while the later is closer to real
use case. Both benchmarks are still quite similar from the way events
generated by the logo are extracted from the flying ball as explained in Sec.
3.2. Indeed, the extracted events’ position is recalculated using the ball
position as a reference.

We selected Triplet Matching [33] as the optical flow algorithm. Triplet
Matching is the fastest optical flow method we found. It has a sub-millisecond
run-time while maintaining competitive accuracy. Moreover, it runs on a CPU,
which limits hardware requirements.

We distinguish two types of spin: backspin/topspin and sidespin. Because the
ball is observed from the top, backspin, and topspin are almost the same, if
only mirrored. With top and backspin, the logo will come into view of the
camera and then disappear again. Sidespin balls, on the other hand, will
either always have their logo hidden or always visible.

####  4.3.1 Ball spinner benchmark

We used table tennis balls with different logo orientations to evaluate our
proposed spin estimation approach, as shown in Fig. 7. Next to the spin
magnitude, our method also estimates the spin axis. The spin magnitude
estimation error and the spin axis estimation error with the ball spinner for
different ball types are shown in Tab. 1.

Entries with ”-” indicate that we did not get any results. For ball 1, the
sidespin could not be estimated since the logo and spin axis coincide. The
logo of balls 6 and 7 is a rotated version of ball 1; therefore, the same
restriction applies.

The errors with the ball spinner for different ball spins are reported in the
top part of Fig. 9. As can be seen in the plot, the MAE for the lowest spin
magnitude of $10\text{rps}$ is very high. The reason is twofold. First, due to
the relatively slow rotation speed, the logo triggers fewer events, which
makes it difficult for the optical flow to get accurate flow estimations.
Second, with the relatively slow rotation speed, the likelihood of observing
the logo over $10$ms is relatively low. With $20$rps, the MAE is already lower
and relatively stable up to $100$rps for all four cases except for the
sidespin with $10$ms observation time. For sidespin with $10$ms observation
time, the MAE increases linearly with the spin speed. This underlines the
difficulty of estimating sidespin.

The MAE of the spin axis estimation errors with the ball spinner for different
ball spins are plotted in the bottom graph of Fig. 9. As can be seen in the
plot, the spin axis estimation error decreases with higher spin values. This
is because higher spin values lead to more events, allowing the optical flow
algorithm to achieve higher accuracy.

####  4.3.2 Ball thrower benchmark

Next to the evaluation with a ball spinner, we also deployed our method in a
setup to estimate the spin of flying balls. A ball thrower shot $20$ balls. We
used velocity settings of $10$, $15$, $20$, and $25$ for the ball thrower
(respectively approximately $4$, $5.5$, $7.5$ and $9$ m/s ) and top- and side-
spin, with spin strength settings ranging from $-5$ to $7$. Both our event-
based approach and a state-of-the-art frame-based approach [39] were tested on
the same observations. The results are listed in Tab. 2.

| Frame | Event
---|---|---
Success rate | 0.79 | 0.81
Spin axis MAE | $26.6\pm 33.4\degree$ | $32.9\pm 38.2\degree$
Spin magnitude MAE |  $8.8\pm 15.6$ rps |  $10.7\pm 17.3$ rps
Table 2: Benchmark with the flying balls from a ball thrower

The success rate in the first row is the ratio between the number of
successful spin estimations and the total number of ball trajectories. As can
be seen, our approach has a success rate slightly above the frame-based
approach. Our approach is a bit behind in terms of spin axis and spin
magnitude MAE, however, with overlapping error ranges. Given that there is
already a considerable body of literature covering frame-based spin estimation
methods and our method is the first event-based one, this should show the
potential of our approach.

In Fig. 10, we show the spin magnitude MAE on the top and the spin axis MAE on
the bottom for a state-of-the-art frame-based approach [39] and our event-
based approach.

Figure 10:  (top) Spin magnitude error with the ball thrower for different
ball spins. (bottom) Spin axis estimation error with the ball thrower.

As expected, we see that the frame-based approach has more problems with
higher spin settings, while the event-based approach struggles with low spin
settings. It is worth noting that though both frame and event cameras have the
same FOV, the frame camera boasts a higher resolution ($\sim\times 1.5$). With
a resolution similar to the frame camera, the event-based spin estimation
would work better. With these complementarity properties, an approach
combining an event-based camera and a frame-based camera could be an option.

###  4.4 Run times

Since our goal is a real time capable spin estimation pipeline, the run-time
is another important metric. We used an Intel Core i7-9700 CPU @ 3.00GHz for
our experiments, with all the components written in Python.

From the moment the events are received from the camera, generating the EROS
time surface, detecting and tracking the ball, and extracting the logo events
takes $12.8\pm 1.2\text{ms}$. We run the spin estimation in post-processing
and report the ratio between the time it took to process all the events and
the duration of the recording. With a ratio of $0.84\pm 0.8$, the spin
estimation can process more data than it receives and, therefore, runs in real
time.

While it can already be considered capable of being in real time, the setup
could be improved. Improving the software design and implementing the software
pipeline in C++ instead of Python is expected to boost performance.

###  4.5 Limitations

While working on the spin estimation for the flying balls from a ball thrower,
we noticed fewer events being generated compared to the ball spinner setup.
This led to an increase in spin estimation failures because there were too few
events generated by the logo for the optical flow estimation. Our
interpretation of this phenomenon is that pixels respond slower to negative
luminosity changes, such as the movement of a logo, after being triggered by a
positive change, such as the appearance of the leading edge of a flying ball.
This is sometimes referred to as the event camera equivalent of motion blur.
Though our method works with the Andro logo displayed in Fig. 11(a), it does
not generalize to all logos. For example, our spin estimation algorithm does
not work with the ball in Fig. 11(b) since the logo does not generate enough
events. This is despite the fact that it works with the ball spinner. We
tested different custom logos to determine the causing factor. The logo lines
must be thicker than a certain threshold to be detected. We drew lines of
different line widths on logoless balls to show this. The ball with the
$0.8$mm line shown in Fig. 11(c) did not generate enough events, but the ball
with $1.4$mm line shown in Fig. 11(d) worked well with our spin estimation
method.

(a) Logo 1

(b) Logo 2

(c) 0.8mm line

(d) 1.4mm line

Figure 11: Different ball logos used to estimate the spin of the ball

As such, only logos with lines of a certain width can be used for our spin
estimation with an event camera. Of course, there are different solutions to
circumvent this. We could use an event camera with a higher resolution or a
lens with a longer focal length. However, with the latter solution, the
camera’s field of view would no longer cover the whole width of table.

We also noticed that the spin estimation has a singularity. Indeed, out of all
the balls shown in Fig. 7, there is one logo position, ball 5, for which the
spin estimation failed completely, whatever the spin type. Ball 5 does not
work for backspin/topspin because the logo is barely visible on the side of
the ball. For sidespin, the estimation failed because the rotation axis and
the logo coincided. This could potentially be circumvented by using an optical
flow algorithm better suited to our specific application.

##  5 Conclusion

In this work, we presented a spin estimation method for table tennis using
optical flow based on the event stream of an event camera. Though this method
was implemented and tested for table tennis, it could be generalized to other
sports. We were able to compensate for the drawbacks of frame-based cameras
with an event-based camera, which allows for more reliable spin estimation.

Although we have shown that our presented approach works for balls on a ball
spinner and also for flying balls, depending on the setup, there are still
some limitations, notably the slower reponse of OFF events of pixels after
being triggered ON.

For future work, we plan to improve our setup and generate a dataset of event
recordings of spinning balls to train a neural network specifically for event-
based spin estimation. This would allow us to bypass the optical flow
estimation. With such a method, we hope to achieve faster and more accurate
spin estimation.

## References

  * [1] Focus adjustment (c++) - metavision sdk docs 4.5.2 documentation.
  * Achterhold et al. [2023] Jan Achterhold, Philip Tobuschat, Hao Ma, Dieter Büchler, Michael Muehlebach, and Joerg Stueckler.  Black-box vs. gray-box: A case study on learning table tennis ball trajectory prediction with spin and impacts.  In _Proceedings of the 5th Annual Learning for Dynamics and Control Conference (L4DC)_ , pages 878–890. PMLR, 2023.
  * Alzugaray and Chli [2020] Ignacio Alzugaray and Margarita Chli.  HASTE: multi-hypothesis asynchronous speeded-up tracking of events.  In _31st British Machine Vision Conference 2020, BMVC 2020_. BMVA Press, 2020.
  * Aung et al. [2018] Myo Tun Aung, Rodney Teo, and Garrick Orchard.  Event-based plane-fitting optical flow for dynamic vision sensors in fpga.  In _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2018.
  * Blank et al. [2015] Peter Blank, Julian Hoßbach, Dominik Schuldhaus, and Bjoern M. Eskofier.  Sensor-based stroke detection and stroke type classification in table tennis.  In _Proceedings of the 2015 ACM International Symposium on Wearable Computers - ISWC ’15_. ACM Press, 2015.
  * Blank et al. [2017] Peter Blank, Benjamin H. Groh, and Bjoern M. Eskofier.  Ball speed and spin estimation in table tennis using a racket-mounted inertial sensor.  In _Proceedings of the 2017 ACM International Symposium on Wearable Computers_. ACM, 2017.
  * Boracchi et al. [2008] Giacomo Boracchi, Vincenzo Caglioti, and Alessandro Giusti.  _Single-Image 3D Reconstruction of Ball Velocity and Spin From Motion Blur - An Experiment in Motion-from-Blur._ SciTePress - Science and and Technology Publications, 2008.
  * Camunas-Mesa et al. [2018] Luis A. Camunas-Mesa, Teresa Serrano-Gotarredona, Sio-Hoi Ieng, Ryad Benosman, and Bernabe Linares-Barranco.  Event-driven stereo visual tracking algorithm to solve object occlusion.  _IEEE Transactions on Neural Networks and Learning Systems_ , 29(9):4223–4237, 2018.
  * Chamorro et al. [2020] William Chamorro, Juan Andrade-Cetto, and Joan Solà.  High speed event camera tracking.  In _31st British Machine Vision Conference 2020, BMVC 2020_. BMVA Press, 2020.
  * Chen et al. [2019] Haosheng Chen, Qiangqiang Wu, Yanjie Liang, Xinbo Gao, and Hanzi Wang.  Asynchronous tracking-by-detection on adaptive time surfaces for event-based object tracking.  In _Proceedings of the 27th ACM International Conference on Multimedia_. ACM, 2019.
  * Chen et al. [2010] Xiaopeng Chen, Ye Tian, Qiang Huang, Weimin Zhang, and Zhangguo Yu.  Dynamic model based ball trajectory prediction for a robot ping-pong player.  In _2010 IEEE International Conference on Robotics and Biomimetics_. IEEE, 2010.
  * Chiberre et al. [2021] Philippe Chiberre, Etienne Perot, Amos Sironi, and Vincent Lepetit.  Detecting stable keypoints from events through image gradient prediction.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2021.
  * Delbruck et al. [2021] Tobi Delbruck, Rui Graca, and Marcin Paluch.  Feedback control of event cameras.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2021.
  * Falanga et al. [2020] Davide Falanga, Kevin Kleber, and Davide Scaramuzza.  Dynamic obstacle avoidance for quadrotors with event cameras.  _Science Robotics_ , 5(40), 2020.
  * Forrai et al. [2023] Benedek Forrai, Takahiro Miki, Daniel Gehrig, Marco Hutter, and Davide Scaramuzza.  Event-based agile object catching with a quadrupedal robot.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023.
  * Furuno et al. [2009] Sho Furuno, Kazuyuki Kobayashi, Tomoyuki Okubo, and Yosuke Kurihara.  A study on spin-rate measurement using a uniquely marked moving ball.  In _2009 ICCAS-SICE_ , pages 3439–3442. IEEE, 2009.
  * Gallego et al. [2022] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jörg Conradt, Kostas Daniilidis, and Davide Scaramuzza.  Event-based vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(1):154–180, 2022.
  * Gao et al. [2021] Yapeng Gao, Jonas Tebbe, and Andreas Zell.  _Robust Stroke Recognition via Vision and IMU in Robotic Table Tennis_ , page 379–390.  Springer International Publishing, 2021.
  * Gava et al. [2022] Luna Gava, Marco Monforte, Massimiliano Iacono, Chiara Bartolozzi, and Arren Glover.  Puck: Parallel surface and convolution-kernel tracking for event-based cameras, 2022.
  * Gehrig et al. [2024] Mathias Gehrig, Manasi Muglikar, and Davide Scaramuzza.  Dense continuous-time optical flow from event cameras.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , page 1–12, 2024.
  * Glover et al. [2022] Arren Glover, Aiko Dinale, Leandro De Souza Rosa, Simeon Bamford, and Chiara Bartolozzi.  luvharris: A practical corner detector for event-cameras.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(12):10087–10098, 2022.
  * Glover and Kaelbling [2014] Jared Glover and Leslie Pack Kaelbling.  Tracking the spin on a ping pong ball with the quaternion bingham filter.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014.
  * Gossard et al. [2023] Thomas Gossard, Jonas Tebbe, Andreas Ziegler, and Andreas Zell.  Spindoe: A ball spin estimation method for table tennis robot.  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023.
  * Im et al. [2022] Woobin Im, Sebin Lee, and Sung-Eui Yoon.  _Semi-supervised Learning of Optical Flow by Flow Supervisor_ , page 302–318.  Springer Nature Switzerland, 2022.
  * Jonschkowski et al. [2020] Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova.  _What Matters in Unsupervised Optical Flow_ , page 557–572.  Springer International Publishing, 2020.
  * Kulkarni and Shenoy [2021] Kaustubh Milind Kulkarni and Sucheth Shenoy.  Table tennis stroke recognition using two-dimensional human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , pages 4576–4584, 2021.
  * Lucas and Kanade [1981] Bruce D. Lucas and Takeo Kanade.  An iterative image registration technique with an application to stereo vision (darpa).  In _Proceedings of DARPA Image Understanding Workshop (IUW ’81)_ , pages 121 – 130, 1981.
  * Nagata et al. [2021] Jun Nagata, Yusuke Sekikawa, and Yoshimitsu Aoki.  Optical flow estimation by matching time surface with event-based cameras.  _Sensors_ , 21(4):1150, 2021.
  * Pfrommer [2022] Bernd Pfrommer.  Frequency cam: Imaging periodic signals in real-time, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  _U-Net: Convolutional Networks for Biomedical Image Segmentation_ , page 234–241.  Springer International Publishing, 2015.
  * Sato and Aono [2020] Soichiro Sato and Masaki Aono.  Leveraging human pose estimation model for stroke classification in table tennis.  In _Working Notes Proceedings of the MediaEval 2020 Workshop, Online, 14-15 December 2020_. CEUR-WS.org, 2020.
  * Scheerlinck et al. [2020] Cedric Scheerlinck, Henri Rebecq, Daniel Gehrig, Nick Barnes, Robert E. Mahony, and Davide Scaramuzza.  Fast image reconstruction with an event camera.  In _2020 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2020.
  * Shiba et al. [2022] Shintaro Shiba, Yoshimitsu Aoki, and Guillermo Gallego.  Fast event-based optical flow estimation by triplet matching.  _IEEE Signal Processing Letters_ , 29:2712–2716, 2022.
  * Su et al. [2013] Hu Su, Zaojun Fang, De Xu, and Min Tan.  Trajectory prediction of spinning ball based on fuzzy filtering and local modeling for robotic ping–pong player.  _IEEE Transactions on Instrumentation and Measurement_ , 62(11):2890–2900, 2013.
  * Szep [2011] Alexander Szep.  Measuring ball spin in monocular video.  In _Proc. 16th Comput. Vis. Winter Workshop_ , pages 83–89. Citeseer, 2011.
  * Tamaki et al. [2024] Sho Tamaki, Satoshi Yamagata, and Sachiko Hashizume.  Spin measurement system for table tennis balls based on asynchronous non-high-speed cameras.  _International Journal of Computer Science in Sport_ , 23(1):37–53, 2024.
  * Tamaki et al. [2004] Toru Tamaki, Takahiko Sugino, and Masanobu Yamamoto.  Measuring ball spin by image registration.  _Proc. 10th Frontiers of Computer Vision_ , pages 269–274, 2004.
  * Tamaki et al. [2012] Toru Tamaki, Haoming Wang, Bisser Raytchev, Kazufumi Kaneda, and Yukihiko Ushiyama.  Estimating the spin of a table tennis ball using inverse compositional image alignment.  In _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2012.
  * Tebbe et al. [2020] Jonas Tebbe, Lukas Klamt, Yapeng Gao, and Andreas Zell.  Spin detection in robotic table tennis.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  _RAFT: Recurrent All-Pairs Field Transforms for Optical Flow_ , page 402–419.  Springer International Publishing, 2020.
  * Theobalt et al. [2004] Christian Theobalt, Irene Albrecht, Jörg Haber, Marcus Magnor, and Hans-Peter Seidel.  Pitching a baseball: tracking high-speed motion with multi-exposure images.  _ACM Transactions on Graphics_ , 23(3):540–547, 2004.
  * Wang et al. [2024] Yuxin Wang, Zhiyong Sun, Yongle Luo, Haibo Zhang, Wen Zhang, Kun Dong, Qiyu He, Qiang Zhang, Erkang Cheng, and Bo Song.  A novel trajectory-based ball spin estimation method for table tennis robot.  _IEEE Transactions on Industrial Electronics_ , page 1–11, 2024.
  * Zhang et al. [2015] Yifeng Zhang, Rong Xiong, Yongsheng Zhao, and Jianguo Wang.  Real-time spin estimation of ping-pong ball using its natural brand.  _IEEE Transactions on Instrumentation and Measurement_ , 64(8):2280–2290, 2015.
  * Zhu et al. [2024] Yabin Zhu, Xiao Wang, Chenglong Li, Bo Jiang, Lin Zhu, Zhixiang Huang, Yonghong Tian, and Jin Tang.  Crsot: Cross-resolution object tracking using unaligned frame and event cameras.  _arXiv preprint arXiv: 2401.02826_ , 2024.

Supplementary Material

##  6 Camera settings

The Prophesee EVK4 has five bias settings. The name ’bias’ comes from the
electronics domain. In electronics, ’biasing’ usually refers to a fixed DC
voltage or current applied to an electronic component, in order to establish
proper operating conditions for the component. However, despite the name, bias
settings do not have to directly correlate with any voltage or current but is
rather just a numerical value to adjust the property of the event camera. The
bias settings of the Prophesee EVK4 are listed in Tab. 3. On all the
corresponding plots, the green line indicates our chosen setting.

$\textit{bias\\_on}=40$ |  Contrast threshold for triggering on events
---|---
$\textit{bias\\_off}=40$ |  Contrast threshold for triggering off events
$\textit{bias\\_fo}=55$ |  Cutoff frequency for the low-pass filter
$\textit{bias\\_hpf}=0$ |  Cutoff frequency for the high-pass filter
$\textit{bias\\_refr}=80$ |  Pixels’ refractory period
Table 3:  List of the biases/settings of the event camera we used. Their tuned
values (default values are $0$) and what they control (this is the naming
convention used for Prophesee cameras)

To make the most out of event cameras, tuning the biases is essential to
capture all the necessary events while maintaining a low level of noise. This
task is quite involved as the biases are interdependent. Estimating the
signal-to-noise ratio in a dynamic scene is also challenging since recreating
the same scene for an objective comparison of the biases is not trivial.
Moreover, optimal biases are often task-specific and cannot be generalized.
Automatic bias tuning was investigated in [13]. The authors used the event
rate (ER) as the metric for finding the optimal biases, by tuning the biases
independently. With our setup, we have a static background and the only moving
object in the scene is the table tennis ball. This makes the ER a reliable
metric for tuning the biases. To recreate identical observations for different
settings, we relied on a ball thrower to have the same ball trajectory every
time. However, the ball thrower can not guarantee the same logo position. For
this reason, we used a robot arm to move the ball exactly the same way, with
the required ball orientation. This method, however, was only used for
qualitative results (Figs. 13, LABEL: and 18), as the moving robot arm
influenced the event rate.

####  6.0.1 bias_on and bias_off

Tuning the pixel sensitivity of the event camera with the biases bias_on and
bias_off is a trade-off between having as little noise as possible while still
capturing relevant events for our task. Increasing bias_on and bias_off will
increase the contrast threshold and, therefore, decreases the sensor’s
sensitivity.

Figure 12(a) displays the event-rate for a static scene with regard to bias_on
and bias_off.

(a) Event-rate for different bias_on and bias_off values while observing a
static scene, where events can be considered to be noise. The y-axis is in
log-scale.

(b) Event-rate for different bias_on and bias_off values while observing a
flying ball (averaged over 5 samples)

Figure 12:  Event-rate for different pixel sensitivity (bias_on = bias_off)

As the scene is completely static, the generated events can be considered
noise. The noise is decreased to an acceptable level starting when bias_on/off
goes over 40.

We also measured the event rate when a ball was flying in front of the camera.
We noticed that increasing bias_on/off further led to an unequal number of
ON/OFF events when observing the ball, as shown in Fig. 12(b). This is due to
the none symetrical behavior of the event pixels for OFF and ON events. To
avoid this behaviour, bias_on/off higher than 60 are to be avoided. As such,
we decided to set the bias_on/off to 40.

In Fig. 13, we show the effects of different bias_on/off on the accumulated
event frame. We can clearly see a decrease in the number of generated events
for increasing bias_on/off.

(a) $\textit{bias\\_on/off}=40$

(b) $\textit{bias\\_on/off}=90$

(c) $\textit{bias\\_on/off}=140$

Figure 13: Accumulated event frames of the moving ball with the logo on the
edge for different bias_on/off values (with an accumulation time of
$3\text{ms}$)

####  6.0.2 bias_fo

The bias_fo controls the pixel’s low-pass cut-off frequency. It can filter out
events generated by fast motions and flickering. Increasing bias_fo will also
increase the cut-off frequency. There is a trade-off between noise and
latency. Indeed, a low-pass filter with a lower cut-off frequency will also
introduce higher latency to the events as they will be triggered with some
delay. This can be observed in Fig. 14(a) where more new events are being
triggered inside the ball.

(a) $\textit{bias\\_fo}=-35$

(b) $\textit{bias\\_fo}=55$

Figure 14: Linear time-surfaces ($\tau=1ms$) of a flying ball with different
bias_fo.

These events are “late” events generated by the edge of the ball. This can
also be observed by the less distinct edge of the ball for $bias\\_fo=-35$. To
avoid such “late” events being mixed up with events from the logo, the bias_fo
was set to 55, which increased the cut-off frequency to the maximum. However,
from the event rate for different bias_fo values shown in Fig. 15, we see that
this setting only affects the event rate for negative bias_fo values.

Figure 15: Event-rate for different bias_fo values while observing a flying
ball (averaged over 5 samples)

So any value between $0$ and $55$ would also work.

####  6.0.3 bias_hpf

The bias_hpf controls the pixel’s high-pass cut-off frequency. It allows
filtering out low-frequency events such as noise and events generated from
slow motions. The event rate for different bias_hpf is shown in Fig. 16. We
set bias_hpf to the minimum of $0$ for filtering out the least number of
events. All in all, the settings for bias_fo and bias_hpf were chosen to
filter out the minimum amount of events.

Figure 16: Event-rate for different bias_hpf values while observing a flying
ball (averaged over 5 samples)

####  6.0.4 bias_refr

The bias_refr controls the pixel’s refractory time, which is the time during
which a pixel does not detect any change in illumination after it emitted an
event. Decreasing the pixel’s refractory period will generate more events for
a large illumination change. A shorter refractory period will also lead to a
higher event rate. The event-rate for different bias_refr is shown in Fig. 17.

(a) Event-rate for different bias_refr values while observing a static scene,
where events can be considered to be noise

(b) Event-rate for different bias_refr values while observing a flying ball
(averaged over 5 samples)

Figure 17: Event-rate for different bias_refr values

We increased bias_refr to $80$ to not miss any events, which is also suggested
by the camera’s documentation. We do not want the pixel to be “dead” at the
ball’s edge when the logo comes into view. Thus, as soon as the edge moves to
another pixel, that pixel should be able to trigger new events. Since the
nominal velocity of the flying ball is around $6000\;\text{pixels/s}$, the
edge will move to another pixel in approximately every $0.1\text{ms}$.
Increasing bias_refr to values higher than 80, i.e., shortening the refractory
period results in an increase in noise, as shown in Fig. 17 where the ON event
rate keeps increasing with bias_refr. On the other hand, a higher refractory
period results in too few events for the spin estimation to work accurately
Fig. 18(c).

(a) $\textit{bias\\_refr}=120$

(b) $\textit{bias\\_refr}=80$

(c) $\textit{bias\\_refr}=-20$

Figure 18:  Accumulated event frames of the moving ball with the logo on the
edge for different bias_refr values (with an accumulation time of
$2\text{ms}$). A shorter refractory period of $120$ results in too many
events/noise on the edge. On the other hand, a longer refractory period of
$-20$ results in too few events for the spin estimation to work accurately.
Therefore, we used a value of $80$, as a trade-off.

##  7 Filters

In addition to the camera’s bias, filters can be applied to the event stream
to cancel redundant information or filter out noise. The two options we
considered are the Spatio-Temporal-Contrast (STC) filter and the TRAIL filter,
as well as their combination. The STC filter filters out isolated events that
are not followed by other events of the same polarity. It does so by only
retaining the second event from a burst of events. The time window during
which the second event is waited for can be tuned.

The TRAIL filter, as its name implies, gets rid of events that happen “behind”
a moving edge with a certain time window, except if the event is of the
opposite polarity. These filters help clean the event stream by reducing noise
and redundant information. Both filters can be combined depending on the
objective, only leaving one event out of a burst, giving much cleaner edges.
It should be noted that the filter’s time window parameter should be tuned
depending on the ball’s velocity.

In Fig. 19 we show how increasing the filter threshold affects the event rate
of a flying ball.

Figure 19: Effect of filter threshold on the event rate of a flying ball for
different filter options

For the STC filter, we keep very few of the observed events at low threshold
values, as there are not enough events that are generated in quick enough
succession to count as one burst. As the filter threshold increases, we still
only keep a portion of the overall events, as for each burst (successive
events at the same location), the filter discards the first one. For the TRAIL
filter, the opposite is true. For each burst, only the first event is kept,
and subsequent events within the threshold are discarded if they have the same
polarity. Thus, with a higher threshold, fewer events are kept. Applying the
STC and then the TRAIL filter (stc-cut-trail) keeps only one event of the same
polarity for each burst. Due to this, stc-cut-trail discards the most events
but leads to the cleanest edges.

Fig. 20 shows the events that the different filters keep depending on their
threshold parameter, with the unfiltered version at the top.

Figure 20:  Time surfaces (positive and negative events, $5$ms accumulation
time), showing the effect of the filter threshold on events kept by the
different filter combinations. Top row: no filter.

For the STC filter, on low threshold values, very few events are being kept,
as not enough events fall into the threshold time. Especially for the front
edge (top in image), we can see that it takes a higher threshold value before
enough events are kept to make it out clearly. It is worth noting that when
the logo follows the front edge closely (as in the figure), the front edge
will sometimes disappear. It is not entirely clear why this happens. However,
we believe it may be because not enough events from the front edge are
generated before the logo’s events are triggered. Due to this, the STC filter
removes the thin line of ON-events from the ball’s edge, as it is treated as
the start of the burst of the logo edge. As clearly visible, the STC filter
removes a lot of noise from the recording, as noise events are usually
isolated and not part of a burst of events generated by a moving edge.
However, behind the initial edge, there are still a lot of trailing events.

The TRAIL filter’s effects are very clearly visible as the threshold value
increases, especially at the back (bottom) of the ball. After an event from
the edge is recorded, subsequent events of the same polarity that fall into
the threshold time, are removed. This means that many trailing events
generated by the edge are cut out. We can clearly see the gap behind the back
edge increase as the filter threshold increases. Note, however, that the logo
is not being removed behind the front edge, as its polarity change resets the
filter window.

Combining the STC and trail filter leaves us with a very clear edge and little
remaining noise. Increasing the threshold over 5000$\mu$s changed very little
about the number and quality of events kept, so we use this as our
$filter\\_threshold$ when applying STC-cut-trail filter.

##  8 Optical flow estimation

Figure 21: HSV color wheel used to represent the optical flow

We represent the flow using the HSV color scheme for all the optical flow
estimations in Fig. 21, where high saturation indicates high flow, and the hue
indicates the flow direction.

In Fig. 22, we show examples of accumulated event frames and flow estimates of
sidespin for different spin values.

Figure 22:  Optical flow for sidespin with $t_{acc}=0.715ms$. The arrows
indicate the direction of the spin.

As clearly visible, higher spin values lead to more events, allowing more
accurate flow estimation.

Examples of accumulated event frames and corresponding flow estimates of
backspin/topspin for different spin values are visualized in Fig. 23.

Figure 23:  Optical flow for backspin/topspin with $t_{acc}=0.715ms$. The
arrows indicate the direction of the spin.

The same as for the sidespin, higher spin values result in more events and,
thus, more accurate optical flow.

##  9 Cleaning the EROS time surface

Even with the STC filter, noise makes it through to the EROS time surface. Not
only noise can pollute the time surface, but also trail events, as seen in
Fig. 24(a). To avoid misdetecting circles with the Hough Transform, we
eliminate these isolated events with a hit-or-miss detection. The EROS time
surface without cleaning is shown in Fig. 24(a) and with cleaning in Fig.
24(b).

(a) No cleaning

(b) With cleaning

Figure 24: The EROS time surface (a) without cleaning and (b) with cleaning

The kernel we run for the hit-and-miss is defined as

$k=\begin{bmatrix}-1&-1&-1&-1\\\ -1&1&1&-1\\\ -1&1&1&-1\\\
-1&-1&-1&-1\end{bmatrix}.$ (7)

##  10 Generating ground truth for the ball detector benchmark

We rely on event accumulation frames and blob detection to automatically
generate ground truth for the ball detection. To do so, we use a large
accumulation time of $t_{acc}=10\text{ms}$ to generate accumulated event
frames. This makes the ball clearly visible for blob detection, as shown in
Fig. 25.

Figure 25:  Example of ball position labeling with blob detection. The red
circle is the ball detected by the blob detection.

##  11 Ball thrower benchmark

To capture the same ball trajectories, the event and frame cameras were
installed next to each other. Despite trying to align them as best as
possible, it is not possible for them to exactly share the same field of view.
We thus needed to calculate the transformation from the event camera to the
frame camera for the benchmark. The transformation between both cameras was
calculated in the same fashion as for a stereo camera setup. A chessboard
pattern displayed on a LCD screen was used for that purpose. The screen was
set to blink for the event camera and to static display for the frame camera.
Thanks to this, the spin calculated with the event camera could be transformed
to match the spin calculated with the frame camera.

Regarding the generation of the ground truth, SpinDOE [23] was used. It can
indeed estimate the table tennis ball spin with high accuracy. It has a
relative error of $1\%$ for the spin magnitude and a spin axis error of
$2.4\degree$. However, the ball thrower used has some stochasticity to its
behaviour: even though the settings are the same, the ball will not have the
same trajectory. To compensate for this, dot-patterned balls were shot 5 times
with the same settings for the ball thrower and the ground truth is assumed to
be the mean spin vector.

| -5 | -4 | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
---|---|---|---|---|---|---|---|---|---|---|---|---|---
25 | 137.81 | 124.60 | 107.42 | 103.23 | 98.22 | 81.83 | 63.69 | 38.87 | 35.72 | 41.38 | 62.41 | - | -
20 | 127.10 | 110.70 | 92.45 | 93.09 | 88.47 | 72.36 | 60.61 | 45.53 | 45.40 | 53.64 | 64.80 | - | -
15 | 119.51 | 98.76 | 81.94 | 81.89 | 72.69 | 66.75 | 63.80 | 57.20 | 57.48 | 63.80 | 68.69 | - | -
10 | 92.31 | 74.41 | 60.74 | 61.23 | 58.74 | 57.85 | 61.46 | 61.80 | 62.27 | 65.54 | 66.11 | - | -
(a) Sidespin

| -5 | -4 | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7
---|---|---|---|---|---|---|---|---|---|---|---|---|---
25 | 111.81 | 90.41 | 64.34 | 65.46 | 53.42 | 34.13 | 17.01 | 16.61 | 35.10 | 54.20 | 74.14 | - |
20 | 99.99 | 78.42 | 55.22 | 53.91 | 39.42 | 13.30 | 2.05 | 22.39 | 39.94 | 54.05 | 68.90 | - | -
15 | 93.11 | 71.01 | 47.27 | 39.65 | 22.80 | 5.65 | 4.77 | 25.30 | 40.17 | 54.37 | 62.67 | 95.77 | -
10 | 72.33 | 54.58 | 36.46 | 23.41 | 9.21 | 0.91 | 7.41 | 17.10 | 25.04 | 34.94 | 38.67 | 59.75 | 88.76
(b) Back/top-spin

Table 4: Ground truth values of the spin magnitude [rps] calculated with
SpinDOE for the ball thrower. The rows represent the different velocity
settings and the columns the spin settings.

In Tab. 4, we list all the ground truth values for the different velocity and
spin settings. The ground thruth values for the higher spins could not always
be calculated because of the motion blur due to high spins. It should be noted
that with the ball thrower used, the ball spin magnitude also increased with
higher ball velocity settings.

  *[ROI]: Region of Interest
  *[MAE]: Mean Absolute Error