# Unveiling the outer dust disc of TW Hya with deep ALMA observations

John D. Ilee1, Catherine Walsh1, Jeff Jennings2, Richard A. Booth3, Giovanni
P. Rosotti4, Richard Teague5, Takashi Tsukagoshi6 and Hideko Nomura7
1School of Physics and Astronomy, University of Leeds, Leeds LS2 9JT, UK
2Institute of Astronomy, University of Cambridge, Madingley Road, Cambridge,
CB3 0HA, UK
3Astrophysics Group, Imperial College London, Blackett Laboratory, Prince
Consort Road, London SW7 2AZ, United Kingdom
4School of Physics & Astronomy, University of Leicester, University Road,
Leicester, LE1 7RH, UK
5Center for Astrophysics | Harvard & Smithsonian, 60 Garden St., Cambridge, MA 02138, USA
6Division of Radio Astronomy, National Astronomical Observatory of Japan,
Osawa 2-21-1, Mitaka, Tokyo 181-8588, Japan
7Department of Earth and Planetary Sciences, Tokyo Institute of Technology,
2-12-1 Ookayama, Meguro, Tokyo, 152-8551, Japan E-mail<EMAIL_ADDRESS>(JDI)

(Accepted 2022 April 29. Received 2022 April 26; in original form 2021
December 17)

###### Abstract

The radial extent of millimetre dust in protoplanetary discs is often far
smaller than that of their gas, mostly due to processes such as dust growth
and radial drift. However, it has been suggested that current millimetre
continuum observations of discs do not trace their full extent due to limited
sensitivity. In this Letter, we present deep (19 $\mu$Jy beam-1) moderate
resolution (0.37″) ALMA observations at 1 mm of the nearest protoplanetary
disc, TW Hya. Using the visibility analysis tool frank, we reveal a structured
millimetre intensity distribution out to 100 au, well beyond previous
estimates of 60–70 au. Our analysis suggests the presence of a new millimetre
continuum gap at 82 au, coincident with similar features seen in optical/near-
infrared scattered light and millimetre molecular line observations.
Examination of the fit residuals confirms the presence of the previously
reported au-scale continuum excess at 52 au ($\mathrm{P.A.}=242.5\degr$). Our
results demonstrate the utility of combining deep, moderate resolution
observations with super-resolution analysis techniques to probe the faintest
regions of protoplanetary discs.

###### keywords:

protoplanetary discs – stars: individual: TW Hya – techniques: interferometric
– submillimetre: planetary systems

††pubyear: 2022††pagerange: Unveiling the outer dust disc of TW Hya with deep
ALMA observations–Unveiling the outer dust disc of TW Hya with deep ALMA
observations

##  1 Introduction

Observations of planet-forming discs have shown that the radial extent of
their millimetre dust component is, generally, smaller than the extent of
molecular gas in the disc (e.g. Panić et al., 2009; Andrews et al., 2012).
Initially, this difference was attributed to limited sensitivity (see Dutrey
et al., 1998), but further observations revealed that the gas and dust cannot
be reproduced with the same surface density profile (Hughes et al., 2008). In
particular, observations with the Atacama Large Millimetre/submillimetre Array
(ALMA) demonstrated that a sharp decrease in dust surface at the outer edge is
required (e.g. de Gregorio-Monsalvo et al., 2013; Andrews et al., 2016).

A proposed explanation for these differing radial extents involves the growth
and inward drift of dust grains through the disc. Observations of discs at cm
wavelengths have confirmed that dust can grow to mm-cm sizes (see, e.g.,
Rodmann et al., 2006; Ricci et al., 2012) and that the maximum grain size
increases toward the inner disc (Pérez et al., 2012; Tazzari et al., 2016).
These characteristics are predicted by models of growth and radial drift
(Birnstiel et al., 2010; Birnstiel et al., 2012). Indeed, such models
naturally result in a sharp outer edge of the dust distribution (or sharp
decrease in the dust-to-gas mass ratio) that is compatible with observations
(Birnstiel & Andrews, 2014). However, observed continuum fluxes and radial
extents suggest that drift needs to be halted (or significantly slowed) in the
majority of discs (Pinilla et al., 2012). This can be achieved via local
pressure maxima induced by substructure in discs, e.g. rings and gaps (see
Andrews, 2020).

Despite this apparently self-consistent picture, Rosotti et al. (2019)
suggested that even the most recent observations of the millimetre dust
continuum in discs are still limited by sensitivity. In contrast to Birnstiel
& Andrews (2014), they argue that currently measured disc radii do not trace
the sharp outer edge in the dust distribution, but rather the radius out to
which the grains are large enough to possess significant (sub-)millimetre
opacity. The dust surface density extends further, but in these regions the
grains are small and the opacity significantly lower (factor $\sim$10). If
this interpretation is correct, then observations of the outer regions of
discs in the (sub-)millimetre should reveal faint emission, given sufficient
sensitivity.

Figure 1: Left: Our fiducial frank fit compared to the real component of the
visibilities Re(V) shown in 20 k$\lambda$ bins. The lower panels show the
residuals between fit and data ($\Delta$) in terms of the uncertainty on the
weighted mean $\sigma$, and the imaginary component Im(V). Right: The radial
intensity profile derived from our fiducial frank fit (solid blue) shown
alongside the intensity profiles of the deprojected tclean image (orange) and
the fiducial frank fit convolved with the tclean beam (dashed blue). Shaded
regions give the standard deviation for each bin. The newly identified dark
gap (D82) and bright ring (B91) are marked.

Such observations require optimal targets for which the surface brightness
sensitivity can be maximised. TW Hya is a young star of spectral type M0.5,
stellar mass $\sim$0.6 M⊙ and age $\sim$8 Myr (Sokal et al., 2018). It is
surrounded by a well studied protoplanetary disc, due to its close proximity
(59.5 pc, Bailer-Jones et al. 2018) and near face-on inclination (5°, Huang et
al. 2018). The TW Hya disc has been characterised across multiple wavelengths,
from scattered light (Debes et al., 2017; van Boekel et al., 2017) to both
thermal dust and molecular line emission (Andrews et al., 2016; Teague et al.,
2017; Huang et al., 2018; Tsukagoshi et al., 2019; Nomura et al., 2021; Macías
et al., 2021). Each of these investigations has revealed a diverse array of
substructure in both the gas and dust within the disc.

In this Letter, we present deep observations of the TW Hya disc at 1 mm
obtained with ALMA (19 $\mu$Jy beam-1, 0.37″). We perform our analysis in the
visibility plane to increase the effective signal-to-noise (S/N), and
characterise the dust continuum emission and substructure in the outer
($\gtrsim$60 au) disc for the first time.

##  2 Observations

TW Hya was observed by ALMA in Band 7 on the 3rd of December 2016 for an on-
source time of 5.7 hours in configuration C40-4 under project code
2016.1.00464.S (P.I. C. Walsh). Baselines ranged from 15–704 m with 39–46
antennas depending on execution, with precipitable water vapour measurements
of 1.0 mm. Quasar J1037$-$2934 was used as both a phase and flux calibrator,
while J1058$+$0133 was used as a bandpass calibrator. The correlator was
configured for a continuum spectral window from 292.8–294.8 GHz with a
spectral resolution of 32.0 km s-1. Data (self-)calibration and imaging were
performed with CASA v5.6.2 (McMullin et al., 2007). A continuum visibility
measurement set was created by combining all channels after flagging those
with line emission. Two rounds of phase self-calibration and one round of
amplitude self-calibration were undertaken, improving the peak S/N by a factor
of 14. The phase centre was set to $11^{\rm h}01^{\rm m}51\aas@@fstack{s}811$,
$-34\degr 42\arcmin 17\aas@@fstack{\prime\prime}267$ throughout. Initial
continuum imaging was performed with tclean using Briggs weighting (robust =
0.5) and multi-scale synthesis (nterms = 1) with scales of 0, 0.36, 1.8, 4.2
and 6.0 arcsec, respectively. The synthesised beam was $0.39\times 0.36$
arcsec ($23\times 21$ au) with a position angle of $-50.1$°. The final
continuum rms was 18.8 $\mu$Jy beam-1 (0.11 mJy arcsec-2) measured from a
signal-free region of the image. We highlight that, in terms of a surface
brightness sensitivity per standardised area, our rms is a factor of 160–210
times lower than recent high resolution observations of TW Hya at similar
frequencies (e.g. 18.3 mJy arcsec-2 in Huang et al. 2018 and 22.8 mJy arcsec-2
in Macías et al. 2021).

Figure 2: Left: Zoom-in of the real component of the visibilities at the
longest baselines in our observations, compared with our fiducial frank fit
(solid blue) and extrapolations to longer baselines as discussed in Section
3.1. Right: The corresponding intensity profiles based on these
extrapolations.

##  3 Analysis

Despite the depth of our observations, the limited spatial resolution of our
data (0.37″, 22 au) prevents a detailed investigation in the image plane. We
therefore fit the visibilities directly with the frankenstein code (Jennings
et al. 2020, v1.1.0, hereafter frank). Such analysis can achieve ‘super’
resolution (factors of $\sim$3 better than traditional imaging techniques,
Jennings et al. 2022) and also acts to improve S/N for axisymmetric structure
(see also Walsh et al., 2014; Zhang et al., 2015; Ilee et al., 2020).

Briefly, frank reconstructs the radial intensity at a set of locations,
$r_{i}$, by using a Discrete Hankel Transform to relate the intensity at these
positions, $I(r_{i})$, to the visibilities. The intensity is then inferred
directly from the observed visibilities using a non-parametric Gaussian
Process prior to regularise the fit. The frank code accounts for the
inclination of the source by assuming the emission comes from an optically
thick and geometrically thin disk, for which the inclination ($i$) and
position angle ($\rm{P.A.}$) can be taken account via

$\displaystyle u=$ $\displaystyle\frac{u^{\prime}}{\cos i}\cos({\rm
P.A}.)+v^{\prime}\sin({\rm P.A.}),$ (1) $\displaystyle v=$
$\displaystyle-\frac{u^{\prime}}{\cos i}\sin({\rm P.A}.)+v^{\prime}\cos({\rm
P.A.}),$ (2) $\displaystyle V=$ $\displaystyle\,V^{\prime}{\cos i},$ (3)

where $V$ is the observed visibility, $u$ and $v$ the corresponding
$uv$-points, and primed quantities refer to the de-projected coordinates. We
assume the inclination and position angle that have been already well
determined from high-resolution observations by Huang et al. (2018), namely
$i=5\degr$ and ${\rm P.A.}=152\degr$, respectively.

Figure 1 (left) shows the comparison between the visibilities and our fiducial
frank fit, where we adopt the fit parameters $\alpha=1.2$, $w_{\rm
smooth}=10^{-3}$, $r_{\rm out}=5.0$″ and $p_{0}=10^{-15}$ Jy2 using 300 radial
points. The fit shows good agreement with maximum residuals of less than
3$\sigma$ across all baselines. Figure 1 (right) shows the derived intensity
profile from this model, along with the intensity profile of the deprojected
tclean image and the frank intensity profile convolved to the same beam,
demonstrating agreement at all radii.

The frank intensity profile exhibits two features not seen in previous
millimetre observations of TW Hya. Firstly, the emission extends to $\sim$100
au, well beyond the typical radial extent of $\sim$60-70 au reported
previously (e.g. Andrews et al., 2016; Huang et al., 2018; Macías et al.,
2021). Secondly, we detect new substructure in this outer region of the disc,
notably a dark gap centred at 82 au (hereafter D82) and a bright ring at 91 au
(hereafter B91). We note that the extremely sharp drop of intensity at 100 au
is a result of the log-scaling of the plot and the intensity profile
approaching zero.

###  3.1 Robustness of the derived intensity profile

Determining a robust estimate of the uncertainty on the radial intensity
profile reconstructed from the visibilities is challenging because intensity
reconstruction is formally an ill-posed problem, with the intensity depending
on the visibilities at unobserved $uv$-spacings. It is therefore necessary to
regularise the fit, which frank achieves via an informative prior that damps
power on unobserved small scales. This means that an uncertainty estimated
directly from the posterior (here $\sim$0.1mJy arcsec-2) is a lower bound. Due
to the high S/N of the data at the largest observed baselines, the uncertainty
due to missing baselines dominates over this estimate. The high S/N also
prevents us from investigating the uncertainty of the fit by varying the
hyperparameters $\alpha$ and $w_{\rm smooth}$ as it is insensitive to these
choices.

We therefore explore how the frank profile changes when estimating the
contribution from the missing long baseline data via several sensible
extrapolations beyond the longest observed baselines in our data. In Figure 2
(left), we show the impact of decaying sinusoidal oscillations with amplitudes
of $\gamma=1.5,3$ and 6 mJy and a wavelength that approximates the variation
in the visibilities between 500–700 k$\lambda$ (also shifted by half a
wavelength, $+k$). We also damp the oscillations on very long baselines by a
Gaussian with width 1 M$\lambda$ to prevent small-scale artefacts in the
intensity profile.

Figure 2 (right) shows the corresponding intensity profiles for these
extrapolations, demonstrating how the properties of the disc substructure
change. Our frank fits to extrapolations with smaller amplitudes (e.g.
fiducial, $\gamma=1.5$ mJy and $3.0$ mJy) all recover gap and ring structures
with slightly different locations and depths, but a similar goodness-of-fit
(calculated across the 400–700 k$\lambda$ range, which are those sensitive to
linear size scales of 30 au and less). The gap and ring structure is mostly
removed for a frank fit to the $\gamma=6.0$ mJy extrapolation, but this is a
mildly worse reproduction of the visibilities compared to the fiducial fit
(where $\Delta\chi^{2}=12$, corresponding to $2.2\sigma$). Therefore, while
the extrapolations demonstrate that the precise morphology of the substructure
depends on the behaviour of the visibilities beyond the largest baselines, we
have been unable to remove the substructure entirely and remain consistent
with the observed visibilities. In the absence of a formal uncertainty, we
propagate a representative uncertainty based on the $\gamma=3.0$mJy
extrapolation in all subsequent analysis.

We also considered other extrapolations, such as a constant visibility
amplitude (a point source) and varying the oscillation wavelength by a factor
2. Although not shown, they produce very similar levels of variation as
discussed above. In addition, we tested frank intensity profiles truncated at
80 au, which produced residual images with $\gtrsim 10\sigma$ artefacts out to
$\sim$100 au, further demonstrating the increased radial extent of the disc in
our observations.

##  4 Discussion

###  4.1 The intensity profile compared to previous observations

Huang et al. (2018) present high resolution (0.13″) observations of TW Hya at
290 GHz, providing a useful comparison. They recover the previously identified
continuum gaps D25, D41 and D47 au, with their emission dropping below
detectable levels between 60–70 au. Figure 3 (top) shows a comparison between
our fiducial frank intensity profile and the Huang et al. (2018) observations
calculated with GoFish (Teague, 2019) assuming the geometry discussed above.
The profiles show excellent agreement both in terms of absolute flux level,
slope, and substructure. Even with the limited spatial resolution of our
observations (a CLEAN beam size of $\sim$22 au) we recover the D25 gap,
however the D41 and D47 gaps are likely too shallow and narrow to be visible
in our data. We highlight the respective rms of our observations, which is a
factor $\sim$100 lower, allowing us to detect the millimetre continuum
emission beyond 60–70 au.

We can also compare our millimetre continuum intensity profile with
observations at other wavelengths, particularly those in scattered light.
Debes et al. (2017) present observations of TW Hya with HST/STIS in the R band
(1.6µm) tracing emission out to 180 au, noting the presence of a wide gap at a
radius of 1.43″ (85 au). Similarly, van Boekel et al. (2017) observed TW Hya
with VLT/SPHERE at H band (1.6µm) detecting a gap centred at 1.52″ (90.5 au).
Figure 3 (bottom) shows the normalised radial profiles of these data scaled by
$R^{2}$ to account for geometric dilution of the illuminating starlight. The
location of our newly-identified millimetre gap (D82) is coincident with the
location of these scattered light gaps to within several au. Variations
between the precise gap locations are not unexpected, given that the
millimetre continuum and scattered light observations are tracing different
sized grains and also due to different physical mechanisms (thermal emission
and scattered light, respectively).

We can also examine how our derived intensity profile would appear on the sky.
Figure 4a shows the profile swept in azimuth and reprojected to the observed
position angle and inclination of TW Hya. Figure 4b shows the corresponding
290 GHz continuum image from Huang et al. (2018) overlaid with the location of
the D82 gap and profile edge at 100 au, which lie beyond the radial extent
probed by these high resolution observations.

Given the extended radial range across which our newly identified substructure
lies, we can compare it to observations of molecular line emission at
similarly large radii. Figure 4c shows the brightness temperature of the 12CO
$J=3$–2 emission toward TW Hya as presented in Huang et al. (2018) and Teague
et al. (2019). The intensity scale has been cropped to highlight the ‘break’
in emission between 80–100 au, the beginning of which is coincident with the
continuum gap D82. It is also interesting to note that the D82 gap shares a
similar radial location to the gap in CS $J=5$–4 emission at $\sim$1.6″ (95
au) presented in Teague et al. (2017). In addition, Nomura et al. (2021)
recently presented high resolution maps of 13CO $J=3$–2, C18O $J=3$–2, CS
$J=7$–6 and CN $N=3$–2 toward TW Hya, all of which show breaks in emission
between 80–90 au that are coincident with D82. Taken together, these
observations are consistent with a scenario in which there is a large gas
surface density depletion close to these radii, and that slight offsets can be
attributed to imaging limitations or the fact that each molecular line is
probing a slightly different region of the depletion. Therefore, it appears
that our millimetre continuum intensity profile shares a similar structure to
both the gas and micron-sized dust in the outer disc.

Figure 3: Top: Radial profile from our fiducial frank fit compared with that
from the 290 GHz continuum image from Huang et al. (2018) (H2018). Image plane
rms values are shown with horizontal dashed lines. Bottom: Radial profiles
from scattered light observations with VLT/SPHERE (H-band) and _HST_ /STIS
(R-band) as presented in van Boekel et al. (2017) and Debes et al. (2017),
respectively. Vertical dotted lines mark the location of the previously-
identified millimetre continuum gaps (D). The D82 feature coincides with the
outer gap seen in scattered light observations. Figure 4: (a): The fiducial
frank intensity profile swept in azimuth and reprojected to the observed
inclination and position angle of TW Hya. Overlaid on all panels are ellipses
showing the radial location of the gap at 82 au (dotted) and ’edge’ of the
disc at 100 au (dashed). (b): 290 GHz continuum image of TW Hya at a
resolution of 0.13″(Huang et al., 2018). (c): Brightness temperature of the
12CO $J=3$–2 emission toward TW Hya (Teague et al., 2019), clipped between
20–70 K to highlight emission in the outer disc. The gap at 82 au coincides
with a ‘break’ in 12CO emission. (d): Imaged residuals from our fiducial frank
fit, where contours of $\pm$3, $5\sigma$ are shown ($\sigma=48$ $\mu$Jy
beam-1). A white cross marks the position of the localised continuum excess
identified by Tsukagoshi et al. (2019) and a green arc traces the position of
spiral structure identified in 12CO emission by Teague et al. (2019). Finally,
a blue dotted line shows the shape of a spiral launched by a hypothetical
planet (location marked with $\oplus$) as modelled by Sturm et al. (2020).

###  4.2 The extent of the millimetre continuum emission

Our detection of millimetre emission in the outer disc has several
implications. Firstly, our observations appear to support the hypothesis of
Rosotti et al. (2019), in that we confirm that deep observations reveal faint
continuum emitting material beyond what has been previously reported as the
outer edge of millimetre dust in the TW Hya disc. However, our observations
are unable to differentiate whether this emission originates from grains of a
different size that would support the idea of an ‘opacity cliff’, or whether
the faint millimetre emission reflects a change in surface density. To
investigate this, similarly deep observations at multiple wavelengths would be
required to calculate a spectral index in these outer disc regions.

Secondly, disc sizes are commonly calculated with respect to some radius
enclosing a percentage of total flux, e.g. the 68 per cent radius ($R_{68}$).
Our frank radial intensity profile has $R_{68}=44.2$ au, very close to
previous measurements for TW Hya at similar frequencies (e.g. $45.8\pm 0.6$
au; Tripathi et al., 2017). However, this is clearly not an accurate
representation of the ‘true’ continuum disc based on our results. In order to
capture the larger extent enclosing this faint outer emission, much higher
percentage flux radii are required (e.g. we find $R_{95}=64.8$ au, and
$R_{99}=88.1$ au). If this faint millimetre emission at large radii is a
common feature amongst many other protoplanetary discs beyond TW Hya, then
this would effectively steepen correlations between disc size and other
quantities (such as mass and luminosity) based on observations with lower
sensitivity (e.g. Andrews et al., 2018; Hendler et al., 2020).

###  4.3 Characteristics of the outermost gap at 82 au

The potential detection of the millimetre counterpart of the outermost gap in
TW Hya raises questions as to its origin, in particular whether such a gap
could be opened by a planet in the disc. Following the procedure of Rosotti et
al. (2016), we measure a gap width of $11.7\pm 3.2$ au at 82 au, which
translates to a planet mass of $10.4\pm 4.8$ M⊕. Previous studies have
performed a similar analysis on the gap observed in scattered light. Debes et
al. (2013) placed an upper limit of the mass of the planetary companion of
between 6–28 M⊕ based on _HST_ observations, and van Boekel et al. (2017)
found their observed VLT/SPHERE gap depth could be reproduced by a 34 M⊕ mass
planet. Taken as a range of possibilities, these values are in agreement with
our derived planet mass based on the millimetre continuum. Taking a slightly
different approach, Mentiplay et al. (2019) used hydrodynamic models coupled
with radiative transfer to simulate multiple observations of TW Hya. They
found that planets with masses between 32–95 M⊕ were able to reproduce the
outermost gap seen in scattered light, but note that such planets excite a
spiral arm in the disc that should be apparent in scattered light observations
(but is as yet undetected). These masses are somewhat higher than our estimate
from D82, but are within a factor of $\sim$3\. The addition of the extra
constraints provided by our intensity profile between 60–100 au may help
refine this planet mass for future studies.

###  4.4 Investigating non-axisymmetric structure

Despite the axisymmetric nature of our intensity profile, we can investigate
any potential non-axisymmetric features in the observations by imaging the
residual visibilities of the fiducial frank fit. We adopted the same imaging
parameters as outlined in Section 2, but explored various robust parameters,
finding that $R=2$ provided the best image fidelity (with beam size
$0.46\times 0.41$ arcsec and position angle $-71.0\degr$). The resulting
residual image is shown in Figure 4d. The strongest residual feature is a
localised excess at a position of $\Delta{\rm RA}=-0.74\arcsec$,
$\Delta\rm{Dec}=0.38\arcsec$, corresponding to a radial distance of $50\pm 4$
au ($\mathrm{P.A.}=242.5\degr$) from the central star. It is detected at
approximately $8\sigma$ with an integrated flux density of $0.46\pm 0.16$ mJy.
A blue cross marks the position of the small scale continuum excess first
identified in Tsukagoshi et al. (2019), coincident with the peak of our excess
within 0.05″(3 au, close to the astrometric precision of ALMA). Estimating the
dust mass of the continuum excess assuming optically thin emission and
adopting the same assumptions as Tsukagoshi et al. (2019) (but extrapolated to
our observing frequency) gives a value of $(3.1\pm 1.1)\times 10^{-2}$ M⊕, in
good agreement with their dust mass of $(2.83\pm 0.06)\times 10^{-2}$ M⊕. This
suggests our deep observations are tracing the same feature within the disc,
which is impressive given the size of our nominal beam (0.44″, 26 au) compared
to the resolved size of the excess (2–3 au).

In addition to the localised excess, there are several arc-like structures in
the residuals between radii of 40–120 au detected with a S/N of between 2–6.
We can compare their morphology to non-axisymmetric structure that has been
previously identified in the TW Hya disc. Teague et al. (2019) identified
spiral structure in the intensity of 12CO $J=3$–2 emission in the disc,
overlaid Figure 4d as a green line, which appears coincident with our
continuum residuals in this region. Sturm et al. (2020) further analysed these
residuals, finding that a hydrodynamic model with planet at a radius of 1.65″
(100 au) would generate a spiral wake consistent with their observed
morphology. We have overlaid this spiral morphology on Figure 4d with a blue
dotted line, demonstrating remarkable agreement with the continuum residual
arcs. It therefore appears that our residual map is detecting the millimetre
continuum counterpart to the spiral structure previously identified in 12CO
molecular line emission.

##  5 Conclusions

We have analysed deep millimetre observations of TW Hya using the visibility
modelling code frank. We summarise our findings below:

  * •

We detect faint, but structured, millimetre continuum emission out to radius
of approximately 100 au. While our 68 per cent flux radius remains relatively
unchanged with respect to previous estimates, our 95 and 98 per cent flux
radii are 64.8 and 88.1 au, respectively, suggesting that these metrics may be
more appropriate measures of the true sizes of discs at millimetre
wavelengths.

  * •

Our findings support the hypothesis of Rosotti et al. (2019) in that previous
millimetre continuum observations of discs are limited by sensitivity. If
ubiquitous, this has implications for the many scaling relations derived from
the continuum sizes of protoplanetary discs.

  * •

Our derived intensity profile shows tentative evidence of a gap in the
millimetre dust disc at 82 au (D82) and a bright ring at 91 au (B91), although
their exact properties are dependent on the behaviour of the visibilities at
unobserved baselines. The gap at 82 au is coincident with several other
features observed in both the gas and dust of the TW Hya disc, including a gap
in scattered light and several molecular lines.

  * •

If a planet were to be responsible for opening such a gap, then it would be a
sub-Neptune with mass $10.4\pm 4.8$ M⊕, in broad agreement with similar
analyses of multiple scattered light observations, and forward hydrodynamic
modelling.

  * •

Examining our fit residuals, we confirm the detection of the localised au-
scale millimetre continuum excess at 52 au in the TW Hya disc first reported
by Tsukagoshi et al. (2019) and derive a similar dust mass of $(3.1\pm
1.1)\times 10^{-2}$ M⊕. We also find tentative hints of non-axisymmetric
spiral structure, which may be the continuum counterpart to similar structures
observed in 12CO emission.

The nature of the radial extension of the millimetre continuum could be
further probed by sensitive multi-frequency observations to shed light on the
properties of the dust emitting in the outer disc. In addition, complementary
analysis techniques in both the visibility and image plane may help to further
refine the properties of substructure in these regions. Nevertheless, our
results demonstrate the utility of deep millimetre observations at
intermediate spatial resolution which, when combined with super-resolution
analysis techniques, can _i_) recover emission from the faint outer regions of
protoplanetary discs, and _ii_) characterise any disc substructure. It is
likely that this technique can be extended to study the outer regions of many
other discs that exhibit substructure in scattered light observations (e.g.
Avenhaus et al., 2018) beyond what would normally considered the edge of the
millimetre dust disc.

## Acknowledgements

We are grateful to Jane Huang, Roy van Boekel and John Debes for making their
data available. We thank the referee, Simon Casassus, for a thorough report.
JDI acknowledges support from the STFC under ST/R000287/1. CW acknowledges
support from the University of Leeds, STFC and UKRI (grant numbers
ST/T000287/1, MR/T040726/1). RAB is supported by a Royal Society University
Research Fellowship. GPR acknowledges support from the Netherlands
Organisation for Scientific Research (NWO, program number 016.Veni.192.233)
and from an STFC Ernest Rutherford Fellowship (grant number ST/T003855/1).
This paper makes use of the following ALMA data: ADS/JAO.ALMA#2016.1.00464.S.

## Data Availability

The data underlying this article will be shared on reasonable request.

## References

  * Andrews (2020) Andrews S. M., 2020, ARA&A, 58, 483
  * Andrews et al. (2012) Andrews S. M., et al., 2012, ApJ, 744, 162
  * Andrews et al. (2016) Andrews S. M., et al., 2016, ApJ, 820, L40
  * Andrews et al. (2018) Andrews S. M., Terrell M., Tripathi A., Ansdell M., Williams J. P., Wilner D. J., 2018, ApJ, 865, 157
  * Avenhaus et al. (2018) Avenhaus H., et al., 2018, ApJ, 863, 44
  * Bailer-Jones et al. (2018) Bailer-Jones C. A. L., Rybizki J., Fouesneau M., Mantelet G., Andrae R., 2018, AJ, 156, 58
  * Birnstiel & Andrews (2014) Birnstiel T., Andrews S. M., 2014, ApJ, 780, 153
  * Birnstiel et al. (2010) Birnstiel T., et al., 2010, A&A, 516, L14
  * Birnstiel et al. (2012) Birnstiel T., Klahr H., Ercolano B., 2012, A&A, 539, A148
  * Debes et al. (2013) Debes J. H., Jang-Condell H., Weinberger A. J., Roberge A., Schneider G., 2013, ApJ, 771, 45
  * Debes et al. (2017) Debes J. H., et al., 2017, ApJ, 835, 205
  * Dutrey et al. (1998) Dutrey A., Guilloteau S., Prato L., Simon M., Duvert G., Schuster K., Menard F., 1998, A&A, 338, L63
  * Hendler et al. (2020) Hendler N., Pascucci I., Pinilla P., Tazzari M., Carpenter J., Malhotra R., Testi L., 2020, ApJ, 895, 126
  * Huang et al. (2018) Huang J., et al., 2018, ApJ, 852, 122
  * Hughes et al. (2008) Hughes A. M., Wilner D. J., Qi C., Hogerheijde M. R., 2008, ApJ, 678, 1119
  * Ilee et al. (2020) Ilee J. D., Hall C., Walsh C., Jiménez-Serra I., Pinte C., Terry J., Bourke T. L., Hoare M., 2020, MNRAS, 498, 5116
  * Jennings et al. (2020) Jennings J., Booth R. A., Tazzari M., Rosotti G. P., Clarke C. J., 2020, MNRAS, 495, 3209
  * Jennings et al. (2022) Jennings J., Booth R. A., Tazzari M., Clarke C. J., Rosotti G. P., 2022, MNRAS, 509, 2780
  * Macías et al. (2021) Macías E., Guerra-Alvarado O., Carrasco-González C., Ribas Á., Espaillat C. C., Huang J., Andrews S. M., 2021, A&A, 648, A33
  * McMullin et al. (2007) McMullin J. P., Waters B., Schiebel D., Young W., Golap K., 2007, in Shaw R. A., Hill F., Bell D. J., eds, Astronomical Society of the Pacific Conference Series Vol. 376, Astronomical Data Analysis Software and Systems XVI. p. 127
  * Mentiplay et al. (2019) Mentiplay D., Price D. J., Pinte C., 2019, MNRAS, 484, L130
  * Nomura et al. (2021) Nomura H., et al., 2021, ApJ, 914, 113
  * Panić et al. (2009) Panić O., Hogerheijde M. R., Wilner D., Qi C., 2009, A&A, 501, 269
  * Pérez et al. (2012) Pérez L. M., et al., 2012, ApJ, 760, L17
  * Pinilla et al. (2012) Pinilla P., Birnstiel T., Ricci L., Dullemond C. P., Uribe A. L., Testi L., Natta A., 2012, A&A, 538, A114
  * Ricci et al. (2012) Ricci L., Trotta F., Testi L., Natta A., Isella A., Wilner D. J., 2012, A&A, 540, A6
  * Rodmann et al. (2006) Rodmann J., Henning T., Chandler C. J., Mundy L. G., Wilner D. J., 2006, A&A, 446, 211
  * Rosotti et al. (2016) Rosotti G. P., Juhasz A., Booth R. A., Clarke C. J., 2016, MNRAS, 459, 2790
  * Rosotti et al. (2019) Rosotti G. P., Tazzari M., Booth R. A., Testi L., Lodato G., Clarke C., 2019, MNRAS, 486, 4829
  * Sokal et al. (2018) Sokal K. R., Deen C. P., Mace G. N., Lee J.-J., Oh H., Kim H., Kidder B. T., Jaffe D. T., 2018, ApJ, 853, 120
  * Sturm et al. (2020) Sturm J. A., Rosotti G. P., Dominik C., 2020, A&A, 643, A92
  * Tazzari et al. (2016) Tazzari M., et al., 2016, A&A, 588, A53
  * Teague (2019) Teague R., 2019, Journal of Open Source Software, 4, 1632
  * Teague et al. (2017) Teague R., et al., 2017, ApJ, 835, 228
  * Teague et al. (2019) Teague R., Bae J., Huang J., Bergin E. A., 2019, ApJ, 884, L56
  * Tripathi et al. (2017) Tripathi A., Andrews S. M., Birnstiel T., Wilner D. J., 2017, ApJ, 845, 44
  * Tsukagoshi et al. (2019) Tsukagoshi T., et al., 2019, ApJ, 878, L8
  * Walsh et al. (2014) Walsh C., et al., 2014, ApJ, 791, L6
  * Zhang et al. (2015) Zhang K., Blake G. A., Bergin E. A., 2015, ApJ, 806, L7
  * de Gregorio-Monsalvo et al. (2013) de Gregorio-Monsalvo I., et al., 2013, A&A, 557, A133
  * van Boekel et al. (2017) van Boekel R., et al., 2017, ApJ, 837, 132


# Scene Adaptive Sparse Transformer for Event-based Object Detection

Yansong Peng1,* Hebei Li1,* Yueyi Zhang1,† Xiaoyan Sun1,2 Feng Wu1,2
1University of Science and Technology of China  2Institute of Artificial
Intelligence, Hefei Comprehensive National Science Center
{pengyansong<EMAIL_ADDRESS>{zhyuey, sunxiaoyan,
<EMAIL_ADDRESS>
###### Abstract

While recent Transformer-based approaches have shown impressive performances
on event-based object detection tasks, their high computational costs still
diminish the low power consumption advantage of event cameras. Image-based
works attempt to reduce these costs by introducing sparse Transformers.
However, they display inadequate sparsity and adaptability when applied to
event-based object detection, since these approaches cannot balance the fine
granularity of token-level sparsification and the efficiency of window-based
Transformers, leading to reduced performance and efficiency. Furthermore, they
lack scene-specific sparsity optimization, resulting in information loss and a
lower recall rate. To overcome these limitations, we propose the Scene
Adaptive Sparse Transformer (SAST). SAST enables window-token co-
sparsification, significantly enhancing fault tolerance and reducing
computational overhead. Leveraging the innovative scoring and selection
modules, along with the Masked Sparse Window Self-Attention, SAST showcases
remarkable scene-aware adaptability: It focuses only on important objects and
dynamically optimizes sparsity level according to scene complexity,
maintaining a remarkable balance between performance and computational cost.
The evaluation results show that SAST outperforms all other dense and sparse
networks in both performance and efficiency on two large-scale event-based
object detection datasets (1Mpx and Gen1).

Code: https://github.com/Peterande/SAST.

††footnotetext: *Joint first author †Corresponding author Figure 1: Detection
performance vs. computational cost on 1Mpx, with marker size indicating model
size. SAST exhibits superiority by employing window-token co-sparsification
and scene-specific sparsity optimization, maintaining low computation while
delivering high performance across varying scenes.

##  1 Introduction

Event cameras asynchronously capture the illumination changes of each pixel in
a bionic way and possess several advantages, such as high temporal resolution
(>10K fps) and a wide dynamic range (>120 dB) [12]. Unlike frame cameras that
capture the whole scene at a fixed rate, event cameras exclusively record
sparse event streams when objects are in motion, resulting in no events
generated in static scenes. This distinctive trait enables event cameras to
operate with low power consumption (<10 mW) [12].

Building on these unique advantages of event cameras, event-based object
detection excels in challenging motion and lighting conditions, and offers
energy-efficient solutions in power-constrained environments [31, 63, 20, 43,
28, 41]. However, as the raw events are asynchronous, traditional Image-based
networks can not be directly applied to them. To overcome this problem, prior
works often convert events into Image-like representations, such as event
voxel [67], event histogram [39], and time surface [26, 51] first. Features
are extracted from these representations using different neural network
architectures, including CNN-based [14, 10, 13, 33], SNN-based [59, 6, 34, 27,
9], and GNN-based networks [30, 49]. Recently, many innovative works have
demonstrated that Vision Transformers can achieve superior performance on
event-based object detection tasks [15, 42, 16]. However, the quadratic
computational complexity of self-attention in Transformers hinders model
scalability [52], which challenges the balance between performance and
efficiency for event-based object detection. The high power consumption of
such dense operation also diminishes the low power consumption advantage of
event cameras. Additionally, we observe that due to the high sparsity of
events, a majority of the computational cost of the Transformer is
unnecessary. Especially for high-resolution event streams, such as those in
the 1Mpx [43] dataset, while higher resolution brings more detail, event
sparsity can fluctuate by several hundred times. During self-attention
computations, the blank regions without any events are also encoded as tokens
and participate in calculations. Our goal is to reduce this unnecessary
computational overhead.

We revisit the mainstream Image-based sparse Transformers, which are also
designed to reduce the computational overhead of self-attention. Most of them
achieve token-level sparsification based on global self-attention [35, 61, 24,
3, 2, 22, 40, 45]. Some studies among them achieve adaptive sparsification
[35, 61], but they are unable to choose the optimal sparsity for individual
samples. What’s more, despite computing much fewer tokens than the original
ViT [8], they still require significantly more computational resources than
window-based Transformers [36, 37, 65, 54] that leverage compute-efficient
window self-attention. Consequently, the computational cost remains a barrier
for high-resolution object detection tasks. SparseViT (SViT) [4] proposes a
sparse window-based Transformer architecture. However, it relies on a manually
selected window pruning ratio, which may discard important windows containing
objects. This results in poor robustness in certain scenarios, particularly
when many objects smaller than the window scale are present.

In this work, we introduce Scene Adaptive Sparse Transformer (SAST) for event-
based object detection. SAST achieves window-token co-sparsification in a
window-based Transformer architecture. As shown in Fig. 1, SAST greatly
reduces computational overhead (A-FLOPs: Attention-related FLOPs, exclusive of
the computations incurred by convolutional layers.), while simultaneously
enhancing performance. Moreover, it leverages the scoring and selection
modules, realizing scene-specific sparsity optimization which can adaptively
choose the optimal sparsification strategy based on the complexity of
different scenes. We also propose the Masked Sparse Window Self-Attention (MS-
WSA), which efficiently performs self-attention on selected tokens with
unequal window sizes and isolates all context leakage to achieve optimal
performance. Our main contributions are summarized in the following.

  * •

We develop a highly efficient and powerful SAST for event-based object
detection, which maintains a remarkable balance between performance and
efficiency.

  * •

We propose innovative scoring and selection modules, which assess the
importance of each window and token and perform co-sparsification on them.

  * •

We devise the MS-WSA, which efficiently performs window self-attention on
selected tokens with unequal window sizes and avoids context leakage.

  * •

Experimental results on 1Mpx and Gen1 datasets demonstrate the superiority of
SAST, surpassing all dense and sparse networks.

##  2 Related Work

Vision Transformers. The Vision Transformer (ViT) [8] stands as a seminal work
providing a vision backbone that applies self-attention to images. Based on
that, many ViT variants have emerged to enhance performance and efficiency.
For example, linear Transformers [60, 56, 5, 44] are explored for finding the
approximation of self-attention. Other works [54, 36, 62, 17, 29] introduce
localized self-attention or hierarchical Transformer architectures.

Sparse Transformers. Sparse Transformers are proposed to improve the
efficiency of ViTs by selectively computing self-attention on partial tokens.
Most of them achieve token-level sparsification based on traditional ViT [35,
61, 24, 3, 2, 22, 40, 45]. However, these sparsified networks still require
significantly more computational resources than window-based Transformers [54,
36]. SViT [4] proposes a window-level sparse Transformer based on Swin
Transformer [36]. However, its manually selected window pruning ratio leads to
the unwanted discarding of important windows. If the scene is dense or the
objects are smaller than the window scale, performance will suffer severely.

Event-Based Transformers. Transformer networks have achieved high performance
across various event-based tasks, including but not limited to classification
[48], object detection [15, 42], and semantic segmentation [16]. However, due
to its high computational complexity, many early attempts only incorporate
self-attention operations on feature maps derived from CNN backbones [23, 58,
53, 66]. Recently, alternative approaches [64, 57, 48, 15, 42] using efficient
window Transformers aim to extract features from event representations
directly [67, 39, 26, 51]. However, a substantial computational burden of
self-attention still exists for high-resolution tasks [16, 15], which limits
the model’s scalability and performance [52].

Figure 2:  (a) The hierarchical architecture of SAST. Four SAST blocks extract
multi-scale features from sparse tokens transformed from events. (b) The
architecture of SAST block, which contains two successive SAST layers. (c) The
architecture of SAST layer. In an SAST layer, tokens are partitioned into
windows and scored by the scoring module first. The selection module selects
important windows and tokens. Then, the selected tokens within selected
windows are sequentially processed through MS-WSA, a sparse MLP layer, and an
optional CB operation. Finally, processed tokens are scattered back and
reversed from windows. Norm layers are omitted for simplification.

Event-Based Object Detection The early efforts in event-based object detection
involve converting event streams into images and feeding them into existing
Image-based detectors [25, 1]. Significant strides were made with the advent
of benchmark large-scale datasets 1Mpx [43] and Gen1 [7]. They paved the way
for the development of innovative networks such as RED [43] and ASTMNet [28],
which introduce memory mechanisms to fully exploit the spatiotemporal
information within event data. Some approaches based on SNNs and GNNs were
also proposed [6, 49], but their performance still lags behind CNN-based
methods. Recently, remarkable performance has been achieved by RVT [15], HMNet
[16], and GET [42] using Transformer networks. However, these networks still
involve significant redundancy in self-attention computations of sparse
events.

##  3 Method

###  3.1 Main Architecture

The overall architecture of SAST is shown in Fig. 2(a). The asynchronous
events are initially converted into event voxel representations [67]. These
event voxels are fed into four hierarchical SAST blocks illustrated in Fig.
2(b) to extract multi-scale features. Within the first block, a convolutional
layer, as in ViT [8], transforms the event voxel into tokens. Subsequently,
the sinusoidal positional encoding is added to the tokens. Two successive SAST
layers extract spatial features from these tokens. At the end of the SAST
block, an LSTM layer propagates temporal information, with its output sent to
subsequent layers. In the following blocks, tokens undergo a convolutional
layer first to reduce the spatial resolution, the remaining process being the
same as in the first block. The extracted features from the second, third, and
fourth blocks are relayed to the Feature Pyramid Network (FPN). The FPN then
forwards the processed features to the detection head, yielding the final
detection results.

Figure 3: The architecture of scoring and selection modules. (a) The scoring
module scores each window and token. The scoring process is regulated based on
event sparsity, with windows and tokens competing to limit selections. (b) The
selection module uses two filters to select important windows and tokens
sequentially based on their scores.

###  3.2 SAST Layer

As shown in Fig. 2(c), in an SAST layer, all the tokens are partitioned into
windows first. After that, a scoring module scores the partitioned tokens,
determining their importance. The selection module selects important windows
and tokens inside windows sequentially based on these scores. The token
selection results are shared between SAST layers within the same block. Next,
the Masked Sparse Window Self-Attention (MS-WSA) is performed on selected
tokens within selected windows. The attention-enhanced tokens are sequentially
processed by a sparse MLP layer and an optional context broadcasting (CB)
operation [19]. The sparse MLP layer indicates that sparse connections are
established exclusively on selected tokens, thereby benefiting from
sparsification and reducing computational load. Finally, the processed tokens
are scattered back and reversed from windows to their original shape.

The SAST layer achieves sparsification on both windows and tokens, ensuring
efficiency and preventing important windows containing objects from being
discarded. Its scene-aware adaptability also guides the network to focus more
on important areas, resulting in better performance. What’s more, the
computational cost of SAST is fully dynamic across different scenes, while
maintaining an overall low level. The SAST layer is compatible with different
Transformer architectures, which means it can be plugged as a universal layer
in any hierarchical Transformer. We then detail the three main components of
the SAST layer: the scoring module, the selection module, and MS-WSA.

###  3.3 Scoring Module

As illustrated in Fig. 3(a), the scoring module is designed to score each
token, determining its importance. Unlike other sparse Transformers obtaining
scores directly from the token value or attention map, SAST has a controllable
and learned scoring module for reasonable scoring.

Initially, the partitioned tokens $T$ are sent to a response unit to obtain
their token responses $R$. The response unit consists of a linear layer
followed by a ReLU layer. On a parallel branch, due to multi-scale design, the
original event voxel is downsampled by a pooling layer first to match the
receptive field of tokens. Event sparsity r is obtained by calculating and
concatenating $B$ non-zero ratios $(r_{1},r_{2},...,r_{B})$ of different voxel
bins. Each ratio reflects the sparsity of a specific subset of events with
unique time ranges and polarities. Then, the event sparsity r is projected to
the same dimension as the tokens through the Exponential Linear layer (ExpL),
resulting in the control factor $F$. Based on the definition of the token
response $R$ and the control factor $F$, the initial score $S_{i}$ is defined
as:

$S_{i}=a\cdot\frac{R}{F}=a\cdot\frac{\text{ReLU}(W_{R}\cdot
T+b_{R})}{\exp(W_{F})\cdot\textbf{r}},$ (1)

where $W_{R}$ and $b_{R}$ represent the weight and bias of the linear layer in
the response unit, while $W_{F}$ represents the weight of the ExpL. The
function $\exp(\cdot)$ denotes the exponential operation, which is involved to
ensure that the control factor $F$ is positive. $a$ is a hyper-parameter
controlling the absolute sparsity level of SAST. According to our observation,
tokens corresponding to more important objects tend to have higher initial
scores.

To make the response unit and the ExpL learnable, we apply Spatial-Temporal-
Polar (STP) weighting to the partitioned tokens $T$. A Sigmoid layer
transforms the token response $R$ and the control factor $F$ into spatial
weight $W_{s}$ and temporal-polar weight $W_{tp}$. The weighted tokens $T\ast$
are produced through the product of these weights, as described by the
equation:

$T\ast=W_{tp}\cdot W_{s}\cdot T=\text{Sigmoid}(R)\cdot\text{Sigmoid}(F)\cdot
T.$ (2)

The STP weighting process allows the model to emphasize tokens based on their
spatial and temporal-polar context, aligning the network’s sparse processing
with the most salient features across both domains.

###  3.4 Selection Module

As illustrated in Fig. 3(b), the selection module is designed to select
important tokens and windows. However, the initial scores derived from
normalized tokens show insufficient contrast for effective differentiation.
Therefore, the competition unit is designed to intensify the competition among
the $S_{i}$. It first calculates the $p$-norm of the $S_{i}$, obtaining the
normalized score for each token and window, respectively. The choice of $p$
determines the exponential relationship between the normalized score and event
sparsity. Softmax operation is further applied to amplify the disparities
between normalized scores, making the dominance of some values more
pronounced. The intensified scores are calculated using the equations:

$\begin{split}&S_{t}=\text{Softmax}(||S_{i}||_{p}^{\text{c}})\\\
&S_{w}=\text{Softmax}(||S_{i}||_{p}^{\text{c, w}}/N_{t})\end{split}$ (3)

where $S_{t}$ and $S_{w}$ are the token scores and window scores,
respectively. $||.||_{p}^{\text{c}}$ denotes computing $p$-norm along the
channel dimension, and $||.||_{p}^{\text{c,w}}$ denotes computing $p$-norm
along both the channel and window dimensions. The division by $N_{t}$ serves
to scale the normalized window scores by the number of tokens in a window.

If the scene is sparse, fewer scores dominate, making it easier to select less
important windows and tokens. Conversely, in dense scenes, the score
distribution is relatively uniform, preserving more windows and tokens to
prevent the loss of important object information. Then, the selection module
specifies two thresholds $\mu_{t}$ and $\mu_{w}$ for the selection process,
defined as:

$\mu_{t}=\frac{b}{N_{t}},\quad\mu_{w}=\frac{b}{N_{w}},$ (4)

where $N_{t}$ is the number of tokens in a window, and $N_{w}$ is the number
of windows. $b$ is a parameter in the range [0.9,1] that controls the
strictness of selection. Therefore, $\mu_{t}$ and $\mu_{w}$ correspond to
values slightly smaller than the mean of token and window scores. Scores
exceeding the threshold reflect their significance, ensuring that the selected
proportion aligns with scene complexity.

In Fig. 3(b), weighted tokens $T\ast$ undergo an initial selection by a window
selector that retains tokens from windows with scores $S_{w}$ exceeding
$\mu_{w}$. Then, a token selector further refines this selection, keeping
tokens with scores $S_{t}$ surpass $\mu_{t}$. The final selected tokens are
represented as $T_{s}$.

Figure 4: Masked Sparse Window Self-Attention (MS-WSA). An illustration of MS-
WSA with window size equal to $2\times 2$. Three and two tokens are selected
from two windows. Due to the column masking, the padded token marked as a grey
dot does not contribute to the attention map.

###  3.5 Masked Sparse Window Self-Attention

Traditional WSA is designed for equal-sized windows, conducting parallel
matrix multiplication on them. As a result, it cannot be applied to selected
tokens with unequal window sizes. Therefore, we design MS-WSA to efficiently
perform self-attention on selected tokens, integrating correlations among them
and isolating context leakages.

As shown in Fig. 4, MS-WSA first pads the selected tokens $T_{s}$ within each
window to the same length. The padding operation involves selecting a minimal
number of tokens from the filtered-out tokens. The padded tokens are
represented as $T_{p}$. Afterward, we apply multi-head self-attention (MHSA)
[55] in parallel to selected windows containing padded tokens. However, the
tokens used for padding, originally unselected, also contribute to the
attention map calculation. Moreover, the padding number is determined by the
largest window in the batch. These uncertainties lead to contextual leakage
both among tokens and across different batches. To isolate these context
leakages, MS-WSA includes an additional masking operation when calculating the
attention map. The overall MS-WSA process can be described as:

$\begin{split}&T_{p}=\text{Pad}(T_{s})\\\
&Q,K,V=T_{p}W^{Q},T_{p}W^{K},T_{p}W^{V}\\\
&T_{A}=\text{UnPad}(\text{Softmax}(Mask+QK^{T})V),\end{split}$ (5)

where $\text{Pad}(\cdot)$ indicates the padding operation. $W^{Q}$, $W^{K}$,
and $W^{V}$ are linear weights to transform padded tokens into query $Q$, key
$K$, and value $V$. The $Mask$ matches the attention map $A=QK^{T}$ in size
and contains large negative values in the columns corresponding to the tokens
used for padding, while other values are set to zero. The masking operation
cuts off the influence of unselected tokens in the Softmax process. After a
matrix multiplication operation between the post Softmax attention map and
$V$, the tokens at padded locations are removed by the $\text{UnPad}(\cdot)$
operation, resulting in attention-enhanced tokens $T_{A}$.

After performing MS-WSA, the attention-enhanced tokens $T_{A}$ sequentially
undergo a sparse MLP layer and an optional context broadcasting (CB) operation
[19]. The CB operation can be expressed as:

$\text{CB}(\widetilde{T}(n))=\frac{1}{2}\widetilde{T}(n)+\frac{1}{2N}\sum_{n=1}^{N}\widetilde{T}(n)\
\text{for}\ n=1,\dots,N\textbf{},$ (6)

where $\widetilde{T}(n)$ represents the $n_{th}$ output token of the sparse
MLP layer. $N$ is the total number of selected tokens. The CB operation
broadcasts information among selected tokens, enriching their information
density to ensure that they will be consistently selected in subsequent
layers. It does not incur additional computation but leads to a looser
sparsification preference for SAST. Finally, the processed tokens are
scattered back to the original partitioned tokens $T$ and reversed from
windows to their original shape.

##  4 Experiments

1Mpx Gen1 Methods Backbone mAP (%) FLOPs (G) mAP (%) FLOPs (G) Params (M) SAM
[31] ResNet50 [18] 23.9 19.0 35.5 6.0 >20 YOLOv3_DVS [20] Darknet-53 [46] 34.6
34.8 31.2 11.1 >60 RED [43] ResNet50 [18] 43.0 19.0 40.0 6.0 24.1 ASTMNet [28]
VGG16 [50] 48.3 75.7 46.7 29.3 >100 AEC [41] CSP-Darknet-53 [21] 48.4 58.2
47.0 20.9 46.5 AEC [41] Deformable-DETR [68] 45.9 >50 44.5 >20 40.0 GET [42]
GET [42] 48.4 10.6 (6.3) 47.9 3.6 (2.2) 21.9 RVT [15] Swin Transformer[36]
46.7 10.4 (6.6) 44.4 3.6 (2.3) 18.5 RVT [15] ConvNeXt [38] 45.5 10.4 (6.6)
42.3 3.6 (2.3) 18.7 RVT [15] (baseline) MaxViT [54] 47.4 10.3 (6.5) 47.2 3.5
(2.2) 18.5 Ours SAST 48.3 (+0.9) 5.6 (1.8, -72$\%$) 47.9 (+0.7) 2.1 (0.8,
-64$\%$) 18.9 Ours SAST-CB 48.7 (+1.3) 6.4 (2.6, -60$\%$) 48.2 (+1.0) 2.4
(1.1, -50$\%$) 18.9

Table 1:  Detection performance compared with state-of-the-art methods on 1Mpx
and Gen1. The reported FLOPs belongs to the backbone. Values in brackets
($\cdot$) indicate the A-FLOPs, excluding operations from convolutional
layers, and is precisely the value we aim to reduce.

Initially, we outline the experimental setup, detailing the datasets used,
evaluation metrics, and various implementation specifics. Next, we compare
SAST with other state-of-the-art works on two large-scale event-based object
detection datasets. We also compare SAST with baseline variants and other
sparsification methods. Finally, we offer insights through ablation studies,
visualizations, and statistical results to thoroughly examine the
effectiveness of our proposed sparsification approach.

###  4.1 Experimental Setup

Datasets. The 1Mpx dataset, commonly used for event-based object detection,
consists of 14.65 hours of events, with a large resolution of $1280\times 720$
pixels. It includes 7 labeled object classes. We follow previous works [28,
43, 15], utilizing 3 classes: car, pedestrian, and two-wheeler for performance
comparison. This dataset has a labeling frequency of 60 Hz and contains over
25M bounding boxes. The Gen1 dataset comprises 39 hours of events. It has a
smaller resolution of $304\times 240$ pixels and contains 2 object classes.
The labeling frequency for this dataset is 20 Hz.

Metrics. We use the COCO mean average precision (mAP) [32] as the main metric.
To measure computational complexity, we compute the average FLOPs (Floating
Point Operations Per Second) over the first 1000 samples in the test sets of
1Mpx and Gen1, referring to Sparse DETR [47] that calculates the FLOPs for the
first 100 samples in the MS COCO dataset. Additionally, we report the average
Attention-related FLOPs (A-FLOPs), which excludes the computations incurred by
convolutional layers during calculation. The model’s size is measured through
its parameter count. We also compare the inference time (runtime) with
baseline variants and different sparsification methods.

Implementation Details. We adopt the predefined train, validation, and test
splits of 1Mpx and Gen1 datasets. The accumulation time of each sample is
chosen as 50 ms. We also follow the dataset preprocessing methods of previous
works [28, 15, 43, 42], such as removing misleading small bounding boxes and
downsampling the events in the 1Mpx dataset into $640\times 360$ for a fair
comparison. We choose RVT-B (RVT) [15] as the baseline and apply different
sparsification methods to its MaxViT backbone, while maintaining consistency
in the design of other layers (FPN, Head, and LSTM). The partition strategy is
the same as RVT, dividing tokens into windows and grids (a window type) in two
successive SAST layers. The augmentation strategies include zoom-in, zoom-out,
and horizontal flipping. We utilize eight NVIDIA TITAN Xp GPUs for training.
For the testing phase, we only use one NVIDIA TITAN Xp GPU.

###  4.2 Experimental Results

Comparison with SOTAs. We evaluate SAST on the 1Mpx and Gen1 datasets. The
results compared with state-of-the-art works are reported in Tab. 1. To
facilitate distinction, we define the model as SAST-CB when the optional CB
operation is utilized.

On the 1Mpx dataset, our methods outperform all Transformer-based networks,
including AEC (Deformable-DETR as the backbone), GET [42], and RVT [15] using
different backbones [36, 54, 38]. SAST achieves a $48.3\%$ mAP with only
$28\%$ of the A-FLOPs of RVT. However, in comparison to all works involving
pure convolutional networks, SAST falls slightly below AEC (CSP-Darknet-53 as
the backbone), which utilizes a more complex event representation, introduces
additional data augmentations, and has 9.4 times more FLOPs. SAST-CB achieves
a superior result of $48.7\%$ mAP, surpassing all other state-of-the-art
networks with merely $11\%$ of AEC’s FLOPs and $40\%$ of the A-FLOPs compared
to RVT.

On the Gen1 dataset, both SAST and SAST-CB are superior to all other methods,
achieving $47.9\%$ and $48.2\%$ mAP, which is $0.7\%$ and $1.0\%$ higher than
RVT. Moreover, the FLOPs and A-FLOPs of SAST are only $60\%$ and $36\%$ of
RVT, respectively. In comparison to the pure convolutional network AEC (CSP-
Darknet-53 as the backbone), SAST and SAST-CB achieve $0.9\%$ and $1.2\%$
performance gain with just $10\%$ and $11\%$ of the FLOPs.

Methods mAP (%) A-FLOPs (G) runtime∗ (ms) RVT-T [15] 41.5 1.8 14.5 RVT-S [15]
44.1 3.8 15.3 RVT [15] (baseline) 47.4 6.5 16.0 AViT [61] 44.5 23.4 37.0 SViT
[4] 45.3 3.3 14.3 SparseTT [11] 47.6 6.5 16.1 SAST (Ours) 48.3 1.8 19.7

Table 2: Detection performance on 1Mpx using RVT variants and different
sparsification methods. SAST achieves optimal performance with the least
computational expense. $\ast$: All the runtime is tested on one NVIDIA TITAN
Xp GPU. Figure 5:  The computational complexity of different networks under
scenes with varying event sparsity in 1Mpx. Each point represents an event
sample. SAST adaptively adjusts its sparsity based on the scene complexity.

Comparison with RVT variants & sparse Transformers. As shown in Tab. 2, we
further compare SAST with RVT variants and sparse Transformers on the 1Mpx
dataset.

RVT variants reduce A-FLOPs significantly by scaling down the model size, at
the cost of substantial performance decline. AViT [35] implements token-level
sparsification based on ViT. However, the computational complexity of its
global self-attention after sparsification remains high. Its A-FLOPs exceeds
$260\%$ of the baseline, with a performance decrease of $2.9\%$. SViT [4],
which builds upon window-based Swin-Transformer [36], achieves window-level
sparsification at a manually set window pruning ratio (set to $50\%$ as in the
paper), lacking adaptability. Although its A-FLOPs is only $50.8\%$ of RVT’s,
due to the removal of a fixed number of windows in all scenarios, its
performance dropped by $2.1\%$. SparseTT [11] achieves a $0.2\%$ performance
gain by sparsifying the attention map. However, such mask-based sparsification
does not actually reduce computational costs. Our proposed SAST, thanks to its
outstanding scene-adaptive sparsification, achieves the highest mAP of
$48.3\%$, with only $7.7\%$ A-FLOPs of AViT and $54.5\%$ A-FLOPs of SViT.

In Fig. 5, we illustrate the computational complexity (A-FLOPs) for the first
1000 samples in the 1Mpx dataset. It can be observed that both RVT and SViT
maintain constant A-FLOPs regardless of changes in the scene. AViT shows some
level of adaptability, which remains evenly distributed around its average
A-FLOPs. Our proposed SAST shows remarkable scene-aware adaptability. It has
very low A-FLOPs in sparse scenes while dynamically optimizing the sparsity
level to maintain high performance in dense scenes.

###  4.3 Ablation Studies

To analyze the proposed sparse Transformer SAST, we conduct a series of
ablative experiments on the 1Mpx dataset.

Scoring Methods Source mAP (%) Params (M) L2 Activation SparseViT [4] 45.1
18.6 Attention Mask SparseTT [11] 45.8 18.6 Head Scores AS-ViT [35] 46.8 18.7
Head Importance SPViT [24] 46.5 18.9 Scoring Module SAST (Ours) 48.3 18.9

Table 3:  Detection performance on 1Mpx by using different scoring methods.
The scoring module of SAST surpasses other established scoring methods in
sparse Transformers.

Selection Target mAP (%) A-FLOPs (G) runtime (ms) Window 46.3 3.7 20.1 Token
47.7 3.0 21.7 Window & Token 48.3 1.8 19.7

Table 4:  Detection performance on 1Mpx across different selection targets.
Selection on both windows and tokens forces the network to focus on the most
salient and crucial features, resulting in improved performance.

Methods Context Leakage mAP (%) A-FLOPs (G) SA Yes (batch) 40.7 16.5 S-SA Yes
(token) 44.2 20.5 MS-SA No Leakage 46.6 21.2 WSA N/A N/A N/A S-WSA Yes (token)
46.2 2.0 MS-WSA No Leakage 48.3 1.8

Table 5: Detection performance on 1Mpx using different self-attention methods.
MS-WSA efficiently performs WSA on selected tokens with unequal window sizes,
optimizing performance by preventing context leakage between batches and
tokens. Figure 6: Visualizations of original events, score heatmaps, and
selection results under four scenes in 1Mpx. As the network progresses through
subsequent SAST blocks, featuring multiple downsampling stages, the scale
(receptive field) of tokens expands.  Figure 7:  The averaged token retaining
ratio of SAST across five scenes containing different objects in 1Mpx. Each
$\triangle$ corresponds to an event sample. The evident fluctuations and
clusters demonstrate the model’s scene-aware adaptability.

Scoring Method. We interchanged our scoring module with various established
methods from existing sparse Transformers while maintaining consistency in
other architectural designs. The scoring module of SAST, as highlighted by the
superior mAP of $48.3\%$ in Tab. 3, outperforms other scoring methods.
Leveraging STP weighting across both spatial and temporal-polar domains, SAST
provides a more effective and context-aware approach for evaluating token
importance.

Selection Method. We compare different selection methods by training two SAST
variants that only select windows and tokens, respectively. As shown in Tab.
4, applying selection to both windows and tokens achieves the highest mAP of
$48.3\%$ and the lowest A-FLOPs of $1.8G$. This strategy limits the amount of
information the model can use, which forces the model to concentrate on the
most salient and crucial features. Consequently, it can lead to a more
efficient representation where the model learns to compress token information
more densely.

Self-Attention Method. We compare different self-attention methods by applying
them to selected tokens with varying window sizes. It can be seen from Tab. 5
that standard self-attention (SA) needs to process all tokens together,
leading to context leakage across different batches and adversely affecting
performance. Sparse Self-Attention (S-SA) employs padding to isolate tokens
from different batches, but still results in context leakage between selected
and padded tokens. Masked Sparse Self-Attention (MS-SA) further includes
masking operation, which prevents both types of context leakage. However, as
global self-attention, it still has high A-FLOPs. Window Self-Attention (WSA)
can not be applied to windows of unequal size; after padding, Sparse Window
Self-Attention (S-WSA) becomes usable but introduces context leakage between
tokens, resulting in performance degradation. Only by applying MS-WSA, which
isolates all context leakage and is fully parallel, is it possible to achieve
optimal performance with the least computational complexity.

###  4.4 Adaptability Analysis

Visualizations. We train SAST on the 1Mpx dataset and infer it on four typical
scenes in the test set for visualizations. The event sparsity and scene
complexity increase across the four scenes progressively. In Fig. 6, we
provide visualizations of the original events, score heatmaps, and the
selection results of windows and tokens. For the score heatmaps and selection
results, figures from left to right depict the transition from first to third
SAST blocks. In a later block, the number of tokens decreases, and their scale
(receptive field) enlarges. These intuitive visualizations allow us to assess
the model’s adaptability more comprehensively. From the visualizations of
score heatmaps, we observe that the network demonstrates its scene-aware
adaptability to assign important tokens higher scores. From the visualizations
of window and token selection results, in complex scenes, the network
proactively reduces the sparsity level to retain more important tokens.

Token Retaining Ratio. The scatter plot in Fig. 7 illustrates the relationship
between the token retaining ratio and event sparsity across five scenes in the
1Mpx dataset. The general trend in the figure shows that SAST retains more
tokens as event sparsity increases, demonstrating its ability to optimize the
sparsity level based on the scene complexity. Furthermore, there are
significant fluctuations in token retaining ratios at equivalent event
sparsity. This reveals that SAST’s optimization is not only based on the event
sparsity but also on the distinct characteristics of each scene, such as the
classes, numbers, and sizes of objects within the scene. This is also
evidenced by the pronounced clustering of samples from distinct scenes,
indicating that SAST can perceive the underlying patterns of different scenes,
adaptively tuning its sparsity strategy accordingly. In essence, the scatter
plot demonstrates that SAST can dynamically tailor its sparsity level to the
specific demands of each scene, thereby exhibiting scene-aware adaptability.

##  5 Conclusion

In this paper, we provide a novel vision Transformer for event-based object
detection, called Scene Adaptive Sparse Transformer (SAST). SAST’s adaptive
sparsification mechanism enables window-token co-sparsification, significantly
reducing the computational overhead. Utilizing the novel scoring module,
selection module, and MS-WSA, SAST showcases scene-aware adaptability,
dynamically optimizing its sparsity across different scenes for high
performance. Our results confirm the effectiveness of SAST, achieving state-
of-the-art mAP on the 1Mpx and Gen1 datasets while maintaining remarkable
computational efficiency.

##  6 Acknowledgement

This work was in part supported by the National Natural Science Foundation of
China (NSFC) under grants 62032006 and 62021001.

## References

  * Cannici et al. [2019] Marco Cannici, Marco Ciccone, Andrea Romanoni, and Matteo Matteucci.  Asynchronous convolutional networks for object detection in neuromorphic cameras.  In _CVPRW_ , 2019.
  * Chen et al. [2021] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang.  Chasing sparsity in vision transformers: An end-to-end exploration.  In _NeurIPS_ , 2021.
  * Chen et al. [2023a] Xiang Chen, Hao Li, Mingqiang Li, and Jinshan Pan.  Learning a sparse transformer network for effective image deraining.  In _CVPR_ , 2023a.
  * Chen et al. [2023b] Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han.  Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer.  In _CVPR_ , 2023b.
  * Choromanski et al. [2021] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller.  Rethinking attention with performers.  In _ICLR_ , 2021.
  * Cordone et al. [2022] Loic Cordone, Benoît Miramond, and Phillipe Thierion.  Object detection with spiking neural networks on automotive event data.  In _Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN)_ , 2022.
  * de Tournemire et al. [2020] Pierre de Tournemire, Davide Nitti, Etienne Perot, Davide Migliore, and Amos. Sironi.  A large scale event-based detection dataset for automotive.  _arXiv preprint arXiv:2001.08499_ , 2020.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * Fang et al. [2021] Wei Fang, Zhaofei Yu, Yanqi Chen, Timothée Masquelier, Tiejun Huang, and Yonghong Tian.  Incorporating learnable membrane time constant to enhance learning of spiking neural networks.  In _ICCV_ , 2021.
  * Fischer and Milford [2020] Tobias Fischer and Michael Milford.  Event-based visual place recognition with ensembles of temporal windows.  _IEEE Robotics and Automation Letters_ , 2020.
  * Fu et al. [2022] Zhihong Fu, Zehua Fu, Qingjie Liu, Wenrui Cai, and Yunhong Wang.  Sparsett: Visual tracking with sparse transformers.  In _IJCAI_ , 2022.
  * Gallego et al. [2022] Guillermo Gallego, Tobi Delbrück, and Garrick Orchard, et al.  Event-based vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Gao et al. [2019] Shan Gao, Guangqian Guo, and C. L. Philip Chen.  Event-based incremental broad learning system for object classification.  In _IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)_ , 2019.
  * Gehrig et al. [2019] Daniel Gehrig, Antonio Loquercio, Konstantinos G. Derpanis, and Davide Scaramuzza.  End-to-end learning of representations for asynchronous event-based data.  In _ICCV_ , 2019.
  * Gehrig and Scaramuzza [2023] Mathias Gehrig and Davide Scaramuzza.  Recurrent vision transformers for object detection with event cameras.  In _CVPR_ , 2023.
  * Hamaguchi et al. [2023] Ryuhei Hamaguchi, Yasutaka Furukawa, Masaki Onishi, and Ken Sakurada.  Hierarchical neural memory network for low latency event processing.  In _CVPR_ , 2023.
  * Han et al. [2021] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang.  Transformer in transformer.  _arXiv preprint arXiv: 2103.00112_ , 2021.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2015.
  * Hyeon-Woo et al. [2023] Nam Hyeon-Woo, Kim Yu-Ji, Byeongho Heo, Doonyoon Han, Seong Joon Oh, and Tae-Hyun Oh.  Scratching visual transformer’s back with uniform attention.  In _ICCV_ , 2023.
  * Jiang et al. [2019] Zhuangyi Jiang, Pengfei Xia, Kai Huang, Walter Stechele, Guang Chen, Zhenshan Bing, and Alois Knoll.  Mixed frame-/event-driven fast pedestrian detection.  In _ICRA_ , 2019.
  * Jocher [2021] Glenn Jocher.  ultralytics/yolov5: v6.0 - yolov5n ’nano’ models, roboflow integration, tensorflow export, opencv dnn support.  _Zenodo_ , 2021.
  * Kim et al. [2021] Sehoon Kim, Sheng Shen, David Thorsley, Amir Gholami, Joseph Hassoun, and Kurt Keutzer.  Learned token pruning for transformers.  _SIGKDD_ , 2021.
  * Kim et al. [2023] Taewoo Kim, Yujeong Chae, Hyun-Kurl Jang, and Kuk-Jin Yoon.  Event-based video frame interpolation with cross-modal asymmetric bidirectional motion fields.  In _CVPR_ , 2023.
  * Kong et al. [2022] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al.  Spvit: Enabling faster vision transformers via latency-aware soft token pruning.  In _ECCV_ , 2022.
  * Lagorce et al. [2015] Xavier Lagorce, Cédric Meyer, Sio-Hoi Ieng, David Filliat, and Ryad Benosman.  Asynchronous event-based multikernel algorithm for high-speed visual features tracking.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2015.
  * Lagorce et al. [2017] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E. Shi, and Ryad B. Benosman.  Hots: A hierarchy of event-based time-surfaces for pattern recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Lee et al. [2016] Junhaeng Lee, Tobi Delbrück, and Michael Pfeiffer.  Training deep spiking neural networks using backpropagation.  _Frontiers in Neuroscience_ , 2016.
  * Li et al. [2022] Jianing Li, Jia Li, Lin Zhu, Xijie Xiang, Tiejun Huang, and Yonghong Tian.  Asynchronous spatio-temporal memory network for continuous event-based object detection.  _IEEE Transactions on Image Processing_ , 2022.
  * Li et al. [2021a] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool.  Localvit: Bringing locality to vision transformers.  _arXiv preprint arXiv:2104.05707_ , 2021a.
  * Li et al. [2021b] Yijin Li, Han Zhou, Bangbang Yang, Ye Zhang, Zhaopeng Cui, Hujun Bao, and Guofeng Zhang.  Graph-based asynchronous event processing for rapid object recognition.  In _ICCV_ , 2021b.
  * Liang et al. [2021] Zichen Liang, Guang Chen, Zhijun Li, Peigen Liu, and Alois Knoll.  Event-based object detection with lightweight spatial attention mechanism.  In _The IEEE International Conference on Advanced Robotics and Mechatronics (ICARM)_ , 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2021a] Mengyun Liu, Na Qi, Yunhui Shi, and Baocai Yin.  An attention fusion network for event-based vehicle object detection.  In _ICIP_ , 2021a.
  * Liu et al. [2021b] Qianhui Liu, Dong Xing, Huajin Tang, De Ma, and Gang Pan.  Event-based action recognition using motion information and spiking neural networks.  In _IJCAI_ , 2021b.
  * Liu et al. [2022a] Xiangcheng Liu, Tianyi Wu, and Guodong Guo.  Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention.  In _CVPR_ , 2022a.
  * Liu et al. [2021c] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021c.
  * Liu et al. [2022b] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo.  Swin transformer v2: Scaling up capacity and resolution.  In _CVPR_ , 2022b.
  * Liu et al. [2022c] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  _CVPR_ , 2022c.
  * Moeys et al. [2016] Diederik Paul Moeys, Federico Corradi, Emmett Kerr, Philip Vance, Gautham Das, Daniel Neil, Dermot Kerr, and Tobi Delbrück.  Steering a predator robot using a mixed frame/event-driven convolutional neural network.  In _The International Conference on Event-based Control, Communication, and Signal Processing (EBCCSP)_ , 2016.
  * Pan et al. [2021] Bowen Pan, Yifan Jiang, Rameswar Panda, Zhangyang Wang, Rogério Feris, and Aude Oliva.  Ia-red${}^{\mbox{2}}$: Interpretability-aware redundancy reduction for vision transformers.  In _NeurIPS_ , 2021.
  * Peng et al. [2023a] Yansong Peng, Yueyi Zhang, Peilin Xiao, Xiaoyan Sun, and Feng Wu.  Better and faster: Adaptive event conversion for event-based object detection.  In _AAAI_ , 2023a.
  * Peng et al. [2023b] Yansong Peng, Yueyi Zhang, Zhiwei Xiong, Xiaoyan Sun, and Feng Wu.  Get: Group event transformer for event-based vision.  In _ICCV_ , 2023b.
  * Perot et al. [2020] Etienne Perot, Pierre de Tournemire, Davide Nitti, Jonathan Masci, and Amos Sironi.  Learning to detect objects with a 1 megapixel event camera.  In _NeurIPS_ , 2020.
  * Qin et al. [2022] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong.  cosformer: Rethinking softmax in attention.  In _ICLR_ , 2022.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh.  Dynamicvit: Efficient vision transformers with dynamic token sparsification.  In _NeurIPS_ , 2021.
  * Redmon and Farhadi. [2018] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  In _CVPR_ , 2018.
  * Roh et al. [2022] Byungseok Roh, JaeWoong Shin, Wuhyun Shin, and Saehoon Kim.  Sparse detr: Efficient end-to-end object detection with learnable sparsity.  In _ICLR_ , 2022.
  * Sabater et al. [2022] Alberto Sabater, Luis Montesano, and Ana C. Murillo.  Event transformer. a sparse-aware solution for efficient event data processing.  In _CVPRW_ , 2022.
  * Schaefer et al. [2022] Simon Schaefer, Daniel Gehrig, and Davide Scaramuzza.  Aegnn: Asynchronous event-based graph neural networks.  In _CVPR_ , 2022.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_ , 2015.
  * Sironi et al. [2018] Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, and Ryad B. Benosman.  Hats: Histograms of averaged time surfaces for robust event-based object classification.  _CVPR_ , 2018.
  * Tay et al. [2022] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler.  Efficient transformers: A survey.  _ACM Computing Surveys_ , 2022.
  * Tian and Andrade-Cetto [2022] Yi Tian and J. Andrade-Cetto.  Event transformer flownet for optical flow estimation.  In _British Machine Vision Conference_ , 2022.
  * Tu et al. [2022] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  In _ECCV_ , 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Vyas et al. [2020] Apoorv Vyas, Angelos Katharopoulos, and Franccois Fleuret.  Fast transformers with clustered attention.  In _NeurIPS_ , 2020.
  * Wang et al. [2022] Zuowen Wang, Yuhuang Hu, and Shih-Chii Liu.  Exploiting spatial sparsity for event cameras with visual transformers.  In _ICIP_ , 2022.
  * Weng et al. [2021] Wenming Weng, Yueyi Zhang, and Zhiwei Xiong.  Event-based video reconstruction using transformer.  _ICCV_ , 2021.
  * Wu et al. [2021] Hao Wu, Yueyi Zhang, Wenming Weng, Yongting Zhang, Zhiwei Xiong, Zhengjun Zha, Xiaoyan Sun, and Feng Wu.  Training spiking neural networks with accumulated spiking flow.  In _AAAI_ , 2021.
  * Wu et al. [2022] Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long.  Flowformer: Linearizing transformers with conservation flows.  In _International Conference on Machine Learning (ICML)_ , 2022.
  * Yin et al. [2022] Hongxu Yin, Arash Vahdat, Jose Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-ViT: Adaptive tokens for efficient vision transformer.  In _CVPR_ , 2022.
  * Yuan et al. [2021] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In _ICCV_ , 2021.
  * Zhang et al. [2021] Jiqing Zhang, Xin Yang, Yingkai Fu, Xiaopeng Wei, Baocai Yin, and Bo Dong.  Object tracking by jointly exploiting frame and event domain.  In _ICCV_ , 2021.
  * Zhang et al. [2022a] Jiqing Zhang, Bo Dong, Haiwei Zhang, Jianchuan Ding, Felix Heide, Baocai Yin, and Xin Yang.  Spiking transformers for event-based single object tracking.  In _CVPR_ , 2022a.
  * Zhang et al. [2022b] Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, , Sercan Ö. Arık, and Tomas Pfister.  Nested hierarchical transformer: Towards accurate, data-efficient and interpretable visual understanding.  In _AAAI_ , 2022b.
  * Zhao et al. [2022] Junwei Zhao, Shiliang Zhang, and Tiejun Huang.  Transformer-based domain adaptation for event data classification.  _ICASSP_ , 2022.
  * Zhu et al. [2019] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Unsupervised event-based learning of optical flow, depth and egomotion.  In _CVPRW_ , 2019.
  * Zhu et al. [2021] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  In _ICLR_ , 2021.

Supplementary Material

##  1 Video Detection & Sparsification Results

In the accompanying multimedia file, titled video.mp4, we present
visualizations corresponding to several event clips in the test set of 1Mpx.
This video includes comparisons between ground truth and SAST’s object
detection results, showcasing SAST’s high performance. It also features the
visualizations of score heatmaps and selection results across different
scenes, providing a clear demonstration of SAST’s scene-aware adaptability.
From the video, it can be observed that SAST assigns higher scores to
important tokens within important windows and performs a series of operations
such as self-attention, MLP, and normalization exclusively on these sparse
tokens, significantly reducing computational costs.

##  2 Additional Experiments

###  2.1 Sparsity Level of SAST.

We adjust the hyper-parameters $a$ and $b$ to limit the sparsification of SAST
and SAST-CB, resulting in 10 sparsity levels. The performance of 20 networks
is illustrated in Fig. 1. SAST and SAST-CB respectively excel at sparser and
denser sparsity levels, but both achieve the best results at a moderate
sparsity level. We interpret these results from an information perspective,
where higher information density (more effective interactions among fewer
tokens) has been shown to benefit the Transformers [19]. SAST-CB achieves this
by broadcasting information among selected tokens, enhancing effective
information interactions within a reduced token set. However, an overly sparse
network can lead to information loss due to excessive compression. Therefore,
the choice between SAST and SAST-CB depends on the specific requirements of
the task at hand: SAST for lower computational load and SAST-CB for enhanced
detection at a slightly higher computational cost.

Figure 1:  Setting different hyper-parameters results in different sparsity
levels and performance of SAST and SAST-CB on 1Mpx. Performance does not
continuously improve with increasing sparsity levels. SAST and SAST-CB each
have their advantages in sparser and denser settings.

1Mpx Gen1 Functions mAP (%) A-FLOPs (G) mAP (%) A-FLOPs (G) Identity 46.5 2.3
46.1 1.2 SoftMax 40.9 1.5 42.0 0.7 Tanh 47.0 1.7 46.7 0.8 Sigmoid 48.3 1.8
47.9 0.8

Table 1: Detection performance on 1Mpx and Gen1 using different weighting
functions. Sigmoid achieves the optimal results.

1Mpx Gen1 Methods Backbone mAP (%) FLOPs (G) mAP (%) FLOPs (G) Params (M)
RVT-L [15] MaxViT-L [54] 47.8 23.2 (19.3) 47.6 7.8 (6.6) 33.2 Ours SAST-L 49.2
(+1.4) 7.5 (3.7, -81$\%$) 48.6 (+1.0) 2.9 (1.7, -74$\%$) 33.6

Table 2:  Detection performance on 1Mpx and Gen1 by training larger variants
of RVT and SAST.

###  2.2 Weighting Method Ablation.

In Tab. 1, we conduct a comparative analysis of various functions used in the
STP weighting process for transitive derivatives. The Sigmoid function, which
smoothly maps scores to weights within the range of 0 to 1, assigning higher
weights to more significant tokens, delivers superior performance on both
datasets.

###  2.3 Bigger Model, Bigger Gain.

We scale up the RVT and SAST by increasing the layer count in the first,
second, and fourth blocks by a factor of two and in the third block by a
factor of six, producing the larger RVT-L and SAST-L variants. As depicted in
Tab. 2, training these larger models on the 1Mpx and Gen1 datasets both result
in performance gains. However, for RVT-L, the proportion of A-FLOPs within the
total FLOPs significantly increases, which echoes the discussions of model
scalability in Sec. 1. At a comparable model size, SAST-L exhibits more
pronounced benefits from its adaptive sparsification. It achieves an
impressive mAP of $49.2\%$ on 1Mpx and $48.6\%$ on Gen1 datasets, with even
fewer FLOPs than the pre-scaled RVT-B. This fully demonstrates the potential
of our proposed sparsification method in achieving a remarkable balance
between performance and computational cost for large models.

##  3 Additional Visualizations

We extend our visualization analysis to the 1Mpx and Gen1 datasets, as shown
in Fig. 2 and Fig. 3. These visualizations encompass the original event data,
score heatmaps, and the selection results of the windows and tokens. The
supplementary visualizations reinforce our findings from the main text,
providing further evidence of the network’s scene-aware adaptability in
assigning higher scores to important tokens and adjusting sparsity levels in
response to the scene complexity.

##  4 Additional Implementation Details.

In Tab. 3, we list the default choices of key hyper-parameters, facilitating
the replication of our study. These parameters are shared for both the 1Mpx
and Gen1 datasets.

$a$ $b$ $p$ Batch Size Steps Learning Rate 0.0002 0.099 1.0 32 600000 0.00056

Table 3: Default hyper-parameters used for training SAST and SAST-CB on 1Mpx
and Gen1. Figure 2: Additional Visualizations of original events, score
heatmaps, and selection results under four scenes in Gen1. As the network
progresses through subsequent SAST blocks, featuring multiple downsampling
stages, the scale (receptive field) of tokens expands.  Figure 3: Additional
Visualizations of original events, score heatmaps, and selection results under
four scenes in 1Mpx. As the network progresses through subsequent SAST blocks,
featuring multiple downsampling stages, the scale (receptive field) of tokens
expands.