# Superevents: Towards Native Semantic Segmentation for Event-based Cameras

Weng Fei Low  National University of SingaporeSingapore ,  Ankit Sonthalia
Birla Institute of Technology &1 Thørväld CircleScience, Pilani, India ,  Zhi
Gao  National University of SingaporeSingapore ,  André van Schaik  Western
Sydney UniversityInternational Center for Neuromorphic SystemsNew South
WalesAustralia<EMAIL_ADDRESS>and  Bharath Ramesh  Western
Sydney UniversityInternational Center for Neuromorphic SystemsNew South
WalesAustralia<EMAIL_ADDRESS>
###### Abstract.

Most successful computer vision models transform low-level features, such as
Gabor filter responses, into richer representations of intermediate or mid-
level complexity for downstream visual tasks. These mid-level representations
have not been explored for event cameras, although it is especially relevant
to the visually sparse and often disjoint spatial information in the event
stream. By making use of locally consistent intermediate representations,
termed as superevents, numerous visual tasks ranging from semantic
segmentation, visual tracking, depth estimation shall benefit. In essence,
superevents are perceptually consistent local units that delineate parts of an
object in a scene. Inspired by recent deep learning architectures, we present
a novel method that employs lifetime augmentation for obtaining an event
stream representation that is fed to a fully convolutional network to extract
superevents. Our qualitative and quantitative experimental results on several
sequences of a benchmark dataset highlights the significant potential for
event-based downstream applications.

Neuromorphic vision, neural networks, semantic segmentation, mid-level
features

††copyright: none††ccs: Computing methodologies Structured outputs††ccs:
Computing methodologies Image segmentation

##  1\. Introduction

In the past decade, a novel class of visual sensors, known as Dynamic Vision
Sensors (DVS), has attracted a lot of attention from the academia as well as
the industry. This is fueled by its exceptional performance compared to
conventional image sensors in terms of temporal resolution, latency, dynamic
range and power consumption. Such breakthrough is achieved via a fundamentally
different principle of operation. Conventional image sensors output a
synchronous sequence of intensity frames, whereas the DVS outputs a sequence
of events asynchronously, whereby each event indicates a change in intensity
at a particular pixel location and time instant.

(a)

(b)

(c)

(d)

Figure 1. (a). An event camera is moved on a slider in front of objects at
different depth (Mueggler et al., 2017); (b) Edge map obtained via Canny edge
detection is visualized for reference; (c) Scene-adaptive optical flow
estimation is subsequently employed for lifetime estimation to produce crisp
event stream images; (d) Superevents extracted using a fully convolutional
network on the event stream image (best viewed on monitor).

Due to the above-mentioned unique advantages, event cameras have been trialed
for a wide range of high-level vision tasks, namely object segmentation
(Alonso and Murillo, 2019), motion segmentation (Mitrokhin et al., 2020),
pattern recognition (Ramesh et al., 2019, 2018), gesture recognition (Chen et
al., 2020a; Chen et al., 2020b), space situational awareness (Bagchi and Chin,
2020; Chin et al., 2019), to name a few. Despite these advances in addressing
these problems using conventional or deep learning methods that have made use
of novel neural architecture design and training schemes suited for event-
based vision, previous works have ignored mid-level representation that are
critical in bridging gap between the sparse, disjoint information in the event
stream and downstream visual tasks. As these novel cameras respond to
structure and texture of the scene while not capturing intensity information,
it is challenging to obtain groupings of events, i.e. mid-level
representations, as done in superpixel extraction for image data (Stutz et
al., 2018).

In computer vision, superpixels are perceptually similar pixels grouped
together to effectively reduce the number of image primitives for subsequent
processing. This mid-level representation has been widely adopted for scene
understanding(Wang et al., 2019) via detecting and segmenting salient regions
(Zhao and Wu, 2019; Zhao et al., 2015; Mostajabi et al., 2015), and also for
high-level tasks such as object detection (Li et al., 2018; Shu et al., 2013)
and tracking (Feng et al., 2019; Yeo et al., 2017). However, these ideas are
yet to be explored in the event-based domain for mid-level feature extraction.
One main reason is possibly that, the standard convolution operation in
convolutional neural networks (CNNs) is defined on a regular image grid with
intensity information. While a few attempts have been made to modify deep
architectures (CamuÃ±as-Mesa et al., 2018; Messikommer et al., 2020),
performing convolution over aysnchronous data remains challenging.

In this paper, we introduce a fully convolutional network (FCN) based
superevents extraction algorithm that is in the spirit of (Yang et al., 2020).
The input to the FCN is a Binary Event Image (BEI) obtained via lifetime
augmentation (Mueggler et al., 2015). In this regard, we make use of the open-
source event-based flow estimation method SOFEA (Low et al., 2020). It is
worth noting that the recent works targeting super-resolution of the event
stream(et al., 2020; Wang et al., 2020) is very different to the task
considered in this paper. Super-resolution aims to reconstruct high resolution
representations directly from the event stream and is unrelated to superevents
(or superpixels) mid-level feature extraction.

The primary contributions of this paper are:

  1. (1)

A scene-adaptive noise rejection method for event-based optical flow
estimation (subsection 3.1) and a robust lifetime augmentation method for
accurate BEI estimation (subsection 3.2).

  2. (2)

A scene-adaptive BEI rendering time interval which approximately normalizes
for the expected apparent displacement magnitude of features with apparent
motion (subsection 3.3).

  3. (3)

A superevents extraction pipeline for event cameras leveraging on the above
contributions using a fully convolutional network.

##  2\. Related Work

As previously mentioned, the output of a DVS is a stream of asynchronous
events, as opposed to a set of images captured at a constant rate by a
standard camera. Due to this unconventional output, the general design
strategy of event-based algorithms has been to either implicitly or explicitly
buffer a certain number of past events to perform a given task. The most
straight-forward method is to accumulate events over a fixed time interval.
For instance, (Acharya et al., pted; Ussa et al., pted; Kogler et al., 2009)
used contrived time intervals of $5-66\mathit{ms}$ to estimate binary event
images for object tracking and stereo vision. Nevertheless, the motion-
dependent sensing aspect of the DVS is a major hindrance for choosing the
optimal time interval to prevent motion blur or a low density event image.

Another way to obtain an event stream representation is to implicitly choose a
time interval by accumulating a fixed number of past events (Ramesh et al.,
2019; Gallego et al., 2016; Rebecq et al., 2016). However, the critical issue
is choosing the number of events, which unfortunately depends on the scene
complexity and spatial resolution of the DVS itself. (Liu and Delbrück, 2018)
proposed a method to accumulate past events until the density of events in a
local spatial neighborhood reaches a predefined threshold. It adaptively
selects the number of events to buffer by considering both scene complexity
and spatial resolution. With that said, all the methods above accumulate all
events in a particular time interval without correcting for apparent motion.

A more principled approach is known as the “motion correction”. In (Gallego et
al., 2018; Zhu et al., 2017), uniform optical flow is first assumed for all
events within a small spatio-temporal neighborhood. Using this assumption, an
event image is then estimated by warping all events in the spatio-temporal
neighborhood to a given reference time, according to the set of optical flow
parameters that best describe the apparent motion of events. The optimal set
of parameters is searched using iterative methods, therefore it is more
computationally expensive compared to all the above methods. The uniform
optical flow assumption is replaced with a spatially uniform Bézier curve
motion assumption in (Seok and Lim, 2020). In general, a relatively large time
interval (or a large number of events) is required for accurate estimation of
the relative intensity change magnitude, due to inherent noise and latency of
the DVS.

Due to the limitations of the above event stream representation strategies,
this paper adopts the concept of “lifetime” introduced by (Mueggler et al.,
2015) to represent how long will it take for an event to trigger a new
associated event in its neighboring pixel location, due to persistent apparent
motion. Hence, by activating pixels of the binary event image according to the
estimated lifetime and position of events, a binary event image can be defined
for any time instant in the past. Lifetime estimation itself can be done using
the so-called “Local Plane Fit”’ (Benosman et al., 2014), which is one of the
most widely adopted approaches for optical flow estimation. Normal flow is
estimated by fitting a plane in a local spatio-temporal neighborhood of the
Surface of Active Events (SAE), which describes the timestamp of the most
recent event for each pixel location, about the incoming event. The single-
shot optical flow estimation algorithm (SOFEA) (Low et al., 2020) considers
selected events in the local spatio-temporal neighborhood with the same
polarity, and thus is heavily robust to noise.

Nonetheless, the limitations of SOFEA make it rather non-ideal for direct BEI
estimation. One of the main issues is that SOFEA suffers from low event
density in select scenes, mainly due to refractory filtering, which implies
that a significant amount of events are lost in the process. In other words,
the refractory period prevents updating the SAE if an event has arrived within
the refractory period. This is undesirable as the estimated BEI (similar to an
edge map) is then not representative of the scene. The use of a refractory
filter also limits the maximum speed of scene dynamics. Apart from that, the
validity of the local planar assumption is dependent on the scene dynamics and
the local scene structure. Thus, events associated to high acceleration motion
or moving edges with large curvatures (e.g. a corner) will most likely be
regarded as noise by the goodness-of-fit noise rejection method, as it is
evaluated with respect to a planar model. Thus, we propose a pipeline for BEI
estimation and lifetime augmentation, termed as SOFEA-BEI, to deal with the
shortcomings of vanilla SOFEA.

##  3\. SOFEA-BEI

An event $\boldsymbol{e}$ can be characterized by a 4-tuple $(x,y,t,p)$, where
$x,y$ represents the spatial location or position of the event in pixel
coordinates, $t$ represents the time of occurrence of the event and
$p\in\\{-1,+1\\}$ represents the polarity of the event. On top of that, the
event stream can be defined as a sequence of events
$(\boldsymbol{e}_{n})_{n\in\mathbb{N}}$ such that $i<j\implies t_{i}\leq
t_{j}$, where
$\boldsymbol{e}_{i}=(x_{i},y_{i},t_{i},p_{i}),\boldsymbol{e}_{j}=(x_{j},y_{j},t_{j},p_{j})$.

Disregarding event polarity, each event carries information in the three-
dimensional $xyt$ spacetime domain. Consequently, the sequence of most recent
events over the entire spatial domain, which is a subset of the event stream
$(\boldsymbol{e}_{n})_{n\in\mathbb{N}}$, can be simply described by a spatio-
temporal surface $\Sigma_{e}$. The spatio-temporal surface $\Sigma_{e}$ is
better known as the Surface of Active Events (SAE), as coined in (Benosman et
al., 2014). The mathematical definition of the SAE is given by:

(1) $\begin{split}\Sigma_{e}:\mathbb{R}^{2}&\to\mathbb{R}\\\
\boldsymbol{p}&\mapsto\Sigma_{e}(\boldsymbol{p})=t\end{split}$

where $\boldsymbol{p}=\begin{bmatrix}x&y\end{bmatrix}^{\top}$.

A simple expression for $\Sigma_{e}$ can be obtained by approximating the SAE
as a plane in a local spatial neighborhood. With the approximation above, the
local planar SAE can be sufficiently described by a set of plane parameters
$\boldsymbol{\Pi}=\begin{bmatrix}a&b&c&d\end{bmatrix}^{\top}$. An interesting
property of the SAE is that its spatial gradient
$\nabla\Sigma_{e}(\boldsymbol{p})=\begin{bmatrix}\frac{\partial\Sigma_{e}(\boldsymbol{p})}{\partial
x}&\frac{\partial\Sigma_{e}(\boldsymbol{p})}{\partial y}\end{bmatrix}^{\top}$
encodes information of normal flow associated to the active events. For a
local planar SAE, $\nabla\Sigma_{e}(\boldsymbol{p})$ can be further defined as
$\begin{bmatrix}-\frac{a}{c}&-\frac{b}{c}\end{bmatrix}^{\top}$. SOFEA improved
upon existing local plane fitting methods by using an efficient greedy
algorithm to optimally select the spatially neighboring events associated to
the incoming event for accurate non-iterative local SAE plane fitting. SOFEA
also incorporated a mathematically simpler description of the local planar SAE
given by the additional constraint that the incoming event lies on it.

The refractory filter originally incorporated in SOFEA is removed from SOFEA-
BEI so as to retain a high event density, and also to eliminate its introduced
limit on the speed of scene dynamics. We then post-process the lower accuracy
optical flow estimates to obtain highly satisfactory lifetimes for the purpose
of BEI rendering. In particular, post-processing is performed via extended
lifetime augmentation and lifetime reset. Apart from that, the goodness-of-fit
noise rejection method in SOFEA is replaced with a novel scene-adaptive noise
rejection method. In order to reduce visual redundancy between successive BEI
renders, a motion-normalized BEI rendering time interval is also introduced.
Figure 2 illustrates the overall pipeline of SOFEA-BEI.

Figure 2. Pipeline of SOFEA for Binary Event Image Estimation (SOFEA-BEI).

###  3.1. Scene-adaptive Noise Rejection

The proposed noise rejection method for flow estimation does not depend on the
goodness-of-fit criteria(Aung et al., 2018) for the plane fitting, i.e. check
whether the fitted plane predicts timestamps of associated events within a
tolerance level. Instead, we utilize an estimated upper bound of the true
underlying distribution of optical flow speed, to implicitly assess the
approximate upper bound likelihood of a given optical flow speed estimate.
More precisely, our method rejects events with optical flow speed estimates
that fall in the tails of the approximate probability upper bound, as it is
highly improbable to observe such a value. Such an approach is possible due to
the fact that spurious noise events and erroneous flow estimates generally
have an extreme flow speed associated to it. Hence, this simple noise
rejection method is adaptive to the scene dynamics and independent of the
local scene structure with practical bounds of the true optical flow speed.

###  3.2. Extended Lifetime Augmentation and Reset

As a variant of the Local Plane Fit method, SOFEA estimates the normal flow of
an event by performing plane fitting on its corresponding local SAE. Thus, the
global SAE can be approximated as a piecewise-continuous set of planes, with
each plane defined for a distinct $1\times 1$ pixel spatial neighbourhood.
Theoretically, a BEI with infinite spatial resolution can then be estimated by
evaluating the SAE at a particular time instant. Nevertheless, this BEI
estimation approach requires highly accurate normal flow estimates, as well as
high computational cost for the evaluation of each local time slice. Hence, we
adopt the lifetime augmentation approach (Mueggler et al., 2015), with some
refinements, for relatively accurate and efficient BEI estimation, even with
low accuracy normal flow estimates.

Ideally, the lifetime of an event signifies the amount of time required for an
associated (new) event to be generated in its neighboring pixel location, due
to apparent motion of their common corresponding feature. Suppose lifetimes of
events are known, BEI can be rendered by simply activating specific pixels,
given by the position of events, for a duration of their augmented lifetimes.
However, event lifetimes depend on the continuous-time trajectory and velocity
of the apparent motion, which are not trivial to estimate. Thus, the lifetime
of an event is approximated as the maximum time required for the event to
“travel” $1$ pixel, supposing the apparent motion velocity of the associated
feature is constant, or equivalently the normal flow of the event is constant.
Note that the lifetime is defined as the maximum time because in Local Plane
Fit methods, only normal flow is estimated. As the normal flow speed always
underestimates the optical flow speed, only the maximum time can be estimated.
Mathematically, the lifetime, $\tau$ of an event is given by:

(2)
$\tau=\frac{1}{\|\boldsymbol{v_{\perp}}\|}=\|\nabla\Sigma_{e}(\boldsymbol{p})\|$

In other words, $\tau$ is merely given by the spatial gradient magnitude of
the associated local SAE.

Nevertheless, the scene representation accuracy of the estimated BEI depends
greatly on the accuracy of the estimated lifetimes, which in turn relies on
the normal flow estimation accuracy. Apart from that, the constant velocity
assumption may be violated in scenes with high acceleration or deceleration,
resulting in an overestimation or underestimation of lifetimes. In order to
enhance the quality of BEI estimates in general scenes and also when low
accuracy normal flow estimates are used, we propose to augment each event with
an extended lifetime, $\tau_{\mathit{ext,bei}}$ instead, given by:

(3) $\tau_{\mathit{ext,BEI}}=\kappa_{\mathit{ext,BEI}}\ \tau,\
\kappa_{\mathit{ext,BEI}}\geq 1$

By doing so, it can be observed that edges in the BEI are approximately
$\kappa_{\mathit{ext,bei}}$ pixels thick, due to the overestimation of
lifetime by an approximate factor of $\kappa_{\mathit{ext,BEI}}$. To alleviate
that, we also perform edge thinning by resetting (i.e. set to zero) the
extended lifetime of the neighboring event, which is in the opposite flow
direction of a given event, because the neighboring event generally indicates
the “source” of the given event. As the optimal value of
$\kappa_{\mathit{ext,bei}}$ might be different for accurate scene
representation of the rendered BEI, and for accurate estimation of the scene
complexity, given by $\widetilde{C}(t_{j})$, an event can also be optionally
augmented with a second extended lifetime for scene complexity estimation, as
follows:

(4) $\tau_{\mathit{ext,\widetilde{C}}}=\kappa_{\mathit{ext,\widetilde{C}}}\
\tau,\ \kappa_{\mathit{ext,\widetilde{C}}}\geq 1$

With both lifetime extension and lifetime reset, the eventual effective
lifetime, $\tau_{\mathit{eff,BEI}}$ or $\tau_{\mathit{eff,\widetilde{C}}}$
gives a relatively better estimate of the true lifetime. This is due to the
larger error tolerance of normal flow estimates, given by the closed-loop
nature of the lifetime estimation process. This allows for a crisp and
accurate scene representation of the estimated BEI, which is visually similar
to an edge map. Isolated active pixels in the estimated BEI can also be
optionally filtered out, as noise in a BEI manifests as impulse noise.

###  3.3. Scene-adaptive BEI Rendering

Although subsection 3.2 allows for continuous-time BEI rendering, it is
computationally infeasible and generally unnecessary, as a new BEI rendering
is usually only required whenever it differs significantly from the most
recent BEI rendered. Thus, a BEI rendering time interval is defined to govern
the rendering time instants. Similar to how standard cameras capture intensity
images at a fixed rate, it is rather reasonable to set it as a constant.
Nevertheless, the amount of redundancy increases as the scene dynamics
decreases. Thus, we propose a scene-adaptive BEI rendering time interval that
approximately normalizes for the expected apparent displacement magnitude of
features with apparent motion. In particular, it approximately normalizes for
the expected number of events generated in an event substream.

####  3.3.1. Event Generation Model

In this subsection, we will first devise a simplified model to describe how
the event stream is formed by numerous features with apparent motion to assist
scene-adaptive BEI rendering:

  1. (1)

$c(t)$ is a deterministic function that defines the number of features with
apparent motion (also known as the scene complexity), for all time $t$. A
feature is defined as a unique surface point in the scene, within the field-
of-view of the DVS, that causes events to be generated if it has apparent
motion.

  2. (2)

$\boldsymbol{v}_{i}(t)=\begin{bmatrix}\|\boldsymbol{v}_{i}(t)\|&\angle\boldsymbol{v}_{i}(t)\end{bmatrix}^{\top}$
is a deterministic vector-valued function that defines the optical flow of
events associated to feature $i\in\\{\ 1,2,\ldots,c(t)\ \\}$ with apparent
motion, for all time $t$, assuming infinite spatial resolution of the image
sensor (hence events are generated in continuous-time). In particular,
$\|\boldsymbol{v}_{i}(t)\|$ gives the speed and $\angle\boldsymbol{v}_{i}(t)$
gives the orientation of the optical flow. $\boldsymbol{v}_{i}(t)$ is also
equivalent to the apparent motion velocity of feature $i\in\\{\
1,2,\ldots,c(t)\ \\}$ at time $t$.

  3. (3)

A short time interval, $\mathcal{I}(t)=(\ t-\Delta T(t),t\ ]$ is defined for
all time $t$, where $\Delta T(t)$ is a deterministic function that gives the
length of the time interval.

  4. (4)

$f_{i}(t)$ is a deterministic function that defines the event generation
frequency associated to feature $i\in\\{\ 1,2,\ldots,c(t)\ \\}$ with apparent
motion, for all time $t$.

With the definitions above, the assumptions are:

  1. (1)

$c(t^{\prime})$ and $\boldsymbol{v}_{i}(t^{\prime}),\forall i\in\\{\
1,2,\ldots,c(t^{\prime})\ \\}$ have the same respective values for all time
$t^{\prime}$ in $\mathcal{I}(t)$. In other words, $c(t)$ and
$\boldsymbol{v}_{i}(t),\forall i\in\\{\ 1,2,\ldots,c(t)\ \\}$ are constant
throughout $\mathcal{I}(t)$.

  2. (2)

Given that $\boldsymbol{v}_{i}(t^{\prime})$ has the same value for all time
$t^{\prime}$ in $\mathcal{I}(t)$, $f_{i}(t^{\prime})$ also has the same value.
In other words, if $\boldsymbol{v}_{i}(t)$ is constant throughout
$\mathcal{I}(t)$, it is also the case for $f_{i}(t)$. Implicitly, this assumes
that all pixels have the same contrast threshold, which could be negligibly
violated in real DVS data due to random hardware mismatch. This assumption
also disregards the subtle variation of $f_{i}(t^{\prime})$ if the apparent
motion direction of feature $i$, or equivalently
$\angle\boldsymbol{v}_{i}(t^{\prime})$, is non-Ordinal or non-Cardinal with
respect to the DVS pixel grid.

Assumptions 1 and 2 can be combined to yield the following:
$c(t^{\prime}),\boldsymbol{v}_{i}(t^{\prime})$ and $f_{i}(t^{\prime})$,
$\forall i\in\\{\ 1,2,\ldots,c(t^{\prime})\ \\}$ have the same respective
values for all time $t^{\prime}$ in $\mathcal{I}(t)$. As a result, the event
generation frequency of the event stream, given by:

(5) $f(t^{\prime})=\sum\limits_{i=1}^{c(t^{\prime})}f_{i}(t^{\prime})$

also has the same value for all time $t^{\prime}$ in $\mathcal{I}(t)$. Also,
it can be observed that a direct relationship exists between
$\|\boldsymbol{v}_{i}(t)\|$ and $f_{i}(t)$. In fact, it is given by:

(6) $\|\boldsymbol{v}_{i}(t)\|=\alpha(\angle\boldsymbol{v}_{i}(t))\ f_{i}(t)$

where $\alpha(\theta)$ is a deterministic function that specifies the
conversion from frequency to pixel units depending on the orientation of
optical flow.

###  3.4. Scene-adaptive Rendering

Scene-adaptive motion normalization can be achieved between successive BEI
renders by using the expected number of events generated in an event substream
within a small temporal window. To formally derive the scene-adaptive BEI
rendering time interval, we first present the following definitions:

  1. (1)

A short time interval, $\mathcal{I}_{\mathit{BEI}}(t)=[\ t,\ t+\Delta
T_{\mathit{BEI}}(t)\ )$ is defined for all time $t$, where $\Delta
T_{\mathit{BEI}}(t)$ is a deterministic function that gives the length of the
time interval (the scene-adaptive BEI rendering time interval).

  2. (2)

$\mathbf{s}_{i}(t)$ is a deterministic vector-valued function that defines the
apparent displacement of feature $i\in\\{\ 1,2,\ldots,c(t)\ \\}$ with apparent
motion from the start to the end of the time interval
$\mathcal{I}_{\mathit{BEI}}(t)$ (i.e. between time $t$ and $t+\Delta
T_{\mathit{BEI}}(t)$). Mathematically, $\mathbf{s}_{i}(t)$ is given by:

(7) $\mathbf{s}_{i}(t)=\int_{t}^{t+\Delta
T_{\mathit{BEI}}(t)}\begin{bmatrix}\boldsymbol{v}_{i,x}(t^{\prime})\\\
\boldsymbol{v}_{i,y}(t^{\prime})\end{bmatrix}dt^{\prime}$

  3. (3)

$\\{\ S(t),\ t\in[\ 0,\infty\ )\ \\}$ is a continuous stochastic process,
whereby it is equally likely for $S(t)$ to give any of
$\|\boldsymbol{s}_{i}(t)\|,\ i\in\\{\ 1,2,\ldots,c(t)\ \\}$. In other words,
$S(t)$ describes the distribution of $\|\boldsymbol{s}_{i}(t)\|$. With that,
the expected value of $S(t)$ is given by:

(8)
$\mathbb{E}[S(t)]=\frac{1}{c(t)}\sum\limits_{i=1}^{c(t)}\|\boldsymbol{s}_{i}(t)\|$

  4. (4)

$\\{\ F(t),\ t\in[\ 0,\infty\ )\ \\}$ is a continuous stochastic process,
whereby it is equally likely for $F(t)$ to give any of $f_{i}(t),\ i\in\\{\
1,2,\ldots,c(t)\ \\}$. In other words, $F(t)$ describes the distribution of
$f_{i}(t)$. Thus, the expected value of $F(t)$ is given by (Equation 5):

(9)
$\mathbb{E}[F(t)]=\frac{1}{c(t)}\sum\limits_{i=1}^{c(t)}f_{i}(t)=\frac{f(t)}{c(t)}$

  5. (5)

$\\{\ V(t),\ t\in[\ 0,\infty\ )\ \\}$ is a continuous stochastic process,
whereby it is equally likely for $V(t)$ to give any of
$\|\boldsymbol{v}_{i}(t)\|,\ i\in\\{\ 1,2,\ldots,c(t)\ \\}$. In other words,
$V(t)$ describes the distribution of $\|\boldsymbol{v}_{i}(t)\|$. With that,
the expected value of $V(t)$ is given by (Equation 6):

(10)
$\mathbb{E}[V(t)]=\frac{1}{c(t)}\sum\limits_{i=1}^{c(t)}\|\boldsymbol{v}_{i}(t)\|=\frac{1}{c(t)}\sum\limits_{i=1}^{c(t)}\alpha(\angle\boldsymbol{v}_{i}(t))\
f_{i}(t)$

Because $\alpha(\angle\boldsymbol{v}_{i}(t))\in[\ 1,\sqrt{2}\ ]$, the
following inequality regarding $\mathbb{E}[V(t)]$ and $\mathbb{E}[F(t)]$
holds:

(11) $\mathbb{E}[F(t)]\leq\mathbb{E}[V(t)]\leq\sqrt{2}\ \mathbb{E}[F(t)]$

Along with the above definitions, we impose similar assumptions, as given by
Assumptions 1 and 2 in subsubsection 3.3.1, but on the time interval
$\mathcal{I}_{\mathit{BEI}}(t)$ instead. Thus, Equation 7 can be simplified
to:

(12) $\mathbf{s}_{i}(t)=\begin{bmatrix}\boldsymbol{v}_{i,x}(t^{\prime})\\\
\boldsymbol{v}_{i,y}(t^{\prime})\end{bmatrix}\Delta T_{\mathit{BEI}}(t)$

Besides, Equation 8 can also be simplified to give:

(13)
$\mathbb{E}[S(t)]=\frac{1}{c(t)}\sum\limits_{i=1}^{c(t)}\|\boldsymbol{v}_{i}(t)\|\
\Delta T_{\mathit{BEI}}(t)=\mathbb{E}[V(t)]\ \Delta T_{\mathit{BEI}}(t)$

Substituting Equation 11 into the above equation, the following bounds can be
imposed on $\mathbb{E}[S(t)]$:

(14) $\mathbb{E}[F(t)]\ \Delta
T_{\mathit{BEI}}(t)\leq\mathbb{E}[S(t)]\leq\sqrt{2}\ \mathbb{E}[F(t)]\ \Delta
T_{\mathit{BEI}}(t)$

Note that $\mathbb{E}[F(t)]\ \Delta T_{\mathit{BEI}}(t)$ actually gives the
expected number of events generated in an event substream within
$\mathcal{I}_{\mathit{BEI}}(t)$ (proof omitted). Similar to the case of
$\Delta T(t)$, the choice of $\Delta T(t)_{\mathit{BEI}}$ is arbitrary,
despite being deterministic. Therefore, we suppose that $\Delta
T(t)_{\mathit{BEI}}$ is defined such that it coincides with:

(15) $\Delta
T(t)_{\mathit{BEI}}=\frac{2}{1+\sqrt{2}}\cdot\frac{\overline{s}}{\hat{F}(t)}$

in the first realization of $\hat{F}(t)$, where $\overline{s}$ is a constant
representing the desired value of $\mathbb{E}[S(t)]$. Substituting Equation 15
into Equation 14 gives the following:

(16)
$\frac{2}{1+\sqrt{2}}\cdot\frac{\mathbb{E}[F(t)]}{\hat{F}(t)}\cdot\overline{s}\leq\mathbb{E}[S(t)]\leq\frac{2\sqrt{2}}{1+\sqrt{2}}\cdot\frac{\mathbb{E}[F(t)]}{\hat{F}(t)}\cdot\overline{s}$

Thus, $\Delta T(t)_{\mathit{BEI}}$ is defined in such a way that
$\overline{s}$ is the midpoint of the above bounds on $\mathbb{E}[S(t)]$.
Equivalently, it is also defined such that $\frac{2}{1+\sqrt{2}}\overline{s}$
is the desired expected number of events generated in an event substream
within $\mathcal{I}_{\mathit{BEI}}(t)$ (i.e. suppose the mean squared error of
$\hat{F}(t)$ is kept small relative to $\mathbb{E}[F(t)]$, $\mathbb{E}[F(t)]\
\Delta T_{\mathit{BEI}}(t)\approx\frac{2}{1+\sqrt{2}}\overline{s}$).
Therefore, motion normalization can be achieved between successive BEI renders
by using $\Delta T(t)_{\mathit{BEI}}$ as the rendering interval.

Figure 3. Schematic representation of the FCN used for extracting superevents
from SOFEA-BEI

##  4\. SOFEA-BEI-FCN Superevents

In this section, we introduce our CNN-based superevents segmentation method
using the output of SOFEA-BEI. We make use of the architecture proposed in
(Yang et al., 2020), which has been shown to perform well on RGB compared to
several state-of-the-art methods. Superpixels as defined for standard images
are supposed to delineate the edges, and thus binary images containing edge-
like structures can also be well-segmented by the fully convolutional neural
network architecture proposed in (Yang et al., 2020).

The model in (Yang et al., 2020) divides a given image into a number of grids
with preset dimensions. These grids form the initial superpixels. The model
then makes use of a series of convolutions followed by one of deconvolutions
in order to achieve the desired segmentation, as illustrated in Figure 3.
Further, features from earlier layers are concatenated to those from deeper
layers via skip connections. The final output is an assignment map

(17) $\begin{split}g:\mathbb{Z}^{H\times W}&\to N_{r}\\\
\boldsymbol{r}&\mapsto g(\boldsymbol{r})=n\end{split}$

where $\boldsymbol{r}=(y,x)$ and $n$ denotes the superpixel that each pixel
has been assigned to. Also, $N_{r}$ denotes the nine superpixels in the
neighbourhood of the given pixel. A lot of the simplicity in (Yang et al.,
2020) is the direct result of this part of their formulation: any given pixel
can only be assigned to one of the nine superpixels in its immediate
neighbourhood. This results in compact superpixels and oversegmenation to an
extent that is greater than what is observed in traditional superpixel
segmentation methods. (Yang et al., 2020) justify this by arguing that since
the superpixels thus obtained will eventually be used in downstream tasks, the
latter may group perceptually similar superpixels if such a need arises.
Hence, this formulation does not compromise on the quality of superpixels for
use in downstream tasks, while also helping to decrease the number of training
and inference time computations.

For the sake of better visualization in Figure 3, superevents with little to
no events have been merged. In practice, this was achieved by computing the
ratio of the number of active events to that of the total number of pixels
inside a given superevent. They were merged together whenever this ratio
turned out to be less than a pre-determined threshold (set to 0.05 in this
work).

(a) APS intensity image

(b) Edge map via (Canny, 1986)

(c) $\kappa_{\mathit{ext,BEI}}=1$, no reset

(d) $\kappa_{\mathit{ext,BEI}}=1$, with reset

(e) $\kappa_{\mathit{ext,BEI}}=3$, no reset

(f) $\kappa_{\mathit{ext,BEI}}=3$, with reset

(g) $\kappa_{\mathit{ext,BEI}}=3$, with reset, augmented with normal flow
orientation

(h) $\kappa_{\mathit{ext,BEI}}=5$, with reset

Figure 4. Comparison of BEI estimated using SOFEA-BEI, under various extended
lifetime augmentation and lifetime reset configurations, on the dynamic_6dof
sequence.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 5. BEI scene representation accuracy comparison between SOFEA-BEI (3rd
row) and (Mueggler et al., 2015) (4th row) on the slider_far (1st column),
urban (2nd column) and dynamic_rotation (3rd column) sequences. APS intensity
images (1st row) and edge maps obtained via Canny edge detection (2nd row) are
also visualized.

##  5\. Experiments

We use sequences from the event camera dataset (Mueggler et al., 2017) for
testing SOFEA-BEI and subsequently the superevents extraction using the FCN.
The superevents concept is introduced for the first time using event cameras,
to the best of our knowledge. Thus, evaluation is focused on comparing SOFEA-
BEI-FCN to SOFEA-BEI-SLIC (Achanta et al., 2012) and APS-FCN with
simultaneously recorded intensity frames from the publicly available dataset
(upper baseline). The ground truth segmentation results for SOFEA-BEI are
obtained from the corresponding APS-Canny edge maps for the calculation of
boundary precision and recall (Yang et al., 2020) with the natural assumption
that APS frames have more information for segmentation compared to BEI. Next
we qualitatively evaluate SOFEA-BEI results comapared to vanilla SOFEA.

(a) slider_depth

(b) shapes_translation

(c) dynamic_rotation

Figure 6. Comparison of SOFEA-BEI superevents, using FCN and SLIC methods, to
APS-FCN superpixels.

###  5.1. Extended Lifetime Augmentation & Reset

We carry out a qualitative analysis on the effectiveness of extended lifetime
augmentation and reset in SOFEA-BEI for enhanced BEI estimation accuracy via
the dynamic_6dof sequence (Mueggler et al., 2017). Figure 4 illustrates the
BEI estimated using SOFEA-BEI, under various extended lifetime augmentation
and lifetime reset configurations, on the dynamic_6dof sequence. An APS
intensity image of the scene is shown in 4(a), and its corresponding edge map
generated using Canny edge detection is given in 4(b). Since an edge map of
the APS image gives a good representation of a BEI, we consider 4(b) as the
reference for the BEI estimates. It is worth noting that there is no fixed
“sampling interval” for the whole pipeline, as SOFEA-BEI is an event-based
method adapted to dynamically handle object/camera motion.

4(c) illustrates the BEI estimated using the original lifetime augmentation
approach (Mueggler et al., 2015) (equivalently, without lifetime extension or
$\kappa_{\mathit{ext,BEI}}=1$) without performing lifetime reset, while 4(d)
shows the BEI estimated with lifetime reset done for edge thinning. It can be
observed that edges in both BEI estimates are generally one pixel thick, as
opposed to BEI estimated using the direct accumulation approach (Figure 1).
However, several pixel thick edges still exist in 4(c) due to the
overestimation of lifetime. This issue is almost completely rectified after
lifetime reset is incorporated, which produces a more crisp representation of
the scene.

On the other hand, extended lifetime augmentation is performed in 4(f) to
4(h). Observe that edges in 4(e) are approximately three pixels thick with
$\kappa_{\mathit{ext,BEI}}=3$ and lifetime reset not done. The incorporation
of lifetime reset in 4(f) introduces a substantial improvement in terms of
scene representation accuracy, when compared to 4(d). Note that the amount of
detail captured is now similar to that of the edge map. Nevertheless, the
amount of noise increases as well, due to events with extended lifetimes not
being reset. Fortunately, there is still an apparent increase in the Signal-
to-Noise Ratio (SNR). 4(g) augments 4(f) with normal flow orientation
estimates, given by the color of the active pixels according to the color
wheel displayed. Observe that active pixels associated to the same edge have a
common color, which corresponds to the direction perpendicular to the edge.
This is desirable for an effective lifetime reset. 4(h) illustrates the BEI
estimated with $\kappa_{\mathit{ext,BEI}}=5$ and lifetime reset. Note that the
amount of detail captured is almost the same as that of
$\kappa_{\mathit{ext,BEI}}=3$, despite a significant increase of noise until
the point where feature tracks can be clearly seen. This suggests that
$\kappa_{\mathit{ext,BEI}}=3$ and lifetime reset is sufficient for SOFEA-BEI
to achieve accurate BEI estimation.

To validate SOFEA-BEI, (Mueggler et al., 2015) is employed as the benchmark to
evaluate its performance. In addition, edge maps generated using Canny edge
detection on the APS intensity image also serves as a ground truth reference
for BEI estimates. The qualitative comparison is performed with (Mueggler et
al., 2015) on three sequences from (Mueggler et al., 2017). This collection of
sequences covers a wide range of scenes with varying complexity, dynamic range
and motion profiles. The details captured in a BEI need not be captured in the
edge map, due to limitations of the edge detection, motion blur in the APS
image, and high dynamic range of the scene.

Figure 5 illustrates the comparison on the three mentioned sequences. In
general, it can be observed that SOFEA-BEI estimates achieve remarkable scene
representation, similar to the corresponding edge maps. BEI estimates of
(Mueggler et al., 2015) however are generally sparse in nature, as it fails to
capture the abundance of detail in the scene. This is heavily reflected in
5(d) and 5(h). As noted earlier, there exist details not captured in the edge
map, but reflected in the SOFEA-BEI estimates. For instance, the concentric
rings of the dartboard in the slider_depth sequence, the top left edges of the
building in the urban sequence and the details of objects in the
dynamic_rotation sequence. This is made possible due to the high dynamic range
of the event stream and accurate SOFEA-BEI estimates.

###  5.2. Superevents

Figure 6 illustrates the boundary precision and recall for different
sequences, which measures the boundary adherence of the superevents/pixels
according to the ground truth with a 3$\times$3 tolerance window as done in
earlier works (Yang et al., 2020). The performance of the APS frames using FCN
is much better compared to its BEI counterpart using the event stream, owing
to the richness of the intensity information. This does not imply that the
event stream is lacking information, but rather the way it is used in this
work as a BEI. To investigate whether the lower performance is due to the use
of the pre-trained network from (Yang et al., 2020), the popular hand-crafted
SLIC (Achanta et al., 2012) was used on BEI to extract superevents. This doubt
is ruled out as SOFEA-BEI-SLIC has generally lower performance, especially in
the shapes dataset where the edge maps form a strong ground truth. A short
clip of the superevents can be viewed here111https://tinyurl.com/hv3bkz5h.
This video shows good boundary separation between the superevents at different
depths, especially when a new object closer to the camera enters the field-of-
view.

To the best of our knowledge, we have introduced the first mid-level
perceptual grouping of events using event-based flow estimation that does not
limit the rate at which superevents are extracted. In theory, superevents can
be extracted for every flow estimate using the proposed scheme although the
scene-adaptive BEI is better suited for generating superevents regardless of
scene and camera dynamics. Overall, superevents have several potential
applications, including native semantic segmentation, generating templates for
object tracking, depth estimation, feature extraction, etc., and showcasing
them is beyond the scope of this paper.

##  6\. Conclusion

This paper proposed the first superevents extraction method for event cameras,
leveraging on a robust binary event image generation scheme and fully
convolutional networks. This opens up a wide range of possibilities in the
neuromorphic vision domain to aggregate sparse, disjoint visual information
into more meaningful intermediate representations. We evaluated the
superevents on the event camera dataset and highlighted the gap in boundary
scores between the intensity and events modalities. Future work to address
this gap using completely asynchronous processing shall be explored.

## References

  * (1)
  * et al. (2020) S. Mohammad Mostafavi I. , Jonghyun Choi, and Kuk-Jin Yoon. 2020\.  Learning to Super Resolve Intensity Images From Events. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Achanta et al. (2012) R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk. 2012\.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 34, 11 (2012), 2274–2282.
  * Acharya et al. (pted) J. Acharya, A. U. Caycedo, V. R. Padala, R. R. Singh Sidhu, G. Orchard, B. Ramesh, and A. Basu. 2019 (Accepted).  EBBIOT: A Low-complexity Tracking Algorithm for Surveillance in IoVT using Stationary Neuromorphic Vision Sensors. In _32nd IEEE International System-on-chip Conference_.  https://bit.ly/2S501Dv
  * Alonso and Murillo (2019) I. Alonso and A. C. Murillo. 2019.  EV-SegNet: Semantic Segmentation for Event-Based Cameras. In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. 1624–1633.
  * Aung et al. (2018) Myo Tun Aung, Rodney Teo, and Garrick Orchard. 2018.  Event-based Plane-fitting Optical Flow for Dynamic Vision Sensors in FPGA. In _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 1–5.
  * Bagchi and Chin (2020) Samya Bagchi and Tat-Jun Chin. 2020.  Event-based star tracking via multiresolution progressive Hough transforms. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2143–2152.
  * Benosman et al. (2014) Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi. 2014.  Event-Based Visual Flow.  _IEEE Transactions on Neural Networks and Learning Systems_ 25, 2 (Feb 2014), 407–417.  https://doi.org/10.1109/TNNLS.2013.2273537
  * CamuÃ±as-Mesa et al. (2018) Luis A. CamuÃ±as-Mesa, Yaisel L. DomÃnguez-Cordero, Alejandro Linares-Barranco, Teresa Serrano-Gotarredona, and BernabÃ© Linares-Barranco. 2018\.  A Configurable Event-Driven Convolutional Node with Rate Saturation Mechanism for Modular ConvNet Systems Implementation.  _Frontiers in Neuroscience_ 12 (2018), 63.
  * Canny (1986) J. Canny. 1986\.  A Computational Approach to Edge Detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ PAMI-8, 6 (1986), 679–698.
  * Chen et al. (2020a) G. Chen, L. Hong, J. Dong, P. Liu, J. Conradt, and A. Knoll. 2020a.  EDDD: Event-Based Drowsiness Driving Detection Through Facial Motion Analysis With Neuromorphic Vision Sensor.  _IEEE Sensors Journal_ 20, 11 (2020), 6170–6181.
  * Chen et al. (2020b) J. Chen, J. Meng, X. Wang, and J. Yuan. 2020b.  Dynamic Graph CNN for Event-Camera Based Gesture Recognition. In _2020 IEEE International Symposium on Circuits and Systems (ISCAS)_. 1–5.
  * Chin et al. (2019) Tat-Jun Chin, Samya Bagchi, Anders Eriksson, and Andre Van Schaik. 2019.  Star tracking using an event camera. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_.
  * Feng et al. (2019) Wei Feng, Ruize Han, Qing Guo, Jianke Zhu, and Song Wang. 2019.  Dynamic saliency-aware regularization for correlation filter-based object tracking.  _IEEE Transactions on Image Processing_ 28, 7 (2019), 3232–3245.
  * Gallego et al. (2016) Guillermo Gallego, Jon EA Lund, Elias Mueggler, Henri Rebecq, Tobi Delbruck, and Davide Scaramuzza. 2016\.  Event-based, 6-DOF Camera Tracking for High-Speed Applications.  _arXiv preprint arXiv:1607.03468_ (2016).
  * Gallego et al. (2018) Guillermo Gallego, Henri Rebecq, and Davide Scaramuzza. 2018\.  A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE, 3867–3876.
  * Kogler et al. (2009) Jürgen Kogler, Christoph Sulzbachner, and Wilfried Kubinger. 2009.  Bio-inspired Stereo Vision System with Silicon Retina Imagers. In _Computer Vision Systems_ , Mario Fritz, Bernt Schiele, and Justus H. Piater (Eds.). Springer Berlin Heidelberg, 174–183.
  * Li et al. (2018) Xin Li, Fan Yang, Hong Cheng, Wei Liu, and Dinggang Shen. 2018.  Contour knowledge transfer for salient object detection. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 355–370.
  * Liu and Delbrück (2018) Min Liu and Tobi Delbrück. 2018.  Adaptive Time-Slice Block-Matching Optical Flow Algorithm for Dynamic Vision Sensors. In _BMVC_.
  * Low et al. (2020) Weng Fei Low, Zhi Gao, Cheng Xiang, and Bharath Ramesh. 2020\.  SOFEA: A Non-Iterative and Robust Optical Flow Estimation Algorithm for Dynamic Vision Sensors. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_.
  * Messikommer et al. (2020) Nico Messikommer, Daniel Gehrig, Antonio Loquercio, and Davide Scaramuzza. 2020.  Event-Based Asynchronous Sparse Convolutional Networks. In _Computer Vision – ECCV 2020_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). 415–431.
  * Mitrokhin et al. (2020) Anton Mitrokhin, Zhiyuan Hua, Cornelia Fermuller, and Yiannis Aloimonos. 2020.  Learning Visual Motion Segmentation Using Event Surfaces. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Mostajabi et al. (2015) Mohammadreza Mostajabi, Payman Yadollahpour, and Gregory Shakhnarovich. 2015.  Feedforward semantic segmentation with zoom-out features. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3376–3385.
  * Mueggler et al. (2015) E. Mueggler, C. Forster, N. Baumli, G. Gallego, and D. Scaramuzza. 2015.  Lifetime estimation of events from Dynamic Vision Sensors. In _2015 IEEE International Conference on Robotics and Automation (ICRA)_. 4874–4881.
  * Mueggler et al. (2017) Elias Mueggler, Henri Rebecq, Guillermo Gallego, Tobi Delbruck, and Davide Scaramuzza. 2017\.  The event-camera dataset and simulator: Event-based data for pose estimation, visual odometry, and SLAM.  _The International Journal of Robotics Research_ 36, 2 (2017), 142–149.
  * Ramesh et al. (2019) B. Ramesh, H. Yang, G. M. Orchard, N. A. Le Thi, S. Zhang, and C. Xiang. 2019\.  DART: Distribution Aware Retinal Transform for Event-based Cameras.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2019), 1–1.  https://doi.org/10.1109/TPAMI.2019.2919301
  * Ramesh et al. (2018) Bharath Ramesh, Shihao Zhang, Zhi Wei Lee, Zhi Gao, Garrick Orchard, and Cheng Xiang. 2018\.  Long-term object tracking with a moving event camera. In _BMVC_.
  * Rebecq et al. (2016) Henri Rebecq, Guillermo Gallego, and Davide Scaramuzza. 2016\.  EMVS : Event-based Multi-View Stereo.  _Bmvc_ (2016), 1–11.
  * Seok and Lim (2020) Hochang Seok and Jongwoo Lim. 2020.  Robust Feature Tracking in DVS Event Stream using Bézier Mapping. In _IEEE Winter Conference on Applications of Computer Vision, WACV_. 1658–1667.
  * Shu et al. (2013) Guang Shu, Afshin Dehghan, and Mubarak Shah. 2013\.  Improving an object detector and extracting regions using superpixels. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3721–3727.
  * Stutz et al. (2018) David Stutz, Alexander Hermans, and Bastian Leibe. 2018\.  Superpixels: An evaluation of the state-of-the-art.  _Computer Vision and Image Understanding_ 166 (2018), 1–27.
  * Ussa et al. (pted) A. Ussa, L. Della Vedova, V. Reddy Padala, D. Singla, J. Acharya, C. L. Zhang, G. Orchard, A. Basu, and B. Ramesh. 2019 (Accepted).  A low-power end-to-end hybrid neuromorphic framework for surveillance applications. In _British Machine Vision Conference - Workshops_.  https://tinyurl.com/y2qslgvt
  * Wang et al. (2020) Lin Wang, Tae-Kyun Kim, and Kuk-Jin Yoon. 2020.  Eventsr: From asynchronous events to image reconstruction, restoration, and super-resolution via end-to-end adversarial learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8315–8325.
  * Wang et al. (2019) Wenguan Wang, Jianbing Shen, Jianwen Xie, Ming-Ming Cheng, Haibin Ling, and Ali Borji. 2019\.  Revisiting video saliency prediction in the deep learning era.  _IEEE transactions on pattern analysis and machine intelligence_ 43, 1 (2019), 220–237.
  * Yang et al. (2020) F. Yang, Q. Sun, H. Jin, and Z. Zhou. 2020\.  Superpixel Segmentation With Fully Convolutional Networks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 13961–13970.
  * Yeo et al. (2017) Donghun Yeo, Jeany Son, Bohyung Han, and Joon Hee Han. 2017\.  Superpixel-based tracking-by-segmentation using markov chains. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1812–1821.
  * Zhao et al. (2015) Rui Zhao, Wanli Ouyang, Hongsheng Li, and Xiaogang Wang. 2015\.  Saliency detection by multi-context deep learning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1265–1274.
  * Zhao and Wu (2019) Ting Zhao and Xiangqian Wu. 2019.  Pyramid feature attention network for saliency detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3085–3094.
  * Zhu et al. (2017) A. Z. Zhu, N. Atanasov, and K. Daniilidis. 2017\.  Event-based feature tracking with probabilistic data association. In _2017 IEEE International Conference on Robotics and Automation (ICRA)_. 4465–4470.


# Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data

###### Abstract

Noise-robust speech recognition systems require large amounts of training data
including noisy speech data and corresponding transcripts to achieve state-of-
the-art performances in face of various practical environments. However, such
plenty of in-domain data is not always available in the real-life world. In
this paper, we propose a generative adversarial network to simulate noisy
spectrum from the clean spectrum (Simu-GAN), where only 10 minutes of
unparalleled in-domain noisy speech data is required as labels. Furthermore,
we also propose a dual-path speech recognition system to improve the
robustness of the system under noisy conditions. Experimental results show
that the proposed speech recognition system achieves 7.3% absolute improvement
with simulated noisy data by Simu-GAN over the best baseline in terms of word
error rate (WER).

Index Terms—  Generative adversarial network, contrastive learning, automatic
speech recognition

##  1 Introduction

Noise-robust automatic speech recognition (ASR) is a challenging task as huge
training data and corresponding transcripts are required to achieve state-of-
the-art word error rate (WER) performances [1, 2, 3, 4]. However, such plenty
of noisy data is not always available under some practical scenarios as well
as collecting target data and transcribing them are also time-consuming and
labor-intensive.

To address the problem of lacking enough noisy data, prior work [5] proposes
to train the ASR system with large amounts of clean data and then finetunes
with limited noisy in-domain data. Another work [6] proposes to extract the
pure noise segments from the limited noisy in-domain data and then recursively
adds them to the large amounts of clean data to generate the simulated noisy
data. Such “mixup” simulated noisy data are then utilized for the subsequent
ASR training.

Recent works [7, 8, 9] in the vision field propose to utilize the generative
adversarial network (GAN) to simulate the target-domain images. Furthermore,
with only a few minutes of real speech as labels, GAN is also successfully
applied in voice conversion task to generate high-quality speech [10]. The
prior works are the inspiration source of this paper.

In this paper, we propose a generative adversarial network to simulate noisy
data (Simu-GAN) with only 10 minutes of unparalleled in-domain noisy data for
supervision, which provides a promising solution for the noise-robust ASR
system with limited in-domain training data. When such clean-to-noisy mapping
in the Simu-GAN is trained well, it can generate a large amount of simulated
in-domain data at the run-time inference.

Specifically, the proposed Simu-GAN consists of a generator and a
discriminator. The generator aims to map the clean spectrum to the noisy
spectrum and the discriminator is introduced to distinguish the simulated
noisy spectrum from the real noisy spectrum [11]. At the training stage, the
multi-layer patch-wise contrastive loss [12] is utilized to learn the mutual
information (i.e., speech content) between the clean spectrum and the
simulated noisy spectrum. The discriminator aims to narrow the differences
between the simulated noisy spectrum and the real noisy spectrum (i.e.,
background noises). At the run-time reference, only the generator is required
to generate the simulated noisy data.

In addition, we propose a dual-path ASR system to evaluate the effectiveness
of the proposed Simu-GAN, where both the noisy data and clean data are
utilized in the training process. The experiment shows that the proposed Simu-
GAN could significantly improve the recognition accuracy and achieve
comparable performances compared with the upper-bound baseline.

Fig. 1: The block diagram of proposed Simu-GAN structure. $X$, $\hat{X}$ and
$Y$ are the clean features, simulated noisy features and real noisy features,
respectively. “RB” denotes the residual blocks, and “MLP” denotes two linear
layers followed by the ReLU activation function.

##  2 Simu-GAN ARCHITECTURE

Given the adequate clean data $X=\\{x\in\mathcal{X}\\}$ in the source domain
and the limited noisy data $Y=\\{y\in\mathcal{Y}\\}$ in the target domain, we
hope to generate the simulated noisy speech $\hat{X}$ that is close to the
real noisy speech $Y$. To this end, we propose the Simu-GAN architecture,
where the clean speech $X$ and the noisy speech $Y$ are not required to be
parallel.

###  2.1 Generator and Discriminator

The generator $G$ is designed to map the clean magnitude $X_{mag}$ to the real
noisy magnitude $Y_{mag}$ as shown in Fig.1. We first feed the clean features
$X_{mag}$ into two 2-D downsample convolutional layers with the kernel size of
(3$\times$3) and the stride of (2$\times$2) as an encoder to learn the
embeddings of the input features. Such encoded embeddings are then inputted to
nine residual blocks (RB) to learn deep representations. Each residual block
includes two convolutional layers with the kernel size of (3$\times$3) and the
stride of (1$\times$1) followed by one dropout layer. Finally, two transposed
convolutional layers with the kernel size of (3$\times$3) and the stride of
(2$\times$2) act as a decoder to upsample the deep representations to the
simulated noisy features $\hat{X}_{mag}$.

We now introduce the discriminator $D$ to distinguish where the inputs come
from (i.e., simulated or real). The simulated magnitude features
$\hat{X}_{mag}$ and the real noisy magnitude features $Y_{mag}$ are fed into
the discriminator, consisting of five 2-D convolutional layers with the kernel
size of (4$\times$4) followed by the LeakyReLU activation function. The stride
takes (2$\times$2) for the first three convolutional layers and (1$\times$1)
for the last two convolutional layers. At the training stage, the adversarial
loss [7] is employed as:

$\mathcal{L}_{GAN}(G,D,X,Y)=\mathbb{E}_{y\sim Y}\log D(y)+\mathbb{E}_{x\sim
X}\log(1-D(G(x))).$ (1)

By minimizing the adversarial loss, the simulated features $\hat{X}_{mag}$
learn to be visually like the real noisy features $Y_{mag}$. Although the
speech content of the simulated and noisy features are different, the
discriminator mainly distinguishes the two features by the background noises
[13].

###  2.2 Multi-layer Patch-wise Contrastive Loss

As the speech content of the clean and noisy features are different, we
introduce the multi-layer patch-wise contrastive loss between the clean
features $X_{mag}$ and the simulated features $\hat{X}_{mag}$ to learn the
mutual information (i.e., speech content information).

Multi-layer and patch-wise. As shown in Figure 1, we reuse the generator $G$
to learn the deep representations of the simulated features $\hat{X}_{mag}$.
We first set one small patch in the simulated representations as “query” and
then select the corresponding patch in the clean representations as the
positive sample and random 256 patches as the negative samples. Such patches
are reshaped via two linear layers with 256 units followed by the ReLU
activation function. The contrastive loss is conducted between the positive
sample and negative samples from 5 interested layers.

Contrastive Loss. We now introduce the contrastive learning to learn the
mutual information by calculating the cross-entropy loss between the “query”
and the positive/negative patches, formulated as:

$\mathcal{L}_{MPC}(G,X)=\sum_{l=1}^{L}\sum_{i=1}^{I}-\log\left[\frac{e^{(\hat{z}_{l}^{i}\cdot
z_{l}^{i}/\tau)}}{e^{(\hat{z}_{l}^{i}\cdot
z_{l}^{i}/\tau)}+\sum_{j=1}^{J}e^{(\hat{z}_{l}^{i}\cdot
z_{l}^{j}/\tau)}}\right]$ (2)

where $z_{l}^{i}$ and $\hat{z_{l}^{i}}$ denote the $i^{th}$ positive patches
in clean representations and simulated representations of the $l^{th}$ layers
in the generator, respectively. $\hat{z_{l}^{j}}$ is the $j^{th}$ negative
patches in simulated representations of the $l^{th}$ layers in the generator.
$\tau$ presents a temperature parameter in the contrastive learning [14].

Additionally, we also apply the multi-layer patch-wise contrastive loss
$\mathcal{L}_{MPC}(G,Y)$ to the real noisy features $Y_{mag}$ to prevent the
generator from making unnecessary changes [12]. Therefore, the total loss
function $\mathcal{L}_{total}$ of the Simu-GAN is formulated as:

$\mathcal{L}_{total}=\mathcal{L}_{GAN}(G,D,X,Y)+\lambda\mathcal{L}_{MPC}(G,X)+\omega\mathcal{L}_{MPC}(G,Y)$
(3)

where $\lambda$ and $\omega$ are both set as 1 in this work.

Fig. 2: The augmented dual-path ASR architecture. Two data flow are fed into
Conformer-based ASR network as two independent batches, and the KL loss is
computed between the outputs of decoder.

##  3 Dual-path ASR System

To improve the robustness of the ASR systems under noisy conditions, we
further propose a dual-path ASR system as shown in Figure 2. Specifically, we
input the simulated speech generated by the Simu-GAN model and the
corresponding clean speech as dual-path inputs into the conformer-based ASR
system. We first introduce two ASR losses $\mathcal{L}_{asr}^{c}$ and
$\mathcal{L}_{asr}^{n}$ calculated for the noisy path and clean path. We then
propose a KL divergence-based consistency loss between the two decoder outputs
of the noisy path and clean path as $\mathcal{L}_{kl}(X_{dec},\hat{X}_{dec})$.
Therefore, the loss of the dual-path ASR system $\mathcal{L}_{dp}$ is
formulated as:

$\mathcal{L}_{dp}=\alpha\mathcal{L}_{kl}(X_{dec},\hat{X}_{dec})+\beta\mathcal{L}_{asr}^{c}+(1-\beta)\mathcal{L}_{asr}^{n}$
(4)

where $\alpha,\beta\in[0,1]$ are the parameters to balance the ASR loss
$\mathcal{L}_{asr}^{c/n}$ and the KL loss
$\mathcal{L}_{kl}(X_{dec},\hat{X}_{dec})$. At the run-time inference, we only
utilize the noisy path to evaluate the test set as the clean data of the test
set is usually not available.

##  4 Experiments and results

###  4.1 Database

We conduct experiments on the dataset from robust automatic transcription of
speech (RATS) program [15], which is recorded with a push-to-talk transceiver
by playing back the clean Fisher data. The RATS has eight channels and could
provide clean speech, noisy speech, and corresponding transcripts for various
training goals. In this work, we select the data in channel A as the in-domain
noisy data. The channel A includes 44.3-hours of training data, 4.9-hours of
validation data, and 8.2-hours of testing data.

At the training stage for the proposed Simu-GAN, we only utilize small amounts
of unparalleled clean/noisy data from the channel A. At the run-time reference
for the Simu-GAN, we use the full clean data of the channel A to generate the
simulated noisy data for subsequent ASR training. Such clean data,
corresponding simulated noisy data and corresponding transcripts are utilized
in the ASR training.

###  4.2 Experimental setup

At the training stage for Simu-GAN, the clean magnitude features were cut into
the segments with the dimension of 129 $\times$ 128\. The network was
optimized by the Adam algorithm [16] and the learning rate started from 0.002.

At the training stage for all ASR baselines, the Conformer-based ASR system
takes 80-dimensional log-mel feature as the input, where the encoder includes
12 Conformer layers and the decoder consists of 6 Transformer layers [17]. The
byte-pair-encoding (BPE) [18] is utilized as the output token with a size of
994 and the shallow fusion [19] is employed to train a language model with the
corresponding transcripts. For the dual-path ASR system, we set the hyper-
parameters $\alpha$ and $\beta$ to 0.4 and 0.7, respectively.

###  4.3 Reference Baselines

To evaluate the effectiveness of the proposed Simu-GAN and the dual-path ASR
system, we built 5 baselines for comparison.

  * •

Clean-ASR [17]: we train the single-path Conformer-based ASR system only with
the clean RATS data.

  * •

SpecAugment [20]: we swap the one frequency bin and one frame as a data
augmentation approach for the clean-ASR system training.

  * •

Finetune [5]: we tune the pre-trained clean-ASR system with the 10 minutes
RATS channel A data.

  * •

Mixup [6]: we generate the simulated noisy data by adding the noises segments
extracting from full RATS channel A data and then train the single-path
Conformer-based ASR system with the “mixup” simulated data.

  * •

Noisy-ASR [17]: we train the single-path Conformer-based ASR system with the
real RATS channel A data, which is the upper-bound performances with full in-
domain data.

###  4.4 Results

####  4.4.1 Effect of the different amount of data for Simu-GAN training

We first analyze and summarize the WER performances of Simu-GAN with different
amounts of training data on the single-path ASR system. Specifically, we
utilize the different amounts of clean/noisy unparalleled data for the Simu-
GAN training, and then the approximate 50-hours clean data (44.3+4.9 hours) of
RATS channel A are utilized to generate the simulated in-domain data for the
subsequent ASR training and evaluation.

From Table 1, we observe that the performances significantly improve as the
amount of clean/noisy training data increases. We obtain the best WER of 68.9%
for the single-path ASR system with 1-hour clean training data and 10 minutes
of noisy training data. To further show the contribution of the proposed Simu-
GAN approach (system 5), we illustrate the magnitude spectrum of an example as
shown in Figure 3. We can see that the proposed Simu-GAN approach can produce
approximately the same spectrum (seen in Figure 3(b)) with the real RATS
channel A sample (seen in Figure 3(c)). More listening samples are available
at Github111https://chrisole.github.io/ICASSP2022-demo/.

Table 1: The comparative study of the different amount of data for the Simu-
GAN training. The WER (%) performances are evaluated on the single-path ASR
system using the real test set of RATS channel A. “Clean speech” denotes the
amount of the clean data of RATS channel A and “Noisy speech” presents the
amount of the RATS channel A data. No speed perturbation is utilized in the
following experiments.

| System
---
No.
Data for Simu-GAN training |  | WER
---
(%)
| Clean speech | Noisy speech |
1 | 1 min | 1 min | 87.2
2 | 2 min | 2 min | 74.2
3 | 5 min | 5 min | 72.7
4 | 10 min | 10 min | 72.4
5 | 1 h | 10 min | 68.9

Fig. 3: The magnitude of a sample (10315_21016.wav) for (a) clean magnitude,
(b) simulated noisy magnitude by Simu-GAN, and (c) real noisy magnitude from
RATS channel A (ground-truth).

####  4.4.2 Effect of the proposed dual-path ASR system

We further report the effect of the proposed dual-path ASR system trained by
the simulated noisy data, corresponding clean data, and transcripts shown in
Table 2. Such simulated noisy data are generated by the proposed Simu-GAN with
different amounts of training data. We observe that the proposed dual-path ASR
systems could further improve the performances compared with the single-path
ASR systems under the same amounts of training data. We obtain the best WER of
60.7% on the proposed dual-path ASR system with simulated data by Simu-GAN,
which is trained by 1-hour clean training data and 10 minutes unparalleled
noisy training data. We adopt this setting for Simu-GAN and the dual-path ASR
system hereafter.

Table 2: The comparative study of the proposed dual-path ASR systems. “# ASR
path” presents the number of path of the ASR systems (single-path or dual-
path). “S.P.” denotes the speed perturbation with $\\{\times 0.9,\times
1.0,\times 1.1\\}$.

| System
---
No.
Data for Simu-GAN training |  | # ASR path
---
S.P. |  | WER
---
(%)
| Clean data | Noisy data |  |  |
4 | 10 min | 10 min | Single | $\times$ | 72.4
10 min | 10 min | Single | $\surd$ | 67.0
5 | 1 h | 10 min | Single | $\times$ | 68.9
1 h | 10 min | Single | $\surd$ | 63.0
6 | 10 min | 10 min | Dual | $\times$ | 65.4
10 min | 10 min | Dual | $\surd$ | 61.5
7 | 1 h | 10 min | Dual | $\times$ | 65.4
1 h | 10 min | Dual | $\surd$ | 60.7

####  4.4.3 Benchmark against other competitive methods

Table 3 summarizes the comparison between the proposed Simu-GAN and other
competitive techniques in terms of the WER (%). We observe that the proposed
Simu-GAN obtained the best performance. Comparing with the “Mixup” methods,
the proposed Simu-GAN achieves the 7.3% absolute WER improvements. In
addition, our best performance is also close to the upper-bound baseline
trained by real in-domain data.

Table 3: The comparative study of other competitive techniques. Speed
perturbation with $\\{\times 0.9,\times 1.0,\times 1.1\\}$ are utilized in the
following experiments.

| Method
---
Name
All data requirements for training |  | WER
---
(%)
| Clean speech | Real noisy speech | Simulated noisy speech |
Clean-ASR | 44.3 h | - | - | 93.4
| SpecAugment [20]
---
(slight)
44.3 h | - | - | 84.7
Finetune [5] | 44.3 h |  | 10 min
---
with label
- | 73.0
Mixup [6] | 44.3 h |  | 44.3 h
---
without label
44.3 h | 68.0
Simu-GAN (ours) | 44.3 h |  | 10 min
---
without label
44.3 h | 60.7
Noisy-ASR | - |  | 44.3 h
---
with label
- | 49.5

##  5 Conclusion

We propose a generative adversarial network to simulate the noisy in-domain
speech from the clean speech (Simu-GAN) with 10 minutes of real noisy samples
as labels to address the problem that the in-domain data is limited. We also
propose a dual-path ASR system to improve the robustness of the ASR systems
under noisy conditions. Experimental results show that the proposed Simu-GAN
achieves the 7.3% absolute WER improvements on the dual-path ASR system
compared with the best baseline.

## References

  * [1] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates _et al._ , “Deep speech: Scaling up end-to-end speech recognition,” _arXiv preprint arXiv:1412.5567_ , 2014.
  * [2] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,” in _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2016, pp. 4960–4964.
  * [3] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina _et al._ , “State-of-the-art speech recognition with sequence-to-sequence models,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4774–4778.
  * [4] H. Hu, T. Tan, and Y. Qian, “Generative adversarial networks based data augmentation for noise robust speech recognition,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 5044–5048.
  * [5] D. Yu, L. Deng, and G. Dahl, “Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition,” in _Proc. NIPS Workshop on Deep Learning and Unsupervised Feature Learning_ , 2010.
  * [6] D. Ma, G. Li, H. Xu, and E. S. Chng, “Improving code-switching speech recognition with data augmentation and system combination,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2019, pp. 1308–1312.
  * [7] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [8] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE Signal Processing Magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [9] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [10] H. Kameoka, T. Kaneko, K. Tanaka, and N. Hojo, “Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks,” in _2018 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2018, pp. 266–273.
  * [11] N. Hou, C. Xu, E. S. Chng, and H. Li, “Domain adversarial training for speech enhancement,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2019, pp. 667–672.
  * [12] T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, “Contrastive learning for unpaired image-to-image translation,” in _European Conference on Computer Vision_. Springer, 2020, pp. 319–345.
  * [13] N. Hou, C. Xu, E. S. Chng, and H. Li, “Learning disentangled feature representations for speech enhancement via adversarial training,” in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 666–670.
  * [14] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3733–3742.
  * [15] D. Graff, K. Walker, S. M. Strassel, X. Ma, K. Jones, and A. Sawyer, “The rats collection: Supporting hlt research with degraded audio data.” in _LREC_. Citeseer, 2014, pp. 1970–1977.
  * [16] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [17] D. Ma, N. Hou, V. T. Pham, H. Xu, and E. S. Chng, “Multitask-based joint learning approach to robust asr for radio communication speech,” _arXiv preprint arXiv:2107.10701_ , 2021.
  * [18] T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing,” _arXiv preprint arXiv:1808.06226_ , 2018.
  * [19] S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang, M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang _et al._ , “A comparative study on transformer vs rnn in speech applications,” in _2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 449–456.
  * [20] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le, “Specaugment: A simple data augmentation method for automatic speech recognition,” _arXiv preprint arXiv:1904.08779_ , 2019.