Speed Invariant Time Surface for Learning to Detect Corner Points with Event-Based Cameras
Jacques Manderscheid Amos Sironi Nicolas Bourdis Davide Migliore Vincent Lepetit









Speed Invariant Time Surface
 for Learning to Detect Corner
  Points with  Event-Based Cameras
    
	Jacques Manderscheid^1, Amos Sironi^1, Nicolas Bourdis^1, Davide Migliore^1 and Vincent Lepetit^2

	^1Prophesee, Paris, France, ^2University of Bordeaux, Bordeaux, France

	{jmanderscheid, asironi, nbourdis, dmigliore}@prophesee.ai, vincent.lepetit@u-bordeaux.fr
					
    
==================================================================================================================================================================================================================================================================================





			
	We propose a learning approach to  corner detection for cameras that
        is stable  even under fast and  abrupt motions.  cameras  offer high
        temporal resolution, power efficiency, and high dynamic range.  However,
        the  properties of  data  are  very different  compared to  standard
        intensity  images, and  simple  extensions of  corner detection  methods
        designed for  these images do  not perform well  on data.   We first
        introduce an efficient  way to compute a time surface  that is invariant
        to the speed  of the objects.  We  then show that we can  train a Random
        Forest to  recognize events generated by  a moving corner from  our time
        surface.  Random Forests  are also extremely efficient,  and therefore a
        good choice to deal with the high capture frequency of cameras—our
        implementation  processes up  to 1.6Mev/s  on a  single CPU.
        Thanks to our  time surface formulation and this  learning approach, our
        method is  significantly more robust  to abrupt changes of  direction of
        the  corners  compared to  previous  ones.   Our method  also  naturally
        assigns a  confidence score  for the  corners, which  can be  useful for
        postprocessing.   Moreover,  we   introduce  a  high-resolution  dataset
        suitable for quantitative evaluation  and comparison of corner detection
        methods for cameras.  We call our approach SILC, for Speed Invariant
        Learned Corners, and  compare it to the  state-of-the-art with extensive
        experiments, showing better performance.


        
	
			
	There is  a public  dataset for  corner detection
		but, there is no common  benchmarck.  In particular people typically evealuate
		accuracy  either by  looking if  they detect  feature events  close to  harris
		corners  from  the  grayimages,  or  they  see  how  long  they  can  track  a
	feature. Similarly to the trackability we used.  
			
		
			
			
	To the best  of our
	knowledge,  our  approach  is  the  first machine  learning  method  applied  to
	individual  eventsnot sure  what  this  meansthis is  less
		clear to me.   What I meant was  that we claissify event by  event, instead of
		classifying the whole stream of  events.  A bit like pixel-wise classification
	vs  image  classification.
			
		Feature point detection is a classical computer vision problem where motion blur, 
	abrupt illumination changes and limited computational resources 
	drastically impact the accuracy of classical frame-based detectors. 
		Event-based cameras offer a new standard of performance for computer vision 
	and overcome these limitations 
	thanks to their high temporal resolution, 
	power efficiency and high dynamic range. 
							
																							 
	Existing approaches for feature detection rely on an adaptation of frame-based corner detectors 
	to the output of an sensor. Typically, this is done by building a 2D spatial representation around an event
	and running a hand-crafted corner detector.
		speed of the objects ?
	I am not sure "etc" is adequate in an abstract. What about the next paragraph ?
	As a consequence, these detectors are unstable under various conditions, including noise, changes of direction
	and speed of the objects.
	However, the properties of data are different compared to standard intensity images.
	As a consequence, these detectors are unstable in presence of noise, changes of direction etc.
					
	By contrast, we propose to learn an event-wise classifier
	to discriminate events generated by a moving corner. 
	The classifier is trained using only the precise timings of the input events 
	and can process up to XXMev/s on a single CPU.
		To the best of our knowledge, this is the first machine learning 
	method classifying individual events.
				
												
													We also introduce a novel quantitative evaluation of feature detectors
	by releasing a VGA benchmarked dataset (the first VGA and the first with such ground truth?). 
		We compare our method to the state-of-the-art with extensive experiments, 
	showing better performance (hopefully :) .
																				
 





§ INTRODUCTION








 
By  capturing   very  efficiently  local  illuminance   changes ('events'),  cameras <cit.> open the door to novel very fast
and  low-power  computer vision  algorithms  able  to  deal with  large  dynamic
ranges <cit.>.   However, because the
events are created asynchronously,  as shown in Fig. <ref> (a), novel
algorithms have  to be  developed to perform  fundamental computer  vision tasks
that are typically performed on regular frame images.


One of the main fundamental tasks is feature point detection, which is important
for applications with very strong dynamics  such as UAV navigation, where motion
blur makes classical  frame-based approaches less robust, or  visual odometry in
High  Dynamic  Range (HDR)  conditions,  among others.   Inspired  by  the  vast
literature  on   frame-based  feature   point  detection,  some   works  adapted
frame-based corner detector to data.  Typically, a local spatial descriptor
is built  around an  event, for  example by  cumulating events  in a  given time
window <cit.>,  or   by  considering  the   times  of  arrival   of  the
events <cit.>.  Then,  a classical test, such  as <cit.> can
be applied to this 2D spatial neighborhood.  However, the resulting detectors do
not take  into consideration the specific  characteristics of data,  such as
different  noise  patterns,  responses  to changes  of  direction,  illumination
changes, etc.  Even  if efforts have been  made in order to  design better tests
for  cameras <cit.>, hand-crafted  detectors  remain
unstable   and  corners  can  not   be   reliably   detected   over
time Fig. <ref> (b-c).


In this  paper, we  propose a  learning approach to  feature  detection.  We
train a classifier  to label individual events as generated  by a moving feature
point or  not.  The main  advantage of taking a  learning approach is  to obtain
more stable corners:  As shown in Fig. <ref>, a typical  error made by
previous detectors is  that  they are  sensitive to  changes of  the
apparent motion of  the feature points.  This is because  corners in cameras
are  not  invariant under  changes  of  direction,  by  contrast to  corners  in
intensity images.  Previous detectors also often erroneously detect points along
edges because of noisy events, while such  points cannot be detected in a stable
way.  Learning  makes the  detection more  robust to  motion changes  and noise,
without having to manually design an ad hoc method.


Our  classification  approach  relies  on   a  novel  formulation  of  the  Time
Surface <cit.>,  which  is  another contribution  of  this
work.  The Time Surface is a representation that accumulates the information from events
over time, and  is a common tool used in vision, including to detect
corner  points.  In  our work,  we also  use  a Time  Surface as  input to  the
classifier,  but we  show  how to  efficiently  create a  Time  Surface that  is
invariant  to the  objects'  speed.   Previous work <cit.>  already
introduced a method for computing a  time surface invariant to speed, however
it is still too slow  to  compute, 
and  incompatible  with the  high  frequency of  cameras.  The  invariance to speed of  our Time Surface is  important both to achieve
classification  performance and also to  keep  the classifier  small, 
which makes computation fast.




One critical aspect of our learning-based approach is indeed that classification
must be performed  extremely fast, otherwise we would lose  the advantage of the
capture efficiency of  cameras.  We therefore chose to  use a Random Forest,
as Random Forests  are very efficient without  having to use a  GPU (unlike Deep
Networks),  which   would  be  counter-productive  since   we  target  low-power
applications. In fact, parallelizing computation as done by  GPUs is not well adapted
to  the  sparse  and  asynchronous  nature of  the  events. 
Our  current implementation processes up to 1.6·10^6 events
per second on  a single CPU.

To evaluate the  quality of our detector, we also  release a new high-resolution
benchmark dataset.   We propose a  metric which  is independent of  ground truth
keypoints  extracted from  gray level  images, which  are often  used but  would
introduce  a  strong bias  in  the  evaluation.   Specifically, we  compare  our
approach to  different detectors in  combination with a simple  nearest neighbor
based tracking, showing that, thanks to the temporal continuity of the events, a
very simple tracking rule can lead to state-of-the-art results.











				














	
 




§ EVENT-BASED CAMERAS




Standard frame-based  cameras capture visual information  by acquiring snapshots
of the observed scene at a fixed rate. This might result 
in motion blur for highly dynamic scenes
and in redundant data generation for static ones.
By contrast,  cameras <cit.>,  a relatively
recent type of cameras, adaptively record information from a scene, depending on
the content.   More precisely, each pixel  in an sensor  is independent from
the rest of the pixel array.  When a change in the log-luminosity intensity at a
location surpasses  a threshold, the  corresponding pixel emits  an event.
An event contains  the 2D location of  the pixel, the timestamp  of the observed
change and its polarity, a binary variable indicating whether the luminosity
increased or decreased.
Thanks  to  these characteristics,  cameras  have temporal
resolution in the order of the microsecond, high dynamic  range,
and are  well suited  for low-power
applications.

Some types of cameras also provide gray-level measurements together with the
change detection events.  The DAVIS <cit.> sensor for example, is a 240×180
sensor able
to   output   intensity   frames   at    standard   frame   rates.    The   ATIS
sensor <cit.>  instead provides  asynchronous intensity  measurements in
form of time differences at the same temporal resolution of the events.
Our  method can  be applied  to any  type of  cameras  and does  not require
gray-level information at run time.  Intensity information has been used in this
work only  to build the  dataset used to train  our classifier, as  explained in
Section <ref>.

Finally, we note that the spatial resolution of
cameras, still limited to low resolution, is 
likely to soon reach standard resolutions of frame-based
cameras <cit.>.  As a consequence, algorithms will have to 
process constantly increasing event rates.
In this  work, we  use and  release the first  dataset  acquired with  a HVGA
(480×360) sensor, showing  that our algorithm is suited to  real time and
high data rate applications.
 


§ RELATED WORK




In this section, we review related work on feature detection,
machine learning approaches applied to cameras and to feature point detection.




  
Features Detection
feature  detection methods can  be divided  into two categories.   The first
class of  methods <cit.> aims  at detecting  a particular
pattern in the stream of events  by applying local template matching rules.  For
example, a  blob detector <cit.> can  be obtained by using  a gaussian
correlation kernel.  When  enough events are received in the  receptive field of
the kernel, a  feature event is generated. The  method is
generalized to generic  shapes, like corners or T-junctions. 
In <cit.>  instead, corners are identified  as intersection of
local  velocity  planes,   which  are  obtained  by  fitting  a   plane  on  the
spatio-temporal  distribution  of  events.   This   class  of  methods  is  very
efficient, but it is sensitive to noise and requires careful tuning of the model
parameters.
		
The  second  class of  methods  instead,  relies  on the  adaptation  of
classical frame-based corner detectors to data.  The basic principle
behind these  approaches is  to build a  local representation  around an
event  and  then apply  classical  corner  tests  on it.   For  example,
in <cit.>  the  Harris  corner  detector is  applied  to  images
obtained  by cumulating  a fixed  number of  events in  a local  spatial
window.   The  Surface of  Active  Events  (or Time  Surface,  presented in
Sec. <ref>)  is a  common representation  used in
vision <cit.>.   It  has  been
used in <cit.>  for corner  detection, where the  FAST 
algorithm <cit.> is adapted to the  pattern of Time Surface of a
moving corner.  In <cit.> this formulation has been extended
to corners with obtuse angles.
These methods  lose accuracy mostly in  case of changes of  feature point motion
direction, because  data do not  share the same characteristics  as standard
intensity images.   In particular, the corner  appearance in a stream  of events
depends  on its  motion, as  shown in  Fig. <ref>.  As  a
consequence,  the constant  appearance  assumption made  in frame-based vision
does not hold in the event-based case.
		
To   overcome   this   problem,   some  works   rely   on   gray   level
images <cit.>.   In  these  approaches,
feature points are detected in the images using standard techniques, and
then tracked in between by  using the event stream.  These methods
have the same  limitation as frame-based approaches,  namely they lose
accuracy in presence of motion  blur, HDR situations, and increase data
rate and the power consumption.   Moreover, they require either an sensors  able   to  acquire   graylevel  information,  or   a  careful
synchronization  of  and  frame-based  sensors.   By contrast, our
method requires at run time only  a single and purely sensor.   We claim that
the time  information carried  by the events  is sufficient  to reliably
detect feature  points, without the  need of intensity  information.  We
propose to  use a machine learning  approach to learn the  appearance of
the features directly from the input data.

Finally,    many    works   focus    on    the    problem   of    feature
tracking <cit.>.   Our  method
can be combined  with any of these tracking algorithms.   However, thanks to the
high temporal  resolution of  the cameras, given  our stable  and efficient
feature detector,  the problem of  data association is  made much simpler  and a
simple nearest neighbor matching rule is sufficient in our experiments to obtain
accurate results. 


 



  
Learning from Events
Machine  learning  approaches  for  cameras can  also  be  divided  in  two
categories.                    In                    the                   first
category <cit.>,
the events from the camera are accumulated for  a given period of time, or for a
fixed number  of events, to  build a dense representation.   This representation
can then be  effectively processed by standard Computer  Vision techniques, like
Convolutional Neural Networks (CNNs).
		
In <cit.> for example, the input events are summed into histograms and
the obtained images are used to predict the steering angle of a car using a CNN.
Similarly, in <cit.>, histograms and time surfaces are used to predict the
optical flow.   In <cit.>,  dense
descriptors are  built by comparing and  averaging the timings of  the events in
local windows. The descriptors  are then passed to a classifier,  such as an SVM
or a  neural network,  for object  recognition or other  vision tasks.   Even if
these approaches  reach good  performance and are  efficient, they  increase the
latency of the system by requiring an artificial accumulation time.
		
A  second class  of  methods  avoids this  limitation  by  processing the  input
event-by-event <cit.>.    The  most
common     model    used     in     cameras     are    Spiking     Neural
Networks <cit.>,
which have been  proven to reach good accuracy for  simple classification tasks.
However, these approaches are difficult to scale because of the large event rate
of    the   cameras.    Hardware    implementations   and    neuromorphic
architectures <cit.> have  been proposed to
overcome this  bottleneck. However, these  architecture are not large  enough to
support large networks and high-resolution cameras as input.
		
By contrast our classifier is  applied event-by-event, keeping the original time
resolution of the camera, while running on a standard CPU.
	  




  
Learning Frame-based Keypoint Detection.
A  number of  works  in frame-based  computer  vision focus  on  the problem  of
learning feature  points, as  we do here  for computer  vision.  One  of the
first   approaches   using   machine    learning   for   corner   detection   is
FAST <cit.>, where  a decision  tree is  used to  generalize brightness
tests  on  contiguous  circular  segments  introduced  in <cit.>.   The
motivation for  using learning in  FAST was to speed  up the detection  by early
rejection of non-corner points.  Other authors <cit.> also showed
how  to learn  a fast  approximation of  existing detectors.   But other  authors
follow this  direction by training  efficient ensembles of decision  trees.  For
example, in <cit.>, the WaldBoost  algorithm <cit.>
is used to learn detectors for specific tasks.
	
More recent works rely on Deep Learning models <cit.>.
For example, in <cit.>, a regressor  is trained to detect stable points
under  large illumination  changes.   In <cit.>, a  fully-convolutional
model is  trained with self-supervision  to jointly predict interest  points and
descriptors.



        
	
	


 


§ METHOD



In this section, we first introduce  our speed invariant formulation of the time
surface, then we  formalize the problem of learning a  feature detector from the
output  of an  camera.  Finally,  we explain  how  to create  a dataset  for
training  a corner  events detector.   An  overview of  our method  is given  in
Fig. <ref>.




 §.§ Asynchronous Event-based Representation of a Visual Scene



As  described in  Section <ref>,  the output  of an  camera  is an  asynchronous stream  of events  {_i}_i∈.  Each  event
_i represents a contrast change at a given pixel and at a given time and can be
formalized as

    _i = (x_i, y_i, t_i, p_i)  ,

where (x_i,y_i)  are the  pixel coordinates  of the  event, t_i≥0  is the
timestamp at which the event was generated, and p_i∈{-1,1} is the polarity
of  the event,  with  -1 and  +1 meaning  respectively  that the  luminosity
decreased or increased at that pixel.

Given an input event _i, corner detectors typically rely on
a spatio-temporal neighborhood of events around _i to build a local descriptor.
The descriptor is then used to test the presence of a corner.
In this work, we consider as in <cit.> the Time Surface <cit.>
as local descriptor. However, we show how the variance of the standard 
Time Surface formulation is not suited for stable corner detections.
We therefore introduce a normalization scheme to make the time surface invariant to speed.
Moreover, we adopt a machine learning approach and use the invariant time surface 
as input to a classifier trained to discriminate corner events.




 §.§ Speed Invariant Time Surface



A common  representation used  in vision is  the Surface  of Active
	Events <cit.>,       also      referred       as      Time
	Surface <cit.>.   The  Time Surface  T  at  a pixel  (x,y)  and
polarity p is defined as

    T(x,y,p) ← t  ,

where t  is the time  of the  last event with  polarity p occurred  at pixel
(x,y).  The  Time Surface, besides being  very efficient to compute,  has been
proved       to      be       highly       discriminative      for       several
tasks <cit.>.  However, we  noticed that local patches
of the Time  Surface can have a  very large variability. In fact,  depending on the
speed, the  direction and the  contrast of the corners,  the aspect of  the time
surface can  vary significantly. 
To  keep the classification model  compact and
efficient, it is thus important to introduce some normalization of its input.


Inspired by <cit.>,  we notice  that only  relative timings,  and not
absolute ones, are  relevant to determine the presence of  a corner. We could therefore
obtain invariance by normalizing 
the times of the events according to their
local speed.  However, normalization  techniques  suitable  for
intensity  images  cannot   be  applied  to  the  time   surface,  as  explained
in <cit.>.  <cit.>  proposed to sort the  times of
the events in a  local patch. However, sorting a  local patch at every
incoming  event is  still too  expensive, and  requires to  store multiple  time
stamps per pixel.


Instead, we show how to obtain efficiently a Time Surface that is independent of
the speed,  by keeping a single value for  each pixel location (x,y)
and every polarity.  We call this novel formulation Speed Invariant Time
Surface.

The intuition for our new formulation goes as follows: Imagine the contour of an
object moving in  one direction. We would  like to capture the  'wave' of events
generated by this  contour, but this wave should have  the same profile whatever
the speed of the contour.  When a  new event arrives, we store a large
value at  its pixel  location in the Time  Surface, and
decrease the values for the surrounding locations. In this way, we progressively
reduce  the   values  for  the   events  previously  generated  by   the  moving
contour. Since  we decrease the  values by a constant  factor, the slope  of the
'wave' is independent of contour's speed.

More precisely,  given a parameter r,  we initialize the Speed  Invariant Time
Surface  S(x,y,p) to  0 for every  pixel location  (x,y) and
every  polarity p. Then,  for every  incoming event
(x,y,p), we consider all pixel locations (x',y') in its neighborhood of size
(2r+1)×(2r+1).   If  S(x',y',p) is  larger  than  S(x,y,p), then  we
subtract  1 to  S(x',y',p).  Finally,  we set  S(x,y,p) to  (2r+1)^2.  The
pseudo-code        of       the        algorithm        is       given        in
Algorithm <ref>.


 Fig. <ref> illustrates  the application  of this  algorithm to
several edges moving from left to right,  each edge moving at a different speed.
For the  standard Time  Surface, each  edge generates  a different  slope behind
it. On  the Speed  Invariant Time  Surface, the slope  is the  same for  all the
edges, and has a  total length of 5 pixels which is equal  to the parameters r
used for this experiment. Also, the Speed Invariant Time Surface is constant
after the slope with a value of r+1. Finally, it can be seen that the values
in the Speed Invariant Time Surface remain between 0 and 2r+1.

In Fig. <ref>,  we compare  the standard  Time Surface,
the sorting normalization method of <cit.>, and our Speed Invariant
Time Surface.  The two normalization approaches achieve
similar  results for  both examples  with  a high  contrast around  the edge  in
comparison with  the standard  Time Surface.   Furthermore, our  Speed Invariant
Time Surface increases this contrast by reducing the values after the edge.





 




 §.§ Learning to Detect Corner Points from Events



 
Previous corner detectors rely on hand-crafted rules which make them unstable
when the input events do not follow their assumptions.
For example, is not able to detect corners during a change of motion. 
In the case of , the detection of obtuse angles lead to a high number of false
detection on edges. Finally, is less robust on
fast motions and computationally too expensive. This is why we
train a classifier to detect  corner points from events.  More exactly, given
an incoming event  _i = (x_i,y_i,p_i,t_i), we first update  our time surface
S as  explained in Section <ref>,  and we extract from  S a
local patch _i of size n× n  centered on the event 2D location (x_i,
y_i).

An ideal detector ^* taking _i as input is defined as

    ^*(_i) = {[                              1 if _i  is a feature event, and;                              0                    otherwise . ].




In principle, any classifier could be used to implement , but in practice we
chose  a  Random Forest <cit.>  because  of  its efficiency.
For the sake of completeness, in the following we briefly describe Random Forests.
A Random Forest classifier  is given by an ensemble of decision trees F_l:

    () = 1/L∑_l=1^LF_l() .

A  decision tree  F_l() classifies  a  sample = (s_1,…,s_K)  by
recursively branching  the nodes of the  tree until the sample  reaches a leaf
node.  In each node a binary split  function is applied to the input to decide
if the sample is sent to the left or the right child node.
  
In  practice, we  use decision  stumps as  split functions <cit.>,
where a descriptor  dimension d_k is compared to a  threshold th.  This is
computationally efficient  and effective in  practice <cit.>.  The
output of the tree is the prediction stored at the leaf reached by the sample,
which in our case is the probability of the sample being a feature point.
  
During training,  each tree is trained  independently in a greedy  manner, one
node at the time.  For a given node, let _N be the training set of samples
that reached this node. The optimal θ = (k,th) for the split
are  obtained  by  minimizing  the Gini  impurity  index <cit.> by exhaustive  search. 
The dataset is then split
in left and right _L, _R and the children nodes are trained recursively.
Training stops when a  maximum depth is
reached, when  a minimum number of  samples is reached or  if the
impurity index is too low.
  
A Random  Forest improves the accuracy  of a single decision  tree by training
multiple  trees. The  trees  need to  be  uncorrelated, and  this  is done  by
randomly sampling  a subset of  the training set for  each tree and  by randomly
subsampling a subset  of the descriptors dimensions at each  node.









𝒩(_i) = {_j : t_j ≤ t_i} is the set of events used as input to 
to the classification method, which will be used in our case to build 
the Speed Invariant Time Surface descriptor,
and the binary  class c_i  of event
_i  is 1  if _i  is a  feature event,  0 otherwise.there  is
	probably regions of the time surface around the point somewhereyes, in theory
	the whole stream of events up to _i can be used. Modified above..







 §.§ Building a Dataset for Corner Detection



To train our classifier , we  need a labeled training set {(_j, c_j)},
where the binary class c_j of event _j  is 1 if _j is a feature event,
0 otherwise.  Building such a dataset in the case of cameras is not trivial.
Manual annotation is impractical, since it  would require to label single events
in a sequence of million of events per second.

We propose to leverage the graylevel measurement provided by a HVGA ATIS sensor,
already discussed in Section <ref>.  For every event, we
apply the Harris corner detector to its location only, in the graylevel image.
If a corner is detected  at this location, we add the event to our training set
as a positive sample, and as a negative sample otherwise.
In practice, even the Harris detector  applied to graylevel images can sometimes
fail in presence  of noise, and we  therefore decided to acquire  our dataset by
recording    well     contrasted    geometric     patterns. 
Examples of this dataset are shown in the supplemen
tary material
The dataset can be downloaded at the following URL[ ].
In  this  way,  we greatly  reduce  the  number  of
outliers in the dataset.  As we will see in Section <ref>, even if the
pattern  we  use  is quite  simple,  the  classifier  learnt  on such  data  can
generalize well in more complex real situations. We also extend this 
	protocol to a DAVIS sensor. In this case the corners are detected on entire frames
	at regular intervals and all the events falling within 2 pixels form a detected corner 
	in a time interval of 5ms
are marked as corner events.









 


§ EVALUATION OF EVENT-BASED DETECTORS





  
Event-based Datasets for Corner Detection


We evaluate  our method  on two datasets.  The first  is the  commonly used
Event-Camera  dataset <cit.>.   It  is  composed  by  recordings  of
different scenes  taken with a  DAVIS sensor. As done  by previous  works, 
we consider  the
 and  subsets for evaluation,
for which camera  poses groundtruth are provided, 
but we keep the simplest  dataset to train our model.



The second dataset we use is a new dataset that we introduce for the first time in this paper. 
Compared to the Event-Camera dataset, ours was acquired using a higher resolution 
HVGA ATIS sensor, 
thus allowing better localization and finer feature detection,
together with higher data rate. 
While the Event-Camera dataset is more generic and adapted to test visual
odometry and SLAM applications, 
the intent of our dataset is to specifically evaluate the accuracy of 
feature detection and tracking algorithms.
We call this dataset the HVGA ATIS Corner Dataset.
It consists of 7 sequences  of increasing difficulty, from  a standard
checkerboard to a complex natural image (Fig. <ref>).
We record planar
patterns, so that ground truth acquisition  is simple and the evaluation is less
affected by triangulation errors.




  
Evaluation Metrics


Previous approaches for  feature detection rely on  Harris corners extracted
from  graylevel  images <cit.>.   This approach  has  the
shortcoming  of  penalizing features  when  not  matched to  any  graylevel
one, even  if these features might  correspond to stable  points in
the events stream. We notice that our  approach, since it  was trained starting from  Harris corners
would have an advantage when using this metric.

Other  works <cit.> instead, consider a detected corner valid if, 
when associated with nearby corners, it forms a well localized track in space and
time.  This metric evaluates how easy it  is to track a given feature.  However,
it does not take into account the correct trajectory of the feature.






We also start from the observation that a stable detector, able to continuously
identify a  feature in the  events stream, would remove 
the  need  of  complex  tracking  and  data
association methods. 
Therefore, in order to assess also the quality of  an detector, we combine it with
a simple  tracking algorithm  based on  nearest neighbor  matching in  space and
time.
After  tracking,  we can evaluate  the  accuracy  of  the  method by  computing  the
reprojection error associated to the feature tracks. 
	
In the case of the planar dataset,
the  reprojection error  is  computed  by estimating  a  homography between  two
different timestamps.  More precisely, given  two timestamps t_1 and t_2, we
collect all the  features that fall within a 5ms  timewindow from these times.
We then consider the last feature for each track. This gives two sets of 2D
correspondences that  can be
used to  estimate the homography  between the two views  of the camera.   We use
RANSAC for a  robust estimation.  Once the homography is  computed, we reproject
points from time t_2 to the reference  time  t_1 and compute the  average
distance between  the reference points and the projected ones. During this process, 
we exclude points detected outside the planar pattern.

In the case of the Event-Camera dataset, which contains 3D scenes, the reprojection error 
is computed by triangulating the tracked points, using the available 3d poses. 
We use the same protocol as in <cit.> and report also the percentage
of tracks with an error smaller than 5 pixels.
Finally we compare the methods in terms of computational
time, as done in <cit.>. 
All our experiments were implemented in C++ and run on a Xeon CPU E5-2603 v4 at 1.70GHz.
 


§ EXPERIMENTS




	

  
Parameters and Baselines
																				
	Our method depends on few parameters, namely the radius r  used to compute the 
	Speed Invariant Time Surface, the size n of the classifier input patch, 
	and the parameters for the Random Forest.
		The parameters were optimized by cross-validation on the training set of
	Section <ref>  to  minimize  the  misclassification
	error.  Once  the best parameters have  been found, we fix  them and use
	them for all the test sequences of Section <ref>.
		For  the descriptor, we  set r=6  and n  = 8.   For the
	Random Forest, we use 10 trees. We stop growing the trees when there are
	less than 50 training samples in a branch.
						        
																					
	We  compare   our  method  against  previously   published  feature
	detectors: the  Harris  detector of <cit.>, which  we denote
	; the  Fast of <cit.> ;  and its recent modification
	 <cit.>.   For all of  these methods we  use the
	publicly available implementation provided by the authors.
																				
	As  done in <cit.>,  we  also apply  an  'trail'  filter
	before each detector.  This filter removes the multiple events generated
	by  a  contrast  step  in  intensity  which  caused  multiple  threshold
	crossing.    The  timings   of  Table <ref>   are
	reported taking into account this filtering.
				
	

 	
						
	

  
Ablation Study
	In the first set of experiments,  we quantify the improvement brought by
	our Speed Invariant Time  Surface formulation against  the standard
	Time Surface.
										
	We train two Random Forests, one  using local patches extracted from the
	Time Surface of <cit.>, and  the second  one on
	Speed Invariant Time Surface patches.  Then, we apply these detectors to
	the HVGA dataset and track the features using nearest
	neighbor  data association  with a  radius of  3 pixels  and a  temporal
	window of 10ms.  From the tracked features we  estimate the homography
	using different time steps, as explained in Section <ref>.
	The    corresponding    reprojection     errors    are    reported    in
	Table <ref>.   As we  can  see our  method has  lower
	reprojection error, the reason is that  the detector trained on the Time
	Surface can not  generalize well, and, a part for  the simple chessboard
	pattern returns  very noisy  detections. We  refer to  the supplementary
	material for a visual comparison of the detected features.

	
	
	
	

  
Evaluation on the HVGA ATIS Corner Dataset
	
     							
	In  this section,  we compare  the  stability of  feature  detectors
	against the  baseline methods  using the homography  reprojection error.
	Since this metric  can be applied only  to planar scenes, we  use it for
	our dataset and  not on the Event-Camera dataset.   For nearest neighbor
	data association, we used a radius of  3 pixels and a temporal window of
	10ms.
					We also  compute the lifetime  of the  tracked features, defined  as the
	difference between  the first and the  last detection of the  track.  We
	then compute the average lifetime of the first 100 features.
										
	We  compute the  homography at  different time  intervals, ranging  from
	25ms   to   200ms.   The   results  for   100ms   are   shown   in
	Fig. <ref>.   
	Detailed values  for other
	values and for each sequence are reported in the supplementary material,
	and show similar  behavior.  
	Our method is able to  track a corner point
	longer while having a low reprojection  error.  The error for is
	slightly  lower,  but  the  tracks for  this  method  are  significantly
	shorter.  Finally,  and are very sensitive  to noise
	and respond on  edges, which make their tracks longer  then , but
	poorly reliable.
										
	Qualitative results are shown in Fig. <ref>. 
	A snapshot example can be found in Fig. <ref>.
	Corresponding video sequences are provided in the supplementary material.
							
	

	
	

  
Evaluation on the Event-Camera Dataset
															   
	We repeat similar  experiments for the Event-Camera  Dataset. Since the
	resolution of  the camera  is lower  for this dataset,  we use  a larger
	temporal window, equal to 50ms, for  the tracking.  Because of the lower
	resolution  and  the  higher  level  of  noise,  this  dataset  is  more
	challenging.
						
										
	The reprojection  error and the percentage of valid tracks
	are   given   in  Tab. <ref>.
	Detailed  values are reported  in the  supplementary
	material.						
	We notice that applying the model trained on the ATIS camera (SILC ATIS) generalizes well on the DAVIS, 
	reaching similar performance than the baseline.
	Since the DAVIS camera provides gray-level images, 
	we also retrained a Random Forest on the DAVIS data (SILC DAVIS).
	We use a small model composed of 10 trees of depth 10.
					We observe that the results obtained with our detector and a simple nearest neighbor tracking are 
	comparable with results obtained with complex trackers <cit.>.
							

	

	
																															
	

												

 


§ CONCLUSION AND FUTURE WORK




We presented an efficient and accurate learning approach for corner detection.
Our method produces more stable and repeatable corners compared to the state of-the-art
and when coupled with a simple tracking algorithm gives good accuracy.
A key component for our approach is a novel formulation of the Time Surface,
which provides a rich representation which is invariant to the local speed of the object.
In the future, we plan to apply the Speed Invariant Time Surface to other
vision tasks, such as low-latency object detection or relocalization.



    Dr. Vincent Lepetit is a senior member of the Institut Universitaire de France.


ieee
10=-1pt

Akopyan15
F. Akopyan, J. Sawada, A. Cassidy, R. Alvarez-Icaza, J. Arthur, P. Merolla,
  N. Imam, Y. Nakamura, P. Datta, G.-J. Nam, et al.
Truenorth: Design and Tool Flow of a 65 Mw 1 Million Neuron
  Programmable Neurosynaptic Chip.
IEEE Transactions on Computer-Aided Design of Integrated
  Circuits and Systems, 2015.

Altwaijry16
H. Altwaijry, A. Veit, S. J. Belongie, and C. Tech.
Learning to detect and match keypoints with deep architectures.
In BMVC, 2016.

Alzugaray18b
I. Alzugaray and M. Chli.
ACE: An Efficient Asynchronous Corner Tracker for Event Cameras.
In 2018 International Conference on 3D Vision (3DV), 2018.

Alzugaray18
I. Alzugaray and M. Chli.
Asynchronous Corner Detection and Tracking for Event Cameras in Real
  Time.
IEEE Robotics and Automation Letters, 2018.

Andreopoulos18
A. Andreopoulos, H. J. Kashyap, T. K. Nayak, A. Amir, and M. D. Flickner.
A Low Power, High Throughput, Fully Event-Based Stereo System.
In CVPR, 2018.

Benosman14
R. Benosman, C. Clercq, X. Lagorce, S. Ieng, and C. Bartolozzi.
Event-Based Visual Flow.
IEEE Trans. Neural Netw. Learning Syst., 2014.

Benosman12
R. Benosman, S.-H. Ieng, C. Clercq, C. Bartolozzi, and M. Srinivasan.
Asynchronous Frameless Event-Based Optical Flow.
Neural Netw.x, 2012.

Bichler12
O. Bichler, D. Querlioz, S. J. Thorpe, J.-P. Bourgoin, and C. Gamrat.
Extraction of Temporally Correlated Features from Dynamic Vision
  Sensors with Spike-Timing-Dependent Plasticity.
Neural Networks, 2012.

Bohte02
S. M. Bohte, J. N. Kok, and H. La Poutre.
Error-Backpropagation in Temporally Encoded Networks of Spiking
  Neurons.
Neurocomputing, 2002.

Brandli14
C. Brandli, R. Berner, M. Yang, S.-C. Liu, and T. Delbruck.
A 240× 180 130 dB 3 μs Latency Global Shutter
  Spatiotemporal Vision Sensor.
IEEE Journal of Solid-State Circuits, 2014.

Breiman01
L. Breiman.
Random Forests.
Machine learning, 2001.

Cannici18
M. Cannici, M. Ciccone, A. Romanoni, and M. Matteucci.
Event-Based Convolutional Networks for Object Detection in
  Neuromorphic Cameras.
In arXiv, 2018.

Cao15
Y. Cao, Y. Chen, and D. Khosla.
Spiking Deep Convolutional Neural Networks for Energy-Efficient
  Object Recognition.
IJCV, 2015.

Clady15
X. Clady, S.-H. Ieng, and R. Benosman.
Asynchronous Event-Based Corner Detection and Matching.
Neural Networks, 2015.

Clady17
X. Clady, J.-M. Maro, S. Barré, and R. B. Benosman.
A Motion-Based Feature for Event-Based Pattern Recognition.
Frontiers in neuroscience, 2017.

Criminisi12
A. Criminisi, J. Shotton, E. Konukoglu, et al.
Decision forests: A unified framework for classification, regression,
  density estimation, manifold learning and semi-supervised learning.
Foundations and Trends® in Computer Graphics and
  Vision, 2012.

Davies18
M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou,
  P. Joshi, N. Imam, S. Jain, et al.
Loihi: A Neuromorphic Manycore Processor with On-Chip Learning.
IEEE Micro, 2018.

Detone18
D. DeTone, T. Malisiewicz, and A. Rabinovich.
Superpoint: Self-supervised interest point detection and description.
In The IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR) Workshops, June 2018.

Di18
P. Di Febbo, C. Dal Mutto, K. Tieu, and S. Mattoccia.
Kcnn: Extremely-efficient hardware keypoint detection with a compact
  convolutional neural network.
In CVPR Workshop on Embedded Vision (EVW), 2018.

Drazen11
D. Drazen, P. Lichtsteiner, P. Häfliger, T. Delbrück, and A. Jensen.
Toward Real-Time Particle Tracking Using an Event-Based Dynamic
  Vision Sensor.
Experiments in Fluids, 2011.

Friedman01
J. Friedman, T. Hastie, and R. Tibshirani.
The Elements of Statistical Learning.
Springer series in statistics New York, NY, USA:, 2001.

Furber14
S. B. Furber, F. Galluppi, S. Temple, and L. A. Plana.
The Spinnaker Project.
Proceedings of the IEEE, 2014.

Gehrig18
D. Gehrig, H. Rebecq, G. Gallego, and D. Scaramuzza.
Asynchronous, Photometric Feature Tracking Using Events and Frames.
In ECCV, 2018.

Glover17
A. Glover and C. Bartolozzi.
Robust Visual Tracking with a Freely-Moving Event Camera.
In Intelligent Robots and Systems (IROS), 2017 IEEE/RSJ
  International Conference on, 2017.

Gutig06
R. Gütig and H. Sompolinsky.
The Tempotron: A Neuron That Learns Spike Timing–Based Decisions.
Nature neuroscience, 2006.

Harris88
C. Harris and M. Stephens.
A Combined Corner and Edge Detector.
In Alvey vision conference, 1988.

Kim16
H. Kim, S. Leutenegger, and A. J. Davison.
Real-Time 3D Reconstruction and 6-DoF Tracking with an Event
  Camera.
In ECCV, 2016.

Kueng16
B. Kueng, E. Mueggler, G. Gallego, and D. Scaramuzza.
Low-Latency Visual Odometry Using Event-Based Feature Tracks.
In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ
  International Conference on, 2016.

Lagorce14
X. Lagorce, C. Meyer, S.-H. Ieng, D. Filliat, and R. Benosman.
Asynchronous Event-Based Multikernel Algorithm for High-Speed Visual
  Features Tracking.
IEEE Transactions on Neural Networks and Learning Systems.

Lagorce17
X. Lagorce, G. Orchard, F. Galluppi, B. E. Shi, and R. B. Benosman.
Hots: A Hierarchy of Event-Based Time-Surfaces for Pattern
  Recognition.
PAMI, 2017.

Li16
H. Li, G. Li, and L. Shi.
Classification of Spatiotemporal Events Based on Random Forest.
In Advances in Brain Inspired Cognitive Systems: International
  Conference, 2016.

Lichtsteiner08
P. Lichtsteiner, C. Posch, and T. Delbruck.
A 128x128 120db 15us Latency Asynchronous Temporal Contrast Vision
  Sensor.
IEEE Journal of Solid State Circuits, 2008.

Linares11
B. Linares-Barranco, T. Serrano-Gotarredona, L. A. Camuñas-Mesa, J. A.
  Perez-Carrasco, C. Zamarreño-Ramos, and T. Masquelier.
On Spike-Timing-Dependent-Plasticity, Memristive Devices, and
  Building a Self-Learning Visual Cortex.
Frontiers in neuroscience, 2011.

Liu16
H. Liu, D. P. Moeys, G. Das, D. Neil, S.-C. Liu, and T. Delbrück.
Combined Frame-And Event-Based Detection and Tracking.
In Circuits and Systems (ISCAS), 2016 IEEE International
  Symposium on, 2016.

Liu18
W. Liu, H. Chen, R. Goel, Y. Huang, A. Veeraraghavan, and A. Patel.
Fast Retinomorphic Event-Driven Representations for Video
  Recognition and Reinforcement Learning.
In arXiv, 2018.

Maqueda18
A. I. Maqueda, A. Loquercio, G. Gallego, N. Garcıa, and D. Scaramuzza.
Event-Based Vision Meets Deep Learning on Steering Prediction for
  Self-Driving Cars.
In CVPR, 2018.

Marti16
D. Martí, M. Rigotti, M. Seok, and S. Fusi.
Energy-Efficient Neuromorphic Classifiers.
Neural computation, 2016.

Masquelier07
T. Masquelier and S. J. Thorpe.
Unsupervised Learning of Visual Features through Spike Timing
  Dependent Plasticity.
PLoS computational biology, 2007.

Mueggler17
E. Mueggler, C. Bartolozzi, and D. Scaramuzza.
Fast Event-Based Corner Detection.
In BMVC, 2017.

Mueggler17b
E. Mueggler, H. Rebecq, G. Gallego, T. Delbruck, and D. Scaramuzza.
The Event-Camera Dataset and Simulator: Event-Based Data for Pose
  Estimation, Visual Odometry, and SLAM.
The International Journal of Robotics Research, 2017.

Neil16
D. Neil, M. Pfeiffer, and S.-C. Liu.
Phased LSTM: Accelerating Recurrent Network Training for Long or
  Event-Based Sequences.
In NIPS, 2016.

Ni12
Z. Ni, A. Bolopion, J. Agnus, R. Benosman, and S. Regnier.
Asynchronous Event-Based Visual Shape Tracking for Stable Haptic
  Feedback in Microrobotics.
IEEE Transactions on Robotics, 2012.

Ni15
Z. Ni, S.-H. Ieng, C. Posch, S. Régnier, and R. Benosman.
Visual Tracking Using Neuromorphic Asynchronous Event-Based
  Cameras.
Neural computation, 2015.

Oconnor13
P. O'Connor, D. Neil, S.-C. Liu, T. Delbruck, and M. Pfeiffer.
Real-Time Classification and Sensor Fusion with a Spiking Deep
  Belief Network.
Frontiers in neuroscience, 2013.

Peng17
X. Peng, B. Zhao, R. Yan, H. Tang, and Z. Yi.
Bag of Events: An Efficient Probability-Based Feature Extraction
  Method for AER Image Sensors.
IEEE transactions on neural networks and learning systems,
  2017.

Posch11
C. Posch, D. Matolin, and R. Wohlgenannt.
A QVGA 143 dB Dynamic Range Frame-Free PWM Image Sensor with
  Lossless Pixel-Level Video Compression and Time-Domain CDS.
Solid-State Circuits, IEEE Journal of, 2011.

Ramesh17
B. Ramesh, H. Yang, G. Orchard, N. A. L. Thi, and C. Xiang.
DART: Distribution Aware Retinal Transform for Event-Based Cameras.
In arXiv, 2017.

Rebecq17
H. Rebecq, T. Horstschaefer, G. Gallego, and D. Scaramuzza.
EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and
  Mapping in Real Time.
IEEE Robotics and Automation Letters, 2017.

Rosten05
E. Rosten and T. Drummond.
Fusing points and lines for high performance tracking.
In ICCV. IEEE, 2005.

Rosten06
E. Rosten and T. Drummond.
Machine Learning for High-Speed Corner Detection.
In ECCV, 2006.

Russell10
A. Russell, G. Orchard, Y. Dong, Ş. Mihalas, E. Niebur, J. Tapson, and
  R. Etienne-Cummings.
Optimization Methods for Spiking Neurons and Networks.
IEEE transactions on neural networks, 2010.

Caviar09
R. Serrano-Gotarredona, M. Oster, P. Lichtsteiner, A. Linares-Barranco,
  R. Paz-Vicente, F. Gómez-Rodríguez, L. Camuñas-Mesa, R. Berner,
  M. Rivas-Pérez, T. Delbruck, et al.
CAVIAR: A 45k Neuron, 5M Synapse, 12G Connects/s AER Hardware
  Sensory–Processing–Learning–Actuating System for High-Speed Visual Object
  Recognition and Tracking.
IEEE Transactions on Neural Networks, 2009.

Serrano13
T. Serrano-Gotarredona and B. Linares-Barranco.
A 128 X 128 1.5% Contrast Sensitivity 0.9% FPN 3 μs Latency 4
  mW Asynchronous Frame-Free Dynamic Vision Sensor Using Transimpedance
  Preamplifiers.
Solid-State Circuits, IEEE Journal of, 2013.

Sheik13
S. Sheik, M. Pfeiffer, F. Stefanini, and G. Indiveri.
Spatio-Temporal Spike Pattern Classification in Neuromorphic
  Systems.
In Biomimetic and Biohybrid Systems. Springer, 2013.

Sironi18
A. Sironi, M. Brambilla, N. Bourdis, X. Lagorce, and R. Benosman.
HATS: Histograms of Averaged Time Surfaces for Robust Event-Based
  Object Classification.
In CVPR, 2018.

Sochman05
J. Sochman and J. Matas.
Waldboost-learning for time constrained sequential detection.
In CVPR. IEEE, 2005.

Sochman09
J. Šochman and J. Matas.
Learning fast emulators of binary decision processes.
International Journal of Computer Vision, 2009.

Son17
B. Son, Y. Suh, S. Kim, H. Jung, J.-S. Kim, C. Shin, K. Park, K. Lee, J. Park,
  J. Woo, et al.
4.1 a 640× 480 Dynamic Vision Sensor with a 9μm Pixel and
  300meps Address-Event Representation.
In Solid-State Circuits Conference (ISSCC), 2017 IEEE
  International, 2017.

Strecha09
C. Strecha, A. Lindner, K. Ali, and P. Fua.
Training for task specific keypoint detection.
In Joint Pattern Recognition Symposium. Springer, 2009.

Tedaldi16
D. Tedaldi, G. Gallego, E. Mueggler, and D. Scaramuzza.
Feature Detection and Tracking with the Dynamic and Active-Pixel
  Vision Sensor (Davis).
In Event-based Control, Communication, and Signal Processing
  (EBCCSP), 2016 Second International Conference on, pages 1–7, 2016.

Vasco16
V. Vasco, A. Glover, and C. Bartolozzi.
Fast Event-Based Harris Corner Detection Exploiting the Advantages
  of Event-Driven Cameras.
In Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ
  International Conference on, 2016.

Verdie15
Y. Verdie, K. Yi, P. Fua, and V. Lepetit.
Tilde: A temporally invariant learned detector.
In Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition, 2015.

Yi16
K. M. Yi, E. Trulls, V. Lepetit, and P. Fua.
Lift: Learned invariant feature transform.
In European Conference on Computer Vision. Springer, 2016.

Zhu17
A. Z. Zhu, N. Atanasov, and K. Daniilidis.
Event-Based Feature Tracking with Probabilistic Data Association.
In Robotics and Automation (ICRA), 2017 IEEE International
  Conference on, pages 4465–4470, 2017.

Zhu18
A. Z. Zhu, L. Yuan, K. Chaney, and K. Daniilidis.
Ev-Flownet: Self-Supervised Optical Flow Estimation for Event-Based
  Cameras.
In arXiv, 2018.










§ APPENDIX


	

