SSTFormer: Bridging Spiking Neural Network and Memory Support
  Transformer for Frame-Event based Recognition
Event camera-based pattern recognition is a newly arising research topic in
recent years. Current researchers usually transform the event streams into
images, graphs, or voxels, and adopt deep neural networks for event-based
classification. Although good performance can be achieved on simple event
recognition datasets, however, their results may be still limited due to the
following two issues. Firstly, they adopt spatial sparse event streams for
recognition only, which may fail to capture the color and detailed texture
information well. Secondly, they adopt either Spiking Neural Networks (SNN) for
energy-efficient recognition with suboptimal results, or Artificial Neural
Networks (ANN) for energy-intensive, high-performance recognition. However,
seldom of them consider achieving a balance between these two aspects. In this
paper, we formally propose to recognize patterns by fusing RGB frames and event
streams simultaneously and propose a new RGB frame-event recognition framework
to address the aforementioned issues. The proposed method contains four main
modules, i.e., memory support Transformer network for RGB frame encoding,
spiking neural network for raw event stream encoding, multi-modal bottleneck
fusion module for RGB-Event feature aggregation, and prediction head. Due to
the scarce of RGB-Event based classification dataset, we also propose a
large-scale PokerEvent dataset which contains 114 classes, and 27102
frame-event pairs recorded using a DVS346 event camera. Extensive experiments
on two RGB-Event based classification datasets fully validated the
effectiveness of our proposed framework. We hope this work will boost the
development of pattern recognition by fusing RGB frames and event streams. Both
our dataset and source code of this work will be released at
https://github.com/Event-AHU/SSTFormer.
Xiao Wang, Zongzhen Wu, Yao Rong, Lin Zhu, Bo Jiang, Jin Tang, Yonghong Tian
SSTFormer: Bridging Spiking Neural Network and Memory Support Transformer for Frame-Event based Recognition

###### Abstract

Event camera-based pattern recognition is a newly arising research topic in recent years. Current researchers usually transform the event streams into images, graphs, or voxels, and adopt deep neural networks for event-based classification. Although good performance can be achieved on simple event recognition datasets, however, their results may be still limited due to the following two issues. Firstly, they adopt spatial sparse event streams for recognition only, which may fail to capture the color and detailed texture information well. Secondly, they adopt either Spiking Neural Networks (SNN) for energy-efficient recognition with suboptimal results, or Artificial Neural Networks (ANN) for energy-intensive, high-performance recognition. However, seldom of them consider achieving a balance between these two aspects. In this paper, we formally propose to recognize patterns by fusing RGB frames and event streams simultaneously and propose a new RGB frame-event recognition framework to address the aforementioned issues. The proposed method contains four main modules, i.e., memory support Transformer network for RGB frame encoding, spiking neural network for raw event stream encoding, multi-modal bottleneck fusion module for RGB-Event feature aggregation, and prediction head. Due to the scarce of RGB-Event based classification dataset, we also propose a large-scale PokerEvent dataset which contains 114 classes, and 27102 frame-event pairs recorded using a DVS346 event camera. Extensive experiments on two RGB-Event based classification datasets fully validated the effectiveness of our proposed framework. We hope this work will boost the development of pattern recognition by fusing RGB frames and event streams. Both our dataset and source code of this work will be released at [https://github.com/Event-AHU/SSTFormer](https://github.com/Event-AHU/SSTFormer).

 Spiking Neural Networks, Transformer Networks, Bottleneck Mechanism, Video Classification, Bio-inspired Computing

## I Introduction

The mainstream video-based classification algorithms are widely developed based on RGB cameras. With the help of deep learning, many representative deep models are proposed, such as TSM [1], C3D [2], Video-SwinTrans [3], and SlowFast [4], etc. These works learn the deep feature representation well and contribute significantly to many practical applications and video-related tasks. However, these models may perform poorly in some extremely challenging scenarios (for example, fast motion, low illumination, and over-exposure) due to the utilization of RGB cameras. This is because RGB cameras adopt a global synchronous exposure mechanism and have a limited frame rate.

Recently, biologically inspired event cameras (also called DVS, Dynamic Vision Sensor) have attracted great interest due to their unique imaging principle and key characteristics including _high dynamic range, high temporal resolution, low latency, low power,_ etc. Different from RGB cameras which record the light intensity of a scene in a synchronous way, each pixel in event cameras outputs an event spike only when the variation of light intensity exceeds a certain threshold. The event that corresponds to a decrease in brightness is termed _ON_ event, otherwise, _OFF_ event. Usually, each event can be represented as a quadruple \(\{x,y,t,p\}\), where \(x,y\) denotes the spatial coordinates, \(t\) is the timestamp, and \(p\) is the polarity (1 and -1 are used to denote the _ON_ and _OFF_ event, respectively). Therefore, event cameras have been applied in many computer vision tasks, such as object detection, visual tracking [5, 6], pose estimation, etc.

There are some works, such as Graph Neural Networks (GNN) [7, 8], Convolutional Neural Networks (CNN) [9], and Spiking Neural Networks (SNN) [10, 11] which have been proposed for pattern recognition of event cameras. For example, Asynchronous Event-based Graph Neural Network (short for AEGNN [8]) is proposed to process events as spatio-temporal graphs in an incremental learning manner. Zhou et al. combine the spiking neurons with Transformer network and

Fig. 1: Illustration of video classification by fusing RGB frames and event stream. The bottom event images are stacked for visualization.