SpikePoint: An Efficient Point-based Spiking Neural Network for Event
  Cameras Action Recognition
Event cameras are bio-inspired sensors that respond to local changes in light
intensity and feature low latency, high energy efficiency, and high dynamic
range. Meanwhile, Spiking Neural Networks (SNNs) have gained significant
attention due to their remarkable efficiency and fault tolerance. By
synergistically harnessing the energy efficiency inherent in event cameras and
the spike-based processing capabilities of SNNs, their integration could enable
ultra-low-power application scenarios, such as action recognition tasks.
However, existing approaches often entail converting asynchronous events into
conventional frames, leading to additional data mapping efforts and a loss of
sparsity, contradicting the design concept of SNNs and event cameras. To
address this challenge, we propose SpikePoint, a novel end-to-end point-based
SNN architecture. SpikePoint excels at processing sparse event cloud data,
effectively extracting both global and local features through a singular-stage
structure. Leveraging the surrogate training method, SpikePoint achieves high
accuracy with few parameters and maintains low power consumption, specifically
employing the identity mapping feature extractor on diverse datasets.
SpikePoint achieves state-of-the-art (SOTA) performance on four event-based
action recognition datasets using only 16 timesteps, surpassing other SNN
methods. Moreover, it also achieves SOTA performance across all methods on
three datasets, utilizing approximately 0.3\% of the parameters and 0.5\% of
power consumption employed by artificial neural networks (ANNs). These results
emphasize the significance of Point Cloud and pave the way for many
ultra-low-power event-based data processing applications.
Hongwei Ren, Yue Zhou, Yulong Huang, Haotian Fu, Xiaopeng Lin, Jie Song, Bojun Cheng
# SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition

###### Abstract

Event cameras are bio-inspired sensors that respond to local changes in light intensity and feature low latency, high energy efficiency, and high dynamic range. Meanwhile, Spiking Neural Networks (SNNs) have gained significant attention due to their remarkable efficiency and fault tolerance. By synergistically harnessing the energy efficiency inherent in event cameras and the spike-based processing capabilities of SNNs, their integration could enable ultra-low-power application scenarios, such as action recognition tasks. However, existing approaches often entail converting asynchronous events into conventional frames, leading to additional data mapping efforts and a loss of sparsity, contradicting the design concept of SNNs and event cameras. To address this challenge, we propose SpikePoint, a novel end-to-end point-based SNN architecture. SpikePoint excels at processing sparse event cloud data, effectively extracting both global and local features through a singular-stage structure. Leveraging the surrogate training method, SpikePoint achieves high accuracy with few parameters and maintains low power consumption, specifically employing the identity mapping feature extractor on diverse datasets. SpikePoint achieves state-of-the-art (SOTA) performance on four event-based action recognition datasets using only 16 timesteps, surpassing other SNN methods. Moreover, it also achieves SOTA performance across all methods on three datasets, utilizing approximately 0.3% of the parameters and 0.5% of power consumption employed by artificial neural networks (ANNs). These results emphasize the significance of Point Cloud and pave the way for many ultra-low-power event-based data processing applications.

## 1 Introduction

Event cameras are a recent development in computer vision that is revolutionizing how visual information is captured and processed (Gallego et al., 2020). They are particularly well-suited for detecting fast moving objects, as they can eliminate redundant information and significantly reduce memory usage and data processing requirements. This is achieved through innovative pixel design resulting in a sparse data output that is more efficient than traditional cameras (Posch et al., 2010; Son et al., 2017). However, most event data processing algorithms rely on complex and deep ANNs, which are not aligned with the low power consumption benefits of event cameras. Instead, combining event-based vision tasks with SNNs has shown great potential thanks to their highly compatible properties, especially in tasks such as action recognition(Liu et al., 2021).

Spiking Neural Networks have emerged as a promising alternative that can address the limitations of traditional neural networks for their remarkable biological plausibility, event-driven processing paradigm, and exceptional energy efficiency (Gerstner and Kistler, 2002). The network's asynchronous operations depend on bionic neurons, communicating information via precisely timed discrete spikes (Neftci et al., 2019). The event-driven processing paradigm enables sparse but potent computing capabilities, where a neuron activates only when it receives or generates a spike (Hu et al., 2021). This property gives the network remarkably high energy efficiency and makes it an ideal candidate for processing event-based data. Nevertheless, existing approaches for combining
event cameras with SNN require the conversion of asynchronous events into conventional frames for downstream processing, resulting in additional data mapping work and a loss of sparsity (Kang et al., 2020)(Berlin and John, 2020). Moreover, this process leads to the loss of detailed temporal information, which is critical for accurate action recognition (Innocenti et al., 2021). Therefore, developing SNN-compatible novel techniques that can operate directly on event data remains a challenge.

Point Cloud is a powerful representation of 3D geometry that encodes spatial information in the form of a sparse set of points and eliminates the need for the computational image or voxel conversion, making it an efficient and ideal choice for representing event data (Qi et al., 2017a). The sparse and asynchronous event data could be realized as a compact and informative 3D space-time representation of the scene, bearing a resemblance to the concept of Point Cloud (Sekikawa et al., 2019). Still, Point Cloud networks need frequent high-data dimension transformations and complex feature extraction operators in ANNs. These may not function optimally in SNNs due to their binarization and dynamic characteristics.

In this paper, we introduce SpikePoint, an end-to-end framework harnessing SNN for the effective and efficient processing of event data in vision-based tasks. Our contributions are as follows: First, we combine event-based vision tasks with SNN by treating the input as Point Clouds rather than stacked event frames to preserve the fine-grained temporal feature and retain the sparsity of raw events. Second, unlike ANN counterparts with a multi-stage hierarchical structure, we design a singular-stage feature extractor structure to harmoniously extract local and global features. This lightweight design achieves effective performance through the back-propagation training method. Lastly, we introduce a pioneering encoding approach to address relative position data containing negative values. This scheme maintains symmetry between positive and negative values, optimizing information representation. We evaluate SpikePoint on diverse event-based action recognition datasets of varying scales, achieving SOTA results on Daily DVS (Liu et al., 2021), DVS ACTION (Miao et al., 2019), and HMDB51-DVS (Bi et al., 2020) datasets, surpassing even traditional ANNs. Additionally, we attain the SNN's SOTA on the DVS128 Gesture dataset (Amir et al., 2017). Notably, our evaluation encompasses an assessment of network power consumption. In comparison to both SNNs and ANNs with competitive accuracy, our framework consistently exhibits exceptional energy efficiency, both in dynamic and static power consumption, reaffirming the unequivocal superiority of our network.

## 2 Related Work

### Event-based action recognition

Action recognition is a critical task with diverse applications in fields such as anomaly detection, entertainment, and security. Two primary methods for event-based action recognition are ANN and SNN (Ren et al., 2023). The ANN approaches have yielded several notable contributions, including IBM's pioneering end-to-end gesture recognition system (Amir et al., 2017) and Cannici's asynchronous event-based full convolutional networks (Cannici et al., 2019), while Chadha et al. developed a promising multimodal transfer learning framework for heterogeneous environments (Chadha et al., 2019). Additionally, Bin Yin et al. proposed a graph-based spatiotemporal feature learning framework and introduced several new datasets (Bi et al., 2020), including HMDB51-DVS. On the other hand, SNN methods have also demonstrated great potential, with Liu et al. presenting a successful model for object classification using address event representation (Liu et al., 2020) and George et al. (George et al., 2020) using multiple convolutional layers and a reservoir to extract spatial and temporal features, respectively. Liu et al. (Liu et al., 2021) have further advanced the field by extracting motion information from asynchronous discrete events captured by event cameras. While these SNNs have improved in terms of efficiency, however, their accuracy still falls short when compared to ANN-based approaches.

### Point Cloud networks in ANN

Point-based methods have revolutionized the direct processing of Point Cloud data as input, with PointNet (Qi et al., 2017a) standing out as a remarkable example. PointNet++ (Qi et al., 2017b) took it a step further by introducing a set abstraction module. While it used a simple MLP in the feature extractor, numerous more advanced feature extractors have recently been developed to
elevate the quality of Point Cloud processing (Wu et al., 2019; Zhao et al., 2021; Ma et al., 2021; Dosovitskiy et al., 2020). To apply these methods to the event stream, Wang et al. (Wang et al., 2019) first tackled the temporal information processing challenge while preserving representation in both the x and y axes, achieving gesture recognition using PointNet++. PAT (Yang et al., 2019) further improved on this model by incorporating self-attention and Gumbel subset sampling, achieving even better performance in the recognition task. Nonetheless, the current performance of the point-based models still cannot compete with frame-based methods in terms of accuracy. Here, we propose SpikePoint as a solution that fully leverages the characteristics of event clouds while maintaining high accuracy, low parameter numbers, and low power consumption.

## 3 SpikePoint

### Event cloud

Event streams are time-series data that record spatial intensity changes in chronological order. Each event can be represented by \(e_{m}=(x_{m},y_{m},t_{m},p_{m})\), where \(m\) represents the event number, \(x_{m}\) and \(y_{m}\) denote the spatial coordinates of the event, \(t_{m}\) indicates the timestamp of the event, and \(p_{m}\) denotes the polarity of the event. To facilitate the effective processing of action recognition data, it is common practice to divide a sample \(AR_{raw}\) into some sliding windows \(AR_{dip}\) by the next formula.

\[AR_{clip}=clip_{i}\left\{e_{k\longrightarrow t}\right\}\mid i\in(1,n_{win}) \mid t_{l}-t_{k}=L \tag{1}\]

where \(L\) is the length of the sliding window, \(k\) and \(l\) represent the start and end event's number of the \(i_{th}\) sliding window, and \(n_{win}\) is the number of the sliding window. To apply the Point Cloud method, four-dimensional events in a \(clip\) have to be normalized and converted into a three-dimensional spacetime event \(AR_{point}\). A straight way to do this is to convert \(t_{m}\) into \(z_{m}\) and ignore \(p_{m}\):

\[AR_{point}=\left\{e_{m}=(x_{m},y_{m},z_{m})\mid m=k,k+1,\ldots,l\right\} \tag{2}\]

With \(z_{m}=\frac{t_{m}-t_{k}}{t_{l}-t_{k}}\mid m\in(k,l)\), and \(x_{m}\),\(y_{m}\) are also normalized between \([0,1]\). After pre-processing, the event cloud is converted into the pseudo-Point Cloud, which comprises explicit spatial information \((x,y)\) and implicit temporal information \(t\). Through pseudo-Point Cloud, SpikePoint is capable of learning spatio-temporal features which is crucial for action recognition.

### Sampling and Grouping

To unify the number of inputs fed into the network, we utilize random sampling \(AR_{point}\) to construct the trainset and testset. Then, we group these points \(PN\) by the Farthest Point Sampling (\(FPS\)) method and the \(K\) Nearest Neighbor (\(KNN\)) algorithm. \(FPS\) is responsible for finding the

Figure 1: The overall architecture of SpikePoint. The raw event cloud is segmented by the sliding window. Then, the global Point Cloud is transformed into \(M\) groups by grouping and sampling. The coordinate is converted into spikes by rate coding, and the results of action recognition are obtained by the local feature extractor, global feature extractor, and classifier in turn.
\(Centroid\) of each group, while \(KNN\) is utilized to identify \(N^{{}^{\prime}}\) group members. This process can be abstracted as follows :

\[Centroid=FPS(PN)\quad\mathcal{G}=KNN(PN,Centroid,N^{{}^{\prime}}) \tag{3}\]

The grouping of \(PN\) into \(\mathcal{G}\) results in a transformation of the data dimension from [\(N\), 3] to [\(N^{{}^{\prime}}\), \(M\), 3]. \(N\) is the number of input points, \(M\) is the number of groups, and \(N^{{}^{\prime}}\) is the number of Point Cloud in each group. Given that the coordinates \([x,y,z]\) among the points within each group exhibit similarity and lack distinctiveness, this situation is particularly severe for SNN rate encoding. The standardization method employed to determine the relative position of the point information \([\Delta x,\Delta y,\Delta z]\) with respect to their \(Centroid\) is as follows:

\[[\Delta x,\Delta y,\Delta z]=\frac{\mathcal{G}-Centroid}{SD(\mathcal{G})}\sim N (0,1),\quad SD(\mathcal{G})=\sqrt{\frac{\sum_{i=1}^{n}(g_{i}-\bar{g})^{2}}{n-1 }}\quad g_{i}\in\mathcal{G} \tag{4}\]

Where \([\Delta x,\Delta y,\Delta z]\) adheres to the standard Gaussian distribution \(N(0,1)\), \(SD\) corresponds to the standard deviation of \(\mathcal{G}\) and \(g=[x_{0},y_{0},t_{0},\ldots,x_{n},y_{n},t_{n}]\). Ultimately, we concatenate the relative position \([\Delta x,\Delta y,\Delta z]\) and \(Centroid\)\([x_{c},y_{c},z_{c}]\) as the final grouping result.

After grouping, we rate encode the Point Cloud coordinates to meet the SNN network's binary input. It is worth mentioning that the reflected distance information \([\Delta x,\Delta y,\Delta z]\) yields positive and negative values, as shown in Fig. 2. While ANN can successfully handle such information due to their utilization of floating point operations, SNN employs rate coding and binarization, and can't process negative values. It is thus necessary to develop a method to effectively handle such information in SNN in order to achieve accurate results.

A straightforward approach is to normalize \([\Delta x,\Delta y,\Delta z]\) to [0,1], but this can lead to **asymmetric information after passing through points equidistant from the centroid**, resulting in limited accuracy. The detailed comparison of accuracy is summarized in Table 8 and discussed later. Alternatively, we take the absolute value of the numerator of Eq. 4 to get the \([\Delta|x|,\Delta|y|,\Delta|z|]\) that can perform the spike transformation. However, after such processing, the direction of the relative distance information of the coordinates is lost and the distribution of the response data is changed from the standard normal distribution to the folded normal distribution. The probability density function is:

\[f(x;\mu,\delta^{2})=\frac{1}{\sqrt{2\pi\delta}}e^{-\frac{(x-\mu)^{2}}{2\delta^ {2}}}\rightarrow\frac{1}{\sqrt{2\pi\delta}}(e^{-\frac{(x-\mu)^{2}}{2\delta^{2 }}}+e^{-\frac{(x+\mu)^{2}}{2\delta^{2}}})(x\geq 0) \tag{5}\]

Where \(\mu\) and \(\delta\) are the mean and standard deviation of the Gaussian distribution, respectively. The expectation \(\dot{\mu}\) after the absolute value is given in Eq. 6. \(erf(z)=\frac{2}{\sqrt{\pi}}\int_{0}^{z}e^{t^{2}}dt\) is the error function.

\[\dot{\mu}=\sqrt{\frac{2}{\pi}}\delta e^{-\frac{x^{2}}{2\delta^{2}}}+\mu\cdot[1 -2\phi(-\frac{\mu}{\delta})],\quad\phi(x)=\frac{1}{2}[1+erf(\frac{x}{\sqrt{2}})] \tag{6}\]

By Eq. 4 - Eq. 6, We maintain codability and shift the expectation of those coordinates to a larger value of \(\sqrt{\frac{2}{\pi}}\) which is calculated in Appendix A.1. However, data distribution has also changed and needs to be compensated. To do so, we replace the input \([\Delta x,\Delta y,\Delta z,x_{c},y_{c},z_{c}]\) mentioned above with \([\Delta|x|,\Delta|y|,\Delta|z|,x_{min},y_{min},z_{min}]\), \([x_{min},y_{min},z_{min}]\) represents the smallest \(x,y,z\) values in a specific group. Hence, the increase of the first three input components is compensated by the decrease of the last three input components and the input remains balanced. It is worth noting that the \(Centroid\) in each sphere is important and \([x_{min},y_{min},z_{min}]\) is not a good indicator of the \(Centroid\), so we introduce a separate branch to extract the global information contained \(Centroid\) and do the fusion of these two features in the middle part of the network.

Figure 2: Visualization of our grouping method. (a) The different spatial positions of \([x_{c},y_{c},z_{c}]\), \([x_{min},y_{min},z_{min}]\) and \([x,y,z]\). (b) The transformation of the distribution after taking absolute.
After implementing the aforementioned modifications to the sampling and grouping module, we conducted an analysis that revealed a significant decrease in both the mean relative error (MRE) of rate coding and the coefficient of variation (CV) of the data. This reduction in MRE and CV provides a fundamental explanation for the efficacy of our proposed methodology as evidenced by the results. For the step-by-step derivation of the formulas and the validation of the dataset, please consult the Appendix A.2 and A.3.

### Singular stage structure

The SpikePoint model is unique in its utilization of a singular stage structure as shown in Fig. 1, in contrast to the hierarchical structure employed by all other ANN-based methods for Point Cloud networks. While this hierarchical paradigm has become the standard design approach for ANNs, it is not readily applicable to SNNs because spike-based features tend to become sparse and indistinguishable as the depth of the stage increases and the training method based on backpropagation has a serious gradient problem. In light of this, we develop **a novel, streamlined network architecture that effectively incorporates the properties of SNNs**, resulting in a simple yet highly efficient model that is adept at abstract local and global geometry features. The specific dimensional changes and SpikePoint's algorithms can be referred to in Appendix A.6 and Algorithm A.6.

#### 3.3.1 Basic Unit

Next, the basic unit of the feature extractor will be introduced in the form of a formula. The first is a discrete representation of LIF neurons. We abstract the spike into a mathematical equation as follows:

\[S_{j}(t)=\sum_{s\in C_{j}}\gamma(t-s),\quad\gamma(x)=\theta(U(t,x)-V_{th}) \tag{7}\]

where \(S\) represents the input or output spike, \(C\) represents the set of moments when the spike is emitted, \(j\) is the \(j^{th}\) input of the current neuron, \(\gamma\) represents the spike function, \(\theta\) represents the Heaviside step function, \(V_{th}\) denotes the threshold of neuron's membrane potential and \(\gamma=1\) if \(U(t,x)-V_{th}\geq 0\) else \(\gamma=0\). We proceed to define the process of synaptic summation in the SNN using the following equation:

\[I[n]=e^{-\frac{\Delta t}{r_{syn}}}I[n-1]+\sum_{j}W_{j}S_{j}[n] \tag{8}\]

The aforementioned equation delineates the mechanism of neuronal synaptic summation, where the current neuron's input is denoted as \(I\), \(\tau_{syn}\) conforms to the synapse time constant, \(\Delta t\) represents the simulation timestep, \(n\) represents the discrete timestep, while \(j\) refers to the antecedent neuron number.

\[U[n+1]=e^{-\frac{\Delta t}{r_{mem}}}U[n]+I[n]-S[n] \tag{9}\]

where the variable \(U\) denotes the membrane potential of the neuron, \(n\) represents discrete timesteps, while \(\tau_{mem}\) is the membrane time constant. Moreover, \(S\) signifies the membrane potential resetting subsequent to the current neuron transmitting the spike.

SNN is prone to gradient explosion and vanishing during training via backpropagation through time (BPTT), owing to the unique properties of their neurons as described above. Drawing on the utilization of conventional regularization techniques, such as dropout and batch normalization, we endeavor to incorporate a residual module to address the issues of overfitting and detrimental training outcomes. We do identity mapping by changing the residual module to the following equation in SNN refer (Hu et al., 2021; Fang et al., 2021; Feng et al., 2022). And the coefficient \(\sigma^{{}^{\prime}}(I_{i}^{l+m-1}+S_{j}^{l})\) in Eq. 29 of error propagation of the corresponding residual term is canceled. A detailed derivation can be found in Appendix A.4, which describes how this connection solves the problem of backpropagation.

\[S^{l}=LIF(I+S^{l-1})\longrightarrow LIF(I)+S^{l-1} \tag{10}\]

We have defined two basic units, \(ResFB\) and \(ResF\), with and without a bottleneck, respectively, as depicted in Fig. 1, and have incorporated them into the overall architecture. Due to the specificity of Point Cloud networks, the operation of feature dimensioning usually uses one-dimensional convolution. To maintain the advantage of the residual module, we do not do anything to the features of the residual connection. As a result, the module's input and output dimensions remain the same.
#### 3.3.2 Local feature Extrator

The local feature extractor plays a crucial role in abstracting features within the Point Cloud group, similar to convolutional neural networks with fewer receptive fields. As each point is processed, it is imperative to adhere to the principle of minimizing the depth and width of the extractor to ensure the efficiency of the SpikePoint. To this end, we have employed the \(ResFB\) unit with a bottleneck to streamline the network design as the following equations, result in more advanced extracted features.

\[X_{1}=[\Delta|x|,\Delta|y|,\Delta|z|,x_{min},y_{min},z_{min}],X_{2}=[x_{c},y_{ c},z_{c}] \tag{11}\]

\[F_{l1}=ResFB(Conv1D(X_{1})) \tag{12}\]

\[F_{l2}=ResFB(Conv1D(X_{2})) \tag{13}\]

\[F_{local}=MaxPool(F_{l1})+F_{l2} \tag{14}\]

We evaluate both the concat and add operations for the two-channel feature fusion in the ablation experiments section.

#### 3.3.3 Global feature Extrator

The local feature extractor aggregates point-to-point features within each group into intra-group feature vectors, while the global feature extractor further abstracts the relationships between groups into inter-group feature tensors. The input's dimension for the final classifier is intimately linked with the width of the global feature extractor. Thus, it is crucial to enable the feature extractor to expand its width as much as possible while utilizing a limited depth, and simultaneously ensuring the extracted features are of high-level abstraction. The feature extractor can be formulated as follows:

\[L(x)=ResF(Conv1D(x)) \tag{15}\]

\[F_{m}=L_{2}(L_{1}(F_{local})) \tag{16}\]

\[F_{global}=MaxPool(Conv1D(F_{m})) \tag{17}\]

The final extracted features, denoted as \(F_{global}\), will be transmitted to the classifier A.7.3 for action recognition. To accommodate small and large datasets, we utilized two distinct ascending dimensionality scales while ensuring consistency in our architecture where specific feature dimensions are illustrated through Appendix Fig. 8.

## 4 Experiment

### Dataset

We evaluate SpikePoint on four event-based action recognition datasets of different scales, and more details about the dataset are presented in Appendix A.5. These datasets have practical applications and are valuable for research in neuromorphic computing.

### Preprocessing

The time set for sliding windows is 0.5 s, 1 s, and 1.5 s, as shown in Table 1. The coincidence area of adjacent windows is set as 0.25 s in DVS128 Gesture and DVS Action datasets, and 0.5 s in other datasets. The testset is 20% randomly selected from the total samples.

\begin{table}
\begin{tabular}{c c c c} \hline \hline DataSet & Classes & Sensor & Avg length \\ \hline DVS128 Gesture (Amir et al., 2017) & 10711 & DAVIS128 & 6.52 s \\ Daily DVS (Liu et al., 2021) & 12 & DAVIS128 & 3 s \\ DVS Action (Miao et al., 2019) & 10 & DAVIS346 & 5 s \\ HMDBS1-DVS (Bi et al., 2020) & 51 & DAVIS240 & 8 s \\ \hline Train & Test & Sliding Window & Overlap \\ \hline
26796 & 66959 & 0.5 s & 0.25 s \\
2916 & 720 & 1.5 s & 0.5 s \\
912 & 116 & 0.5 s & 0.25 s \\
30720 & 3840 & 0.5 s & 0.5 s \\ \hline \hline \end{tabular}
\end{table}
Table 1: Specific information on the four datasets.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Name & Method & Param & Acc \\ \hline TBR+3D (Innocenti et al., 2021) & ANN & 12.25 M & 99.6\% \\ PointNet+ (Qi et al., 2017b) & ANN & 1.48 M & 95.3\% \\ EV-VOCN (Deng et al., 2021) & ANN & 0.82 M & 95.7\% \\ VVM-GCN (Xie et al., 2022) & ANN & 0.86 M & 97.5\% \\ \hline SEW-ResNet (Fang et al., 2021) & SNN & - & 97.9\% \\ Deep SNN(16 layers) (Amir et al., 2017) & SNN & - & 91.8\% \\ Deep SNN (Isevers) (Sheeath \& Orchard, 2018) & SNN & - & 93.6\% \\ Conv-RNN SNN(Isevers)(Xing et al., 2020) & SNN & - & 92.0\% \\ Conv+Reservoir SNN (George et al., 2020) & SNN & - & 65.0\% \\ HMAX-based SNN (Liu et al., 2020) & SNN & - & 70.1\% \\ Motion-based SNN (Liu et al., 2021) & SNN & - & 92.7\% \\ \hline
**SpikePoint** & **SNN** & **0.58** M & **98.74\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: SpikePoint’s performance on DVS128 Gesture.
### Network Structure

In this paper, a relatively small version network is used for the Daily DVS and DVS Action, and a larger version network is used for the other two datasets. It should be emphasized that the architectures are identical, with only a shift in dimensionality in the feature extraction process as presented in Appendix A.7.

### Results

**DVS128 Gesture:** The DVS128 Gesture dataset is extensively evaluated by numerous algorithms, serving as a benchmark for assessing their efficiency and effectiveness. Compared to the other SNN methods, SpikePoint achieves SOTA results with an accuracy of 98.74%, as shown in Table 2. Comparing the ANN methods, our SpikePoint model demonstrates superior accuracy compared to lightweight ANN such as EV-VGCNN and VMV-GCN, despite the latter employing a larger number of parameters. This indicates that Point Clouds' rich and informative nature may confer an advantage over voxels in conveying complex and meaningful information. The current SOTA employed by ANNs utilizes nearly 21 times more parameters than SpikePoint, resulting in a 0.86% increase in performance. Additionally, it should be added that SpikePoint divides a test event stream into many test subsets through the sliding window. During the training process, the accuracy of the SpikePoint on the test subsets is 97.1%.

The results of a test stream are obtained by counting and voting on the test subsets of a sequence.

**Daily DVS:** SpikePoint achieves SOTA on this dataset with 97.92% accuracy on the testset, and outperforms all ANN and SNN networks. Notably, it uses only 0.3% of its parameters compared to the high-performing ANN network, as shown in Table 3. Upon visualization of the dataset, we find that it has relatively low noise, and the number of events is evenly distributed in the time domain.

The action description is well captured by the 1024 points sampled using random sampling. With this in mind, we conduct the ablation study on this dataset, including the processing of negative values (absolute or [0,1]), the utilization of dual channel inputs, and the techniques employed for feature fusion, which are discussed in detail in the Ablation study.

**DVS Action:** SpikePoint also obtains SOTA results on this dataset, as shown in Table 4. In the same way, we visualize the addat files in the dataset but find that most of the data have heavy background noise, causing us to collect a lot of noise during random sampling. So we preprocess the dataset and finally achieve 90.6% accuracy. This reflects a problem random sampling will not pick up useful points when there is too much background noise. More reasonable sampling

**HMDB51-DVS:** Compared with the first three datasets which have relatively few categories, HMDB51-DVS has 51 different categories. Our experiments demonstrate that SpikePoint excels not only on small-scale datasets but also demonstrates impressive adaptability to larger ones. As shown in Table 5, SpikePoint outperforms all ANN methods, despite using very few parameters. This indicates that SpikePoint has excellent generalization capabilities. It should be added that the SpikePoint used for DVS Gesture and HMDB51-DVS is both the larger model and the number of parameter differences are only due

\begin{table}
\begin{tabular}{c c c c} \hline Name & Method & Param & Acc \\ \hline ST-EVNet (Wang et al., 2020) & ANN & 1.6 M & 88.7\% \\ PointNet (Qi et al., 2017a) & ANN & 3.46 M & 75.1\% \\ Deep SNN (Iéyes) with (Li et al., 2019) & SNN & - & 71.2\% \\ HMAX-based SNN (Xiao et al., 2019) & SNN & - & 55.0\% \\ Motion-based SNN (Liu et al., 2021a) & SNN & - & 78.1\% \\ \hline
**SpikePoint** & **SNN** & **0.16 M** & **90.6\%** \\ \hline \end{tabular}
\end{table}
Table 4: Model’s performance on DVS ACTION.

\begin{table}
\begin{tabular}{c c c c} \hline Name & Method & Param & Acc \\ \hline T3D(Carreira and Zisserman, 2017) & ANN & 49.19 M & 96.2\% \\ TAN(Liu et al., 2021b) & ANN & 24.8 M & 96.5\% \\ VMV-GCN (Xie et al., 2022) & ANN & 0.84 M & 94.1\% \\ TimeSformer (Bertsins et al., 2021) & ANN & 121.27 M & 90.6\% \\ HMAX-based SNN (Xiao et al., 2019) & SNN & - & 68.3\% \\ HMAX-based SNN (Liu et al., 2020) & SNN & - & 76.9\% \\ Motion-based SNN (Liu et al., 2021a) & SNN & - & 90.3\% \\ \hline
**SpikePoint** & **SNN** & **0.16 M** & **97.92\%** \\ \hline \end{tabular}
\end{table}
Table 3: SpikePoint’s performance on Daily DVS.

\begin{table}
\begin{tabular}{c c c} \hline Name & Method & Param & Acc \\ \hline T3D(Carreira and Zisserman, 2017) & ANN & 49.19 M & 96.2\% \\ TAN(Liu et al., 2021b) & ANN & 24.8 M & 96.5\% \\ VMV-GCN (Xie et al., 2022) & ANN & 0.84 M & 94.1\% \\ TimeSformer (Bertsins et al., 2021) & ANN & 121.27 M & 90.6\% \\ HMAX-based SNN (Xiao et al., 2019) & SNN & - & 68.3\% \\ HMAX-based SNN (Liu et al., 2020) & SNN & - & 76.9\% \\ Motion-based SNN (Liu et al., 2021a) & SNN & - & 90.3\% \\ \hline
**SpikePoint** & **SNN** & **0.16 M** & **97.92\%** \\ \hline \end{tabular}
\end{table}
Table 3: SpikePoint’s performance on Daily DVS.
to the different categories classified by the final classifier. As the final output layer is a voting layer, the difference in parameter quantities is 0.21 M.

### Power Consumption

We assume that multiply-and-accumulate (MAC) and accumulate (AC) operations are implemented on the 45 nm technology node with \(V_{DD}=0.9~{}V\)(Horowitz, 2014), where \(E_{MAC}=4.6~{}pJ\) and \(E_{AC}=0.9~{}pJ\). We calculate the number of synaptic operations (SOP) of the spike before calculating the theoretical dynamic energy consumption by \(SOP=firerate\times T\times FLOPs\), where \(T\) is the timesteps, and FLOPs is float point operations per sample. OPs in Table 6 refer to SOPs in SNNs and FLOPs in ANNs. The dynamic energy is calculated by \(Dynamic=OPs\times E_{MAC\,or\,AC}\) and the static energy is calculated by \(Static=Para.\times spp\times L_{sample}\), where spp is the static power per parameter in SRAM and \(L_{sample}\) is the sample time length, assuming always-on operation. The static power consumption of 1bit SRAM is approximately 12.991 pW in the same technology node (Saun & Kumar, 2019). Spikepoint's power consumption exhibits substantial advantages over ANNs and also surpasses the performance of existing SNNs.

### Ablation Study

\(ResF\) **ablation:** To verify the efficacy of the proposed feature extractor \(ResF\), we conduct ablation experiments on the DVS ACTION dataset by varying only the feature extractor variable, keeping the overall architecture, hyperparameters, and datasets consistent. As depicted in Fig. 3(a), three groups of experiments are conducted: the first group utilizes the residual structure with the same architecture as ANN, the second group uses the SpikePoint model without residual connection, and the third group employs the SpikePoint. The results of experiments are marked with blue, orange, and gray, respectively, in Fig. 3(b). The curves demonstrate the superiority of the \(ResF\) block, and the results zoom in on the accuracy curves at the beginning and stabilization periods of the model. SpikePoint converges fastest at the beginning of training and has the highest accuracy after stabilization. Compared with the model without residual, it can be proved that this module can improve the model's fitting ability. SpikePoint is better in convergence speed and convergence accuracy than copying the ANN scheme.

**Structural ablation:** In the structure ablation experiment, we conduct four comparison experiments. The first group involves the standard SpikePoint. The second group is the network with only a local feature extractor. The third group focuses solely on global feature extraction of the

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Model** & **Input** & **Timestep** & **Accuracy(\%)** & **OPN(\%)** & **Dynamic(mJ)** & **Para.(M)** & **Static(mJ)** \\ \hline
**SpikePoint** & **Point** & **16** & **98.7** & **0.9** & **0.82** & **0.58** & **0.756** \\ \hline
**SPW-ResNet(UBS)(Zheng et al., 2021) & Frame & 40 & 96.9 & 4.79 & 4.56 & 11.7 & 15.305 \\ Spiking(Zheng et al., 2021) & Frame & 16 & **98.3** & 3.72 & 4.26 & 2.6 & 3.401 \\ Spiking(Zhao et al., 2021) & Frame & 16 & 97.9 & 6.33 & 10.75 & 2.6 & 3.401 \\ Deep SNN(6)(Aunz et al., 2017) & Frame & 16 & 91.8 & 2.74 & 2.49 & 1.7 & 2.223 \\ Deep SNN(6)(Sheretha & Orchard, 2018) & Frame & 16 & 93.6 & 2.13 & 1.94 & 1.3 & 1.7 \\ \hline
**PILF (Fang et al., 2019)** & Frame & 20 & 97.6 & 2.98 & 2.71 & 17.4 & 22.759 \\ \hline
**PILF(Jann
Point Cloud set without any grouping or sampling operations, as shown in Fig. 4(a). The fourth group is the SNN version of PointNet, which with an architecture that is essentially identical to the third group except for differing dimensions of \([32,64,128,256]\) and \([64,128,256,512,1024]\), respectively. The results provide evidence of the effectiveness of combining local and global features. As depicted in Fig. 4(b), the gray and orange lines represent architectures with only global and local feature extractors, respectively. Not surprisingly, these models perform poorly on the DVS Action dataset, with an accuracy of only 40%. To compare with PointNet, we adjust the dimensionality of each stage to be consistent with it, and the resulting training curve is indicated by the yellow color. We observe that the model shows relatively better results when the dimensionality reaches 1024.

Timestep ablation:The timestep serves as a crucial hyperparameter profoundly influencing the accuracy and power consumption in SNN. The choice of the key parameter is a result of the controlled experiment with varying timestep. We do seven sets of comparison experiments on Daily DVS and DVS Action respectively, and the results are shown as the accuracy of the testset in Table 7.

Grouping ablation:To overcome the challenge of rate encoding input with negative values, we introduce five relevant variables, each corresponding to the number of columns in Table 8. We then conducted several experiments to verify their validity. The results demonstrate the effectiveness of taking absolute values for input, which are superior to normalization, with 2 and 5 in Table 8. And \([x_{min},y_{min},z_{min}]\) can have a certain corrective effect with 2 and 3, 6 and 8. However, \(centroid\) affects the results more than \([x_{min},y_{min},z_{min}]\) in the single channel with 4 and 5. Another branch is used to compensate for this phenomenon with 4 and 6. The dual path structure is better than the single path with 5 and 6, and experiments have shown that the \(Add\) operation is better than the \(Concat\) operation with 6 and 7.

## 5 Conclusion

In this paper, we introduce a full spike event-based network that effectively matches the event cameras' data characteristics, achieving low power consumption and high accuracy. SpikePoint as a singular-stage structure is capable of extracting both global and local features, and it has demonstrated excellent results on four event-based action recognition datasets using back-propagation but not converting ANNs to SNNs. Going forward, our aim is to extend the applicability of SpikePoint to other fields of event-based research, such as SLAM and multimodality (Ding et al., 2023; Ren et al., 2021).

\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline Time steps & 2 & 4 & 8 & 12 & **16** & 24 & 32 \\ \hline DailyDVS Acc.(\%) & 92.75 & 95.17 & 96.53 & 97.22 & **97.92** & 96.7 & 96.01 \\ DVS Action Acc.(\%) & 60.93 & 71.88 & 80.09 & 85.65 & **90.6** & 88.39 & 81.03 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Ablation study of SNN’s timesteps.

Figure 4: Structural ablation experiment (a) and the result (b) on DVS ACTION dataset.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline No. & Absolute & \([x_{min}...]\) & \([x_{c}...]\) & Branch & Fusion & Performance \\ \hline
1 & \(\times\) & \(\times\) & ✓ & single & 97.22\% \\
2 & \([0,1]\) & \(\times\) & ✓ & single & 96.53\% \\
3 & \([0,1]\) & ✓ & \(\times\) & single & 97.36\% \\
4 & ✓ & ✓ & \(\times\) & single & 97.63\% \\
5 & ✓ & \(\times\) & ✓ & single & 97.78\% \\ \hline
6 & ✓ & ✓ & \(\times\) & **double** & **Add** & **97.92\%** \\
7 & ✓ & ✓ & \(\times\) & double & Concat & 97.50\% \\
8 & ✓ & \(\times\) & ✓ & double & Add & 97.50\% \\
9 & \(\times\) & \(\times\) & ✓ & double & Add & 97.22\% \\
10 & [0,1] & \(\times\) & ✓ & double & Add & 96.25\% \\ \hline \hline \end{tabular}
\end{table}
Table 8: Ablation study on grouping in Daily DVS dataset.