Spikingformer: Spike-driven Residual Learning for Transformer-based
  Spiking Neural Network
Spiking neural networks (SNNs) offer a promising energy-efficient alternative
to artificial neural networks, due to their event-driven spiking computation.
However, state-of-the-art deep SNNs (including Spikformer and SEW ResNet)
suffer from non-spike computations (integer-float multiplications) caused by
the structure of their residual connection. These non-spike computations
increase SNNs' power consumption and make them unsuitable for deployment on
mainstream neuromorphic hardware, which only supports spike operations. In this
paper, we propose a hardware-friendly spike-driven residual learning
architecture for SNNs to avoid non-spike computations. Based on this residual
design, we develop Spikingformer, a pure transformer-based spiking neural
network. We evaluate Spikingformer on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS
and DVS128 Gesture datasets, and demonstrate that Spikingformer outperforms the
state-of-the-art in directly trained pure SNNs as a novel advanced backbone
(75.85$\%$ top-1 accuracy on ImageNet, + 1.04$\%$ compared with Spikformer).
Furthermore, our experiments verify that Spikingformer effectively avoids
non-spike computations and significantly reduces energy consumption by
57.34$\%$ compared with Spikformer on ImageNet. To our best knowledge, this is
the first time that a pure event-driven transformer-based SNN has been
developed.
Chenlin Zhou, Liutao Yu, Zhaokun Zhou, Zhengyu Ma, Han Zhang, Huihui Zhou, Yonghong Tian
# Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network

###### Abstract

Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks, due to their event-driven spiking computation. However, state-of-the-art deep SNNs (including Spikformer and SEW ResNet) suffer from non-spike computations (integer-float multiplications) caused by the structure of their residual connection. These non-spike computations increase SNNs' power consumption and make them unsuitable for deployment on mainstream neuromorphic hardware, which only supports spike operations. In this paper, we propose a hardware-friendly spike-driven residual learning architecture for SNNs to avoid non-spike computations. Based on this residual design, we develop Spikingformer, a pure transformer-based spiking neural network. We evaluate Spikingformer on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS and DVS128 Gesture datasets, and demonstrate that Spikingformer outperforms the state-of-the-art in directly trained pure SNNs as a novel advanced backbone (75.85\(\%\) top-1 accuracy on ImageNet, + 1.04\(\%\) compared with Spikformer). Furthermore, our experiments verify that Spikingformer effectively avoids non-spike computations and significantly reduces energy consumption by 57.34\(\%\) compared with Spikformer on ImageNet. To our best knowledge, this is the first time that a pure event-driven transformer-based SNN has been developed. Codes will be available at Spikingformer.

## 1 Introduction

Being regarded as the third generation of neural network [1], the brain-inspired Spiking Neural Networks (SNNs) are potential competitors to Artificial Neural Networks (ANNs) due to their high biological plausibility, event-driven property and low power consumption on neuromorphic hardware [2]. In particular, the utilization of binary spike signals allows SNNs to adopt low-power accumulation (AC) instead of the traditional high-power multiply-accumulation (MAC), leading to significant energy efficiency gains and making SNNs increasingly popular [3].

As SNNs get deeper, their performance has been significantly improved [4; 5; 6; 7; 8; 9; 10]. ResNet with skipping connection has been extensively studied to extend the depth of SNNs [5; 8]. Recently, SEW ResNet [5], a representative convolution-based SNN, easily implements identity mapping and overcomes the vanishing/exploding gradient problems of Spiking ResNet [11]. SEW ResNet is the first deep SNN directly trained with more than 100 layers. Spikformer [8], a directly trained representative transformer-based SNN with residual connection, is proposed by leveraging both self-attention capability and biological properties of SNNs. It is the first successful exploration for applying flourishing transformer architecture into SNN design, and shows powerful performance.
However, both Spikformer and SEW ResNet face the challenge of non-spike computations (integer-float multiplications) caused by ADD residual connection. This not only limits their ability to fully leverage the benefits of event-driven processing in energy efficiency, but also makes it difficult to deploy and optimize their performance on neuromorphic hardware [12; 3].

Developing a pure SNN to address the challenge of non-spike computation in Spikformer and SEW ResNet while maintaining high-performance is extremely critical. In this paper, inspired by the architecture design of Binary Neural Network (BNN) [13; 14; 15; 16], we propose Spike-driven Residual Learning architecture for SNN to avoid the non-spike computations. Based on this residual design, we develop a pure transformer-based spiking neural network, named Spikingformer. We evaluate the performance of Spikingformer on static dataset ImageNet[17], CIFAR[18] (including CIFAR10 and CIFAR100) and neuromorphic datasets (including CIFAR10-DVS and DVS128 Gesture). The experimental results show that Spikingformer effectively avoids integer-float multiplications in Spikformer. In addition, as a novel advanced SNN backbone, Spikingformer outperforms Spikformer in all above datasets by a large margin (e.g. + 1.04\(\%\) on ImageNet, + 1.00\(\%\) on CIFAR100).

## 2 Related Work

### Convolution-based Spiking Neural Network

There are two mainstream methods to obtain deep convolution-based SNN models: ANN-to-SNN conversion and direct training through surrogate gradient.

**ANN-to-SNN Conversion.** In ANN-to-SNN conversion [19; 20; 21; 22; 23; 24], a pre-trained ANN is converted to a SNN by replacing the ReLU activation layers with spiking neurons and adding scaling operations like weight normalization and threshold balancing. This conversion process suffers from long converting time steps and constraints on the original ANN design.

**Direct Training through Surrogate Gradient.** In the field of direct training, SNNs are unfolded over simulation time steps and trained with backpropagation through time [25; 26]. Due to the non-differentiability of spiking neurons, surrogate gradient method is employed for backpropagation [27; 28]. SEW ResNet[5] is a representative convolution-based SNN model by direct training, and is the first to increase the number of layers in SNNs to be larger than 100. However, the ADD gate in residual connections of SEW ResNet produces non-spike computations of integer-float multiplications in deep convolution layers. [3] has identified the problem of non-spike computations in SEW ResNet and Spikformer, and attempts to solve it through adding an auxiliary accumulation pathway during training and removing it during inference. This strategy needs tedious extra operations and results in a significant performance degradation compared with the original models.

### Transformer-based Spiking Neural Network.

Most existing SNNs borrow architectures from convolutional neural networks (CNNs), so their performance is limited by the performance of CNNs. The transformer architecture, originally designed for natural language processing [29], has achieved great success in many computer vision tasks, including image classification [30; 31], object detection [32; 33; 34] and semantic segmentation [35; 36]. The structure of transformer leads to promise for a novel kind of SNNs, with great potential to break through the bottleneck of SNNs' performance. So far, two main related works: Spikformer[8] and SpikFormer[37] have proposed spiking neural networks based on transformer architecture. Although Spikformer replaces the activation function used in the feedforward layers with a spiking activation function, there are still a lot of non-spike operations remained, including floating point multiplication, division, exponential operation. Spikformer proposes a novel Spiking Self Attention (SSA) module by using spike-form Query, Key, and Value without softmax, and achieves state-of-the-art performances on many datasets. However, the structure of Spikformer with residual connection still contains non-spike computation. In our study, we adopt the SSA module from Spikformer, and modify the residual structure to be purely event-driven, which is hardware-friendly and energy efficient while improving performance.
## 3 Methods

### Drawbacks of Spikformer and SEW ResNet

At present, Spikformer [8] is the representative work combining deep SNNs with transformer architecture, while SEW ResNet [5] is the representative work of convolution-based deep SNNs. The residual learning plays an extremely important role in both Spikformer and SEW ResNet, but the ADD residual connections in Spikformer and SEW ResNet lead to non-spike computation (integer-float multiplications), which are not event-driven computation. As shown in Fig.1(a), the residual learning of Spikformer and SEW ResNet could be formulated as follows:

\[O_{l}=\mathrm{SN}_{l}(\mathrm{ConvBN}_{l}(O_{l-1}))+O_{l-1}=S_{l }+O_{l-1}, \tag{1}\] \[O_{l+1}=\mathrm{SN}_{l+1}(\mathrm{ConvBN}_{l+1}(O_{l}))+O_{l}=S _{l+1}+O_{l}. \tag{2}\]

where \(S_{l}\) denotes the residual mapping learned as \(S_{l}=\mathrm{SN}(\mathrm{ConvBN}(O_{l-1}))\). This residual design inevitably brings in non-spike data and thus MAC operations in the next layer / block. In particular, \(S_{l}\) and \(O_{l-1}\) are spike signals, and their output \(O_{l}\) are non-spike signal whose range is \(\{0,1,2\}\). Non-spike data destructs event-driven computation in the next convolution layer when computing \(S_{l+1}\) of \(O_{l+1}\). As the depth of the network increases, the range of non-spike data values transmitted to the deeper layer of the network will also expand. In our implementations of Spikformer, the range of the non-spike data could increase to \(\{0,1,2,...,16\}\) when testing Spikformer-8-512 on ImageNet 2012. Obviously, the range of non-spike data is approximately proportional to the number of residual blocks in Spikformer and SEW ResNet.

In fact, integer-float multiplications are usually implemented in the same way as floating-point multiplication in hardware. In this case, the network will incur high energy consumption, approaching to the energy consumption of ANNs with the same structure, which is unacceptable for SNNs.

Figure 1: The residual learning in Spikformer, SEW ResNet and Spikingformer (Ours). (a) shows the residual learning of Spikformer and SEW ResNet, which contain non-spike computation (integer-float multiplications) in \(\mathrm{ConvBN}\) layer. (b) shows our proposed spike-driven residual learning in Spikingformer, which could effectively avoid floating-point multiplication and integer-float multiplications, following the spike-driven principle. Note that Mutistep LIF is spike neuron Leaky Integrate-and-Fire (LIF) model [5; 8] with time steps \(T>1\). Same with Spikformer, \(T\) is an independent dimension for spike neuron layer. In other layers, it is merged with the batch size. We use \(\mathrm{ConvBN}\) to represent a convolution layer and its subsequent BN layer in this work.
### Spike-driven Residual Learning in Spikingformer

Fig.1(b) shows our proposed spike-driven residual learning in Spikingformer. It could effectively avoid floating-point multiplications and integer-float multiplications, following the spike-driven principle. The spike-driven residual learning could be easily formulated as follows:

\[O_{l}=\mathrm{ConvBN}_{l}(\mathrm{SN}_{l}(O_{l-1}))+O_{l-1}=S_{l}+O_ {l-1}, \tag{3}\] \[O_{l+1}=\mathrm{ConvBN}_{l+1}(\mathrm{SN}_{l+1}(O_{l}))+O_{l}=S_ {l+1}+O_{l}. \tag{4}\]

We proposed \(\mathrm{SN}\) - \(\mathrm{ConvBN}\) for residual learning in replacement of \(\mathrm{ConvBN}\) - \(\mathrm{SN}\) in Spikformer and SEW ResNet. In our structure, \(S_{l}+O_{l-1}\) belongs to floating point addition operation, which is same as the addition operation in \(\mathrm{SN}\) layer. Floating point addition operation is the most essential operation of SNN. Obviously, the output \(O_{l}\) is also floating point and will pass through a \(\mathrm{SN}\) layer before participating in the next \(\mathrm{ConvBN}\) computation. Therefore, the pure spike-form feature will be generated after the processing of \(\mathrm{SN}\) layer and the computation of \(\mathrm{ConvBN}\) layer will become pure floating point addition operation, following spike-driven principle and reducing energy consumption vastly.

### Spikingformer Architecture

We propose Spikingformer, which is a novel and pure transformer-based spiking neural network through integrating spike-driven residual blocks. In this section, the details of Spikingformer are discussed. The pipeline of Spikingformer is shown in Fig.2.

Our proposed Spikingformer contains a Spiking Tokenizer (ST), several Spiking Transformer Blocks and a Classification Head. Given a 2D image sequence \(I\in\mathbb{R}^{T\times C\times H\times W}\) (Note that \(C\)=\(3\) in static datasets like ImageNet 2012, \(C\)=\(2\) in neuromorphic datasets like DVS-Gesture), we use the Spiking Tokenizer block for downsampling and patch embedding, where the inputs can be projected as spike-form patches \(X\in\mathbb{R}^{T\times N\times D}\). Obviously, the first layer of Spiking Tokenizer also play a spike encoder role when taking static images as input. After Spiking Tokenizer, the spiking patches \(X_{0}\) will pass to the \(L\) Spiking Transformer Blocks. Similar to the standard ViT encoder block, a Spiking Transformer Block contains a Spiking Self Attention (SSA) [8] and a Spiking MLP block. In the last, a fully-connected-layer (FC) is used for Classification Head. Note that we use a global average-pooling (GAP) before the fully-connected-layer to reduce the parameters of FC and improve the classification classification capability of Spikingformer.

\[X=\mathrm{ST}(I),\quad I\in\mathbb{R}^{T\times C\times H\times W}, X\in\mathbb{R}^{T\times N\times D} \tag{5}\] \[X^{\prime}_{l}=\mathrm{SSA}\left(X_{l-1}\right)+X_{l-1},X^{ \prime}_{l}\in\mathbb{R}^{T\times N\times D},l=1\ldots L\] (6) \[X_{l}=\mathrm{SMLP}\left(X^{\prime}_{l}\right)+X^{\prime}_{l},X_ {l}\in\mathbb{R}^{T\times N\times D},l=1\ldots L\] (7) \[Y=\mathrm{FC}\left(\mathrm{GAP}\left(X_{L}\right)\right)\quad \text{or}\quad Y^{\prime}=\mathrm{FC}\left(\mathrm{GAP}(\mathrm{SN}\left(X_{ L}\right))\right) \tag{8}\]

**Spiking Tokenizer.** As shown in Fig.2, Spiking Tokenizer mainly contains two functions: 1) convolutional spiking patch embedding, and 2) downsampling to project the feature map into a

Figure 2: The overview of Spikingformer, which consists of a Spiking Tokenizer, several Spiking Transformer Blocks and a Classification Head.
smaller fixed size. The spiking patch embedding is similar to the convolutional stream in Vision Transformer [38; 39], where the dimension of spike-form feature channels gradually increased in each convolution layer and finally matches the embedding dimension of patches. In addition, the first layer of Spiking Tokenizer is utilized as a spike encoder when using static images as input. As shown in Eq. 9 and Eq. 10, the convolution part of ConvBN represents the 2D convolution layer (stride-1, 3 x 3 kernel size). \(\mathrm{MP}\) and \(\mathrm{SN}\) represent maxpooling (stride-2) and mutistep spiking neuron, respectively. Eq. 9 is used for Spiking Patch Embedding without Downsampling (SPE), Eq. 10 is Spiking Patch Embedding with Downsampling (SPED). We could use multiple SPEs or SPEDs for specific classification tasks with different downsampling requirements. For example, we use 4 SPEDs for ImageNet 2012 dataset classification with input size as 224*224 (using 16 times downsampling). we use 2 SPEs and 2 SPEDs for CIFAR datset classification with input size as 32*32 (using 4 times downsampling). After the processing of the Spiking Tokenizer block, the input \(I\) is split into an image patch sequence \(X\in\mathbb{R}^{T\times N\times D}\).

\[I_{i} =\mathrm{ConvBN}(\mathrm{SN}(I)) \tag{9}\] \[I_{i} =\mathrm{ConvBN}(\mathrm{MP}(\mathrm{SN}(I))) \tag{10}\]

**Spiking Transformer Block.** A Spiking Transformer Block contains Spiking Self Attention (SSA) mechanism and Spiking MLP block. our Spiking Self Attention part is similar with SSA in Spikformer [8], which is a pure spike-form self attention. However, we make some modifications: 1) we change the spiking neuron layer position according to our proposed spike-driven residual mechanism, avoiding the multiplication of integers and floating-point weights. 2) we choose ConvBN in replacement of \(LinearBN\) (linear layer and batch normalization) in Spikformer. Therefore, the SSA in Spikingformer can be formulated as follows:

\[X^{\prime}=\mathrm{SN}(X), \tag{11}\] \[Q=\mathrm{SN}_{Q}(\mathrm{ConvBN}_{Q}(X^{\prime})),K=\mathrm{SN} _{K}(\mathrm{ConvBN}_{K}(X^{\prime})),V=\mathrm{SN}_{V}(\mathrm{ConvBN}_{V}(X^ {\prime}))\] (12) \[\mathrm{SSA}(Q,K,V)=\mathrm{ConvBN}(\mathrm{SN}(QK^{T}V*s)) \tag{13}\]

where \(Q,K,V\in\mathbb{R}^{T\times N\times D}\) are pure spike data (only containing 0 and 1). \(s\) is the scaling factor as in [8], controlling the large value of the matrix multiplication result. The Spiking MLP block consists of two SPEs which is formulated in Eq.9. Spiking Transformer Block is shown in Fig.2, and it is the main component of Spikingformer.

**Classification Head.** We use a fully-connected-layer as the classifier behind the last Spiking Transformer Block. In detail, the classifier could be realized in four forms: \(\mathrm{AvgPooling\,\text{-}\,FC}\), \(\mathrm{SN\,\text{-}\,AvgPooling\,\text{-}\,FC}\), \(\mathrm{FC\,\text{-}\,AvgPooling\,\text{-}\,SC}\) - \(\mathrm{AvgPooling\,\text{-}\,FC}\) - \(\mathrm{AvgPooling\,\text{-}\,FC}\) and formulated as follows:

\[Y =\mathrm{FC}(\mathrm{AvgPooling}(X_{L})) \tag{14}\] \[Y =\mathrm{FC}(\mathrm{AvgPooling}(\mathrm{SN}(X_{L})))\] (15) \[Y =\mathrm{AvgPooling}(\mathrm{FC}(X_{L}))\] (16) \[Y =\mathrm{AvgPooling}(\mathrm{FC}(\mathrm{SN}(X_{L}))) \tag{17}\]

\(\mathrm{AvgPooling\,\text{after}\,FC}\) (like \(\mathrm{SN\,\text{-}\,FC}\) - \(\mathrm{AvgPooling\,\text{-}\,AvgPooling}\)) could be considered as computing the average of neuron firing, a post-processing of network, but in this way \(\mathrm{FC}\) usually requires numerous parameters. \(\mathrm{AvgPooling\,\text{before}\,FC}\) (like \(\mathrm{AvgPooling\,\text{-}\,FC}\), \(\mathrm{SN\,\text{-}\,AvgPooling\,\text{-}\,FC}\)) could effectively reduce parameters compared with the previous ways. Only \(\mathrm{SN\,\text{-}\,FC}\) - \(\mathrm{AvgPooling\,\text{-}\,FC}\) avoid floating-point multiplication operation, but it needs more \(\mathrm{FC}\) parameters than \(\mathrm{AvgPooling\,\text{-}\,FC}\) or \(\mathrm{SN\,\text{-}\,AvgPooling\,\text{-}\,FC}\). In addition, it also reduces the classification ability of the network. In this paper, we mainly adopt the way of \(\mathrm{AvgPooling\,\text{ahead}\,\text{of}\,FC}\), and choose \(\mathrm{AvgPooling\,\text{-}\,FC}\) as the classifier of Spikingformer by default. Some experimental analysis on the classification head will be discussed in Sec.5.1.

### Theoretical Synaptic Operation and Energy Consumption Calculation

The homogeneity of convolution allows the following BN and linear scaling transformation to be equivalently fused into the convolutional layer with an added bias when deployment [40; 41; 11; 3]. Therefore, when calculating the theoretical energy consumption, the consumption of BN layers could be ignored. We calculate the number of the synaptic operations of spike before calculating theoretical energy consumption for Spikingformer.

\[SOP^{l}=fr\times T\times FLOPs^{l} \tag{18}\]
where \(l\) is a block/layer in Spikingformer, \(fr\) is the firing rate of the block/layer and \(T\) is the simulation time step of spike neuron. \(FLOPs^{l}\) refers to floating point operations of block/layer \(l\), which is the number of multiply-and-accumulate (MAC) operations. And \(SOP^{l}\) is the number of spike-based accumulate (AC) operations. We estimate the theoretical energy consumption of Spikingformer according to [42; 7; 12; 43; 44; 45; 46]. We assume that the MAC and AC operations are implemented on the 45nm hardware [12], where \(E_{MAC}=4.6pJ\) and \(E_{AC}=0.9pJ\). The theoretical energy consumption of Spikingformer can be calculated as follows:

\[E_{Spikingformer}^{rgb}=E_{AC}\times\left(\sum_{i=2}^{N}SOP_{\text{Conv}}^{i} +\sum_{j=1}^{M}SOP_{\text{SSA}}^{j}\right)+E_{MAC}\times\left(FLOP_{\text{Conv }}^{1}\right) \tag{19}\]

\[E_{Spikingformer}^{neuro}=E_{AC}\times\left(\sum_{i=1}^{N}SOP_{\text{Conv}}^{i} +\sum_{j=1}^{M}SOP_{\text{SSA}}^{j}\right) \tag{20}\]

Eq.19 shows energy consumption of Spikingformer for static datasets with RGB images input. \(FLOP_{Conv}^{1}\) is the first layer encoding static RGB images into spike-form. Then the SOPs of \(N\) SNN Conv layers and \(M\) SSA layers are added together and multiplied by \(E_{AC}\). Eq.20 shows energy consumption of Spikingformer for neuromorphic datasets.

## 4 Experiments

In this section, we carry out experiments on static dataset ImageNet [17], static dataset CIFAR [18] (including CIFAR10 and CIFAR100) and neuromorphic datasets (including CIFAR10-DVS and DVS128 Gesture [47]) to evaluate the performance of Spikingformer. The models for conducting experiments are implemented based on Pytorch [48], SpikingJelly [49] and Timm [50].

### ImageNet Classification

**ImageNet** contains around \(1.3\) million \(1000\)-class images for training and \(50,000\) images for validation. The input size of our model on ImageNet is set to the default \(224\times 224\). The optimizer

\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline Methods & Architecture & Param & OPs & Time & Energy Con- & Top-1 Acc \\  & & (M) & (G) & Step & sumption(mJ) & \\ \hline TET[51] & Spiking-ResNet-34 & 21.79 & - & 6 & - & 64.79 \\  & SEW ResNet-34 & 21.79 & - & 4 & - & 68.00 \\ Spiking ResNet[4] & ResNet-34 & 21.79 & 65.28 & 350 & 59.30 & 71.61 \\  & ResNet-50 & 25.56 & 78.29 & 350 & 70.93 & 72.75 \\ STBP-tdBN[6] & Spiking-ResNet-34 & 21.79 & 6.50 & 6 & 6.39 & 63.72 \\  & SEW ResNet-34 & 21.79 & 3.88 & 4 & 4.04 & 67.04 \\ SEW ResNet-50 & SEW ResNet-50 & 25.56 & 4.83 & 4 & 4.89 & 67.78 \\  & SEW ResNet-101 & 44.55 & 9.30 & 4 & 8.91 & 68.76 \\  & SEW ResNet-152 & 60.19 & 13.72 & 4 & 12.89 & 69.26 \\ MS-ResNet[7] & ResNet-104 & 44.55+ & - & 5 & - & 74.21 \\ \hline ANN[8] & Transformer-8-512 & 29.68 & 8.33 & 1 & 38.34 & 80.80 \\ \hline Spikformer[8] & Spikformer-8-384 & 16.81 & 6.82 & 4 & 12.43 & 70.24 \\  & Spikformer-8-512 & 29.68 & 11.09 & 4 & 18.82 & 73.38 \\  & Spikformer-8-768 & 66.34 & 22.09 & 4 & 32.07 & 74.81 \\ \hline \multirow{3}{*}{**Spikingformer**} & Spikingformer-8-384 & 16.81 & 3.88 & 4 & **4.69(-62.27\(\%\))** & **72.45(+2.21)** \\  & Spikingformer-8-512 & 29.68 & 6.52 & 4 & **7.46(-60.36\(\%\))** & **74.79(+1.41)** \\ \cline{1-1}  & Spikingformer-8-768 & 66.34 & 12.54 & 4 & **13.68(-57.34\(\%\))** & **75.85(+1.04)** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results on ImageNet-1k classification. Power is calculated as the average theoretical energy consumption of an image inference on ImageNet, whose detail is shown in Eq19. Same as Spikformer, our Spikingformer-\(L\)-\(D\) represents a Spikingformer model with \(L\) spiking transformer blocks and \(D\) feature embedding dimensions. OPs refers to SOPs in SNN and FLOPs in ANN. Note that the default input resolution of inference is 224\(\times\)224.
is AdamW and the batch size is set to \(192\) or \(288\) during \(310\) training epochs with a cosine-decay learning rate whose initial value is \(0.0005\). The scaling factor is \(0.125\) when training on ImageNet and CIFAR. Four SPEDs in Spiking Tokenizer splits the image into \(196\)\(16\times 16\) patches.

Same as Spikformer, We try a variety of models with different embedding dimensions and numbers of transformer blocks for ImageNet, which has been shown in Tab. 1. We also show the comparison of synaptic operations (SOPs) [52] and theoretical energy consumption. On one hand, Spkingformer follows spike-driven computation rules, effectively avoiding floating-point multiplication and integer-float multiplications. The histogram of the input data for each transformer block of Spikingformer and Spikformer is shown in Appendix E of Supplementary Material, our Spikingformer effectively avoids producing non-spike data of non-spike computations in Spikformer. On the other hand, Spikingformer-8-512 achieves 74.79\(\%\) top-1 classification accuracy on ImageNet using 4 time steps, significantly outperforms Spikformer-8-512 by 1.41\(\%\), outperforms MS-ResNet model by 0.58\(\%\) and outperforms SEW ResNet-152 model by 5.53\(\%\). Spikingformer-8-512 is with 7.463 mJ theoretical energy consumption, which reduces energy consumption by 60.36\(\%\), compared with 18.819 mJ of Spikformer-8-512. Spikingformer-8-768 achieves 75.85\(\%\) top-1 classification accuracy on ImageNet using 4 time steps, significantly outperforms Spikformer-8-768 by 1.04\(\%\), outperforms MS-ResNet model by 1.64\(\%\) and outperforms SEW ResNet-152 model by 6.59\(\%\). Spikingformer-8-768 is with 13.678 mJ theoretical energy consumption, which reduces energy consumption by 57.34\(\%\), compared with 32.074 mJ of Spikformer-8-768. In addition, we recalculate the energy consumption of Spikformer in Appendix G because the non-spike computation of Spikformer can not be directly calculated by Sec.3.4. The main reason why Spikingformer can significantly reduce energy consumption compared with Spikformer is that Spikingformer could effectively avoid integer-float multiplications, and the secondary reason is that our models have lower firing rate on ImageNet, which is shown in Appendix E.

### CIFAR Classification

**CIFAR10/CIFAR100** provides 50, 000 train and 10, 000 test images with 32 \(\times\) 32 resolution. The difference is that CIFAR10 contains 10 categories for classification, but CIFAR100 contains

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline Methods & Architecture & Param & Time & CIFAR10 & CIFAR100 \\  & & (M) & Step & Top-1 Acc & Top-1 Acc \\ \hline Hybrid training[53] & VGG-11 & 9.27 & 125 & 92.22 & 67.87 \\ Diet-SNN[54] & ResNet-20 & 0.27 & 10/5 & 92.54 & 64.07 \\ STBP[55] & CIFARNet & 17.54 & 12 & 89.83 & - \\ STBP NeuNorm[56] & CIFARNet & 17.54 & 12 & 90.53 & - \\ TSSL-BP[57] & CIFARNet & 17.54 & 5 & 91.41 & - \\ STBP-tdBN[6] & ResNet-19 & 12.63 & 4 & 92.92 & 70.86 \\ TET[51] & ResNet-19 & 12.63 & 4 & 94.44 & 74.47 \\ MS-ResNet[7] & ResNet-110 & - & - & 91.72 & 66.83 \\  & ResNet-482 & - & - & 91.90 & - \\ \hline ANN[8] & ResNet-19* & 12.63 & 1 & 94.97 & 75.35 \\  & Transformer-4-384 & 9.32 & 1 & 96.73 & 81.02 \\ \hline  & Spikformer-4-256 & 4.15 & 4 & 93.94 & 75.96 \\ Spikformer[8] & Spikformer-2-384 & 5.76 & 4 & 94.80 & 76.95 \\  & Spikformer-4-384 & 9.32 & 4 & 95.19 & 77.86 \\  & Spikformer-4-384-400E & 9.32 & 4 & 95.51 & 78.21 \\ \hline  & Spikingformer-4-256 & 4.15 & 4 & **94.77(+0.83)** & **77.43(+1.47)** \\
**Spikingformer** & Spikingformer-2-384 & 5.76 & 4 & **95.22(+0.42)** & **78.34(+1.39)** \\  & Spikingformer-4-384 & 9.32 & 4 & **95.61(+0.42)** & **79.09(+1.23)** \\  & Spikingformer-4-384-400E & 9.32 & 4 & **95.81(+0.30)** & **79.21(+1.00)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results on CIFAR10/100 classification. Spikingformer improves network performance in all tasks, compared with Spikformer. Note that Spikingformer-4-384-400E means Spikingformer contains four spiking transformer blocks and 384 feature embedding dimensions, trained with 400 epochs. Other models of Spikingformer are trained with 310 epochs by default, being consistent with Spikformer.
100 categories, owning better distinguishing ability for classification algorithm. The batch size of Spikingformer is set to 64. We choose two SPEs and two SPEDs in the Spiking Tokenizer block to split the input image into 64 4 x 4 patches.

The experimental results are shown in Tab.2. From the results, We find that the performance of Spikingformer models surpass all the models of Spikformer with same number of parameters. In CIFAR10, our Spikingformer-4-384-400E achieves 95.81\(\%\) classification accuracy, significantly outperforms Spikformer-4-384-400E by 0.30\(\%\) and outperforms MS-ResNet-482 by 3.91\(\%\). In CIFAR100, Spikingformer-4-384-400E achieves 79.21\(\%\) classification accuracy, significantly outperforms Spikformer-4-384-400E by 1.00\(\%\) and outperforms MS-ResNet-110 by 12.38\(\%\). To the best of our knowledge, our Spikingformer achieves the state-of-the-art in directly trained pure spike-driven SNN model on both CIFAR10 and CIFAR100. ANN-Transformer model is 1.12\(\%\) and 1.93\(\%\) higher than Spikformer-4-384 for CIFAR10 and CIFAR100, respectively. In this experiment, we also find that the performance of Spikingformer has positive correlation with block numbers, dimensions and training epochs within a certain range in CIFAR dataset.

### DVS Classification

**CIFAR10-DVS Classification.** CIFAR10-DVS is a neuromorphic dataset converted from the static image dataset by shifting image samples to be captured by the DVS camera, which provides 9, 000 training samples and 1, 000 test samples. We compare our method with SOTA methods on DVS-Gesture. In detail, we adopt four SPEDs in the Spiking Tokenizer block due to 128*128 image size of CIFAR10-DVS, and adopt 2 spiking transformer blocks with 256 patch embedding dimension. The number of time steps of the spiking neuron is 10 or 16. The number of training epochs is 106, which is the same with Spikformer. The learning rate is initialized to 0.1 and decayed with cosine schedule.

The results of CIFAR10-DVS are shown in Tab.3. Spikingformer achieves 81.3\(\%\) top-1 accuracy with 16 time steps and 79.9\(\%\) accuracy with 10 time steps, significantly outperforms Spikformer by 1.3\(\%\) and 0.7\(\%\) respectively. To our best knowledge, our Spikingformer achieves the state-of-the-art in directly trained pure spike-driven SNN models on CIFAR10-DVS.

**DVS128 Gesture Classification.** DVS128 Gesture is a gesture recognition dataset that contains 11 hand gesture categories from 29 individuals under 3 illumination conditions. The image size of DVS128 Gesture is 128*128. The main hyperparameter setting in DVS128 Gesture classification is the same with CIFAR10-DVS classification. The only difference is that the number of training epoch is set as 200 for DVS Gesture classification, which is the same with Spikformer.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{2}{c}{CIFAR10-DVS} & \multicolumn{2}{c}{DVS128 Gesture} \\ \cline{2-5}  & Time Step & Acc & Time Step & Acc \\ \hline LIAF-Net [58]\({}^{\text{TNNLS-2021}}\) & 10 & 70.4 & 60 & 97.6 \\ TA-SNN [59]\({}^{\text{ICCV-2021}}\) & 10 & 72.0 & 60 & 98.6 \\ Rollout [60]\({}^{\text{Front. Neurosci-2020}}\) & 48 & 66.8 & 240 & 97.2 \\ DECOLLE [61]\({}^{\text{Front. Neurosci-2020}}\) & - & - & 500 & 95.5 \\ tdBN [6]\({}^{\text{AAAI-2021}}\) & 10 & 67.8 & 40 & 96.9 \\ PLIF [62]\({}^{\text{ICCV-2021}}\) & 20 & 74.8 & 20 & 97.6 \\ SEW ResNet [5]\({}^{\text{NeurIPS-2021}}\) & 16 & 74.4 & 16 & 97.9 \\ Dspike [63]\({}^{\text{NeurIPS-2021}}\) & 10 & 75.4 & - & - \\ SALT [64]\({}^{\text{Neural Netw-2021}}\) & 20 & 67.1 & - & - \\ DSR [23]\({}^{\text{CVPR-2022}}\) & 10 & 77.3 & - & - \\ MS-ResNet [7] & - & 75.6 & - & - \\ \hline Spikformer[8] (Our Implement) & 10 & 78.6 & 10 & 95.8 \\  & 16 & 80.6 & 16 & 97.9 \\ \hline
**Spikingformer (Ours)** & 10 & **79.9(+1.3)** & 10 & **96.2(+0.4)** \\  & 16 & **81.3(+0.7)** & 16 & **98.3(+0.4)** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Results on neuromorphic datasets, CIFAR10-DVS and DVS128 Gesture. Note that the result of Spikformer is based on our implementation according to its open-source code.
We compare our method with SOTA methods on CIFAR10-DVS in Tab.3. Spikingformer achieves 98.3\(\%\) top-1 accuracy with 16 time steps and 96.2\(\%\) accuracy with 10 time steps, outperforms Spikformer by 0.4\(\%\) and 0.4\(\%\) respectively.

## 5 Discussion

### Further Analysis for Last Layer

We carry out analysis of the last layer of Spikingformer in CIFAR10 and CIFAR100 dataset to study its impacts, and the results are shown in Tab.4. The last layer has significant impacts on model performances although it only occupies a very small component in Spikingformer. Our experiments show that Spikingformer-\(L\)-\(D^{*}\) with spike signals performs worse than Spikingformer-\(L\)-\(D\) on the whole. This can be attributed to the fact that the \(\mathrm{AvgPooling}\) layer of \(\mathrm{AvgPooling}\) - FC processes floating point numbers, which have stronger classification abilities than spike signals in SN - AvgPooling - FC. However, Spikingformer-\(L\)-\(D^{*}\) still outperforms Spikformer-\(L\)-\(D\), which uses a SN - AvgPooling - FC layer as the classifier. On one hand, Spikingformer effectively avoids integer-float multiplications of Spikformer in residual learning. On the other hand, Spikingformer has better spike feature extraction and classification ability than Spikformer. These results further verify the effectiveness of Spikingformer as a backbone.

### More Discussion for \(Activation\)-\(Conv\)-\(BatchNorm\) Paradigm

\(Activation\)-\(Conv\)-\(BatchNorm\) is a fundamental building block in Binary Neural Networks (BNN) [13; 14; 15; 16]. MS-ResNet[7] has inherited \(Activation\)-\(Conv\)-\(BatchNorm\) in directly trained convolution-based SNNs to successfully extend the depth up to 482 layers on CIFAR10, without experiencing degradation problem. MS-ResNet mainly verifies the ability of \(Activation\)-\(Conv\)-\(BatchNorm\) to overcome the problem of gradient explosion/vanishing and performance degradation in convolution-based SNN model. In contrast, to the best of our knowledge, Spikingformer is the first transformer-based SNN model that uses the \(Activation\)-\(Conv\)-\(BatchNorm\) paradigm to achieve pure spike-driven computation. This work further validates the effectiveness and fundamentality of \(Activation\)-\(Conv\)-\(BatchNorm\) as a basic module in SNN design. Specifically, Spikingformer achieves state-of-the-art performance on five datasets, and outperforms MS-ResNet by a significant margin: for ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS, DVS-Guesture, MS-ResNet (our Spikingformer) achieves 74.21\(\%\) (75.85\(\%\)), 91.90\(\%\) (95.81\(\%\)), 66.83\(\%\) (79.21\(\%\)), 75.6\(\%\) (81.3\(\%\)), - (98.3\(\%\)), respectively.

## 6 Conclusion

In this paper, we propose spike-driven residual learning for SNN to avoid non-spike computations in Spikformer and SEW ResNet. Based on this residual design, we develop a pure spike-driven transformer-based spiking neural network, named Spikingformer. We evaluates Spikingformer on ImageNet, CIFAR10, CIFAR100, CIFAR10-DVS and DVS128 Gesture datasets. The experiment results verify that Spikingformer effectively avoids integer-float multiplications in Spikformer. In addition, Spikingformer, as a newly advanced SNN backbone, outperforms Spikformer in all above datasets by a large margin (eg. + 1.04\(\%\) on ImageNet, + 1.00\(\%\) on CIFAR100). To our best knowledge, Spikingformer is the first pure spike-driven transformer-based SNN model, and achieves state-of-the-art performance on the above datasets in directly trained pure SNN models.

\begin{table}
\begin{tabular}{l l l l} \hline \hline Dataset & Models & Time & Top-1 \\  & & Step & Acc \\ \hline \multirow{3}{*}{CIFAR10} & Spikingformer-4-384-400E & 4 & 95.81 \\  & Spikingformer-4-384-400E\({}^{*}\) & 4 & 95.58 \\  & Spikformer-4-384-400E & 4 & 95.51 \\ \hline \multirow{3}{*}{CIFAR100} & Spikingformer-4-384-400E\({}^{*}\) & 4 & 79.21 \\  & Spikingformer-4-384-400E\({}^{*}\) & 4 & 78.39 \\ \cline{1-1}  & Spikformer-4-384-400E & 4 & 78.21 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Discussion results on the last layer of Spikingformer. Spikingformer-\(L\)-\(D^{*}\) means Spikingformer with the last layer of \(\mathrm{SN}\) - \(\mathrm{AvgPooling}\) - FC. Spikingformer-\(L\)-\(D\) means Spikingformer with the last layer of \(\mathrm{AvgPooling}\) - FC by default.
Acknowledgment

This work is supported by grants from National Natural Science Foundation of China 62236009 and 62206141.
