Training Full Spike Neural Networks via Auxiliary Accumulation Pathway
Due to the binary spike signals making converting the traditional high-power
multiply-accumulation (MAC) into a low-power accumulation (AC) available, the
brain-inspired Spiking Neural Networks (SNNs) are gaining more and more
attention. However, the binary spike propagation of the Full-Spike Neural
Networks (FSNN) with limited time steps is prone to significant information
loss. To improve performance, several state-of-the-art SNN models trained from
scratch inevitably bring many non-spike operations. The non-spike operations
cause additional computational consumption and may not be deployed on some
neuromorphic hardware where only spike operation is allowed. To train a
large-scale FSNN with high performance, this paper proposes a novel Dual-Stream
Training (DST) method which adds a detachable Auxiliary Accumulation Pathway
(AAP) to the full spiking residual networks. The accumulation in AAP could
compensate for the information loss during the forward and backward of full
spike propagation, and facilitate the training of the FSNN. In the test phase,
the AAP could be removed and only the FSNN remained. This not only keeps the
lower energy consumption but also makes our model easy to deploy. Moreover, for
some cases where the non-spike operations are available, the APP could also be
retained in test inference and improve feature discrimination by introducing a
little non-spike consumption. Extensive experiments on ImageNet, DVS Gesture,
and CIFAR10-DVS datasets demonstrate the effectiveness of DST.
Guangyao Chen, Peixi Peng, Guoqi Li, Yonghong Tian
# Training Full Spike Neural Networks via Auxiliary Accumulation Pathway

###### Abstract

Due to the binary spike signals making converting the traditional high-power multiply-accumulation (MAC) into a low-power accumulation (AC) available, the brain-inspired Spiking Neural Networks (SNNs) are gaining more and more attention. However, the binary spike propagation of the Full-Spike Neural Networks (FSNN) with limited time steps is prone to significant information loss. To improve performance, several state-of-the-art SNN models trained from scratch inevitably bring many non-spike operations. The non-spike operations cause additional computational consumption and may not be deployed on some neuromorphic hardware where only spike operation is allowed. To train a large-scale FSNN with high performance, this paper proposes a novel Dual-Stream Training (DST) method which adds a detachable Auxiliary Accumulation Pathway (AAP) to the full spiking residual networks. The accumulation in AAP could compensate for the information loss during the forward and backward of full spike propagation, and facilitate the training of the FSNN. In the test phase, the AAP could be removed and only the FSNN is remained. This not only keeps the lower energy consumption but also makes our model easy to deploy. Moreover, for some cases where the non-spike operations are available, the APP could also be retained in test inference and improve feature discrimination by introducing a little non-spike consumption. Extensive experiments on ImageNet, DVS Gesture, and CIFAR10-DVS datasets demonstrate the effectiveness of DST.

Machine Learning, ICML

## 1 Introduction

In the past few years, Artificial Neural Networks (ANNs) have achieved great success in many tasks (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; Szegedy et al., 2015; Girshick et al., 2014; Liu et al., 2016; Redmon et al., 2016; Chen et al., 2021, 2020; Ma et al., 2022; Chen and Chen, 2018). However, with ANNs getting deeper and larger, computational and power consumption are growing rapidly. Hence, Spiking Neural Networks (SNNs), inspired by biological neurons, have recently received surging attention and are regarded as a potential competitor of ANNs due to their high biological plausibility, event-driven property, and low power consumption (Roy et al., 2019) on neuromorphic hardware.

To obtain an effective SNN, several works (Kim et al., 2018; Xing et al., 2019; Hwang et al., 2021; Hu et al., 2018; Sengupta et al., 2019; Han et al., 2020; Lee et al., 2020; Zheng et al., 2021; Samadzadeh et al., 2020; Rathi and Roy, 2020; Rathi et al., 2020) are proposed to convert the trained ANN to SNN by replacing the raw activation layers (such as ReLU) with spiking neurons. Although this type of method could achieve state-of-the-art accuracy on many image classification datasets, they often require a large number of time steps which causes high computational consumption and also limit the application of SNN, and the direct promotion of exploring the characteristics of SNN is limited. Hence, a series of methods are proposed to train SNN from scratch. Based on the surrogate gradient backpropagation method (Tavanaei et al., 2019), several recent works train deep SNN by improving the Batchnorm (Zheng et al., 2021) or residual connection structure (Fang et al., 2021; Zhou et al., 2023; Xiao et al., 2022; Deng et al., 2022), and narrow the gap between SNN and ANN effectively. To obtain high performance, several SOTA SNN models (Fang et al., 2021; Zheng et al., 2021; Zhou et al., 2023) inevitably bring many non-spike operations with ADD residual connections. Although effective, these methods may suffer two main drawbacks: Firstly, the energy efficiency advantage of SNNs mainly comes from the fact that the binary spike signals make converting the traditional high-power multiply-accumulation (MAC) into a low-power accumulation (AC) available, the non-spike operations don't fit this characteristic and will bring high computation consumption. Secondly, several neuromorphic hardware only supports
spiking operation, and these models cannot be deployed directly (Horowitz, 2014).

Hence, it is necessary to develop a full-spike neural network (FSNN) that only contains spike operations. However, the binary spike propagation with limited time steps is prone to significant information loss, and limits the performance of FSNN. For example, the Spiking ResNet in Figure 1 is prone to vanishing gradient problems in deep networks (Fang et al., 2021). To compensate for the loss of information forward and backward from full spike propagation, we propose a novel Dual-stream Training (DST) method, where the whole network contains a _full spike propagation_ stream and a _auxiliary spike accumulation_ stream. The former includes full-spike inference of FSNN, while the latter is a plug-and-play Auxiliary Accumulation Pathway (AAP) to the FSNN. In training, the AAP is able to compensate for the loss of information forward and backward from full spike propagation by spike accumulation, which could help alleviate the vanishing gradient problem of the Spiking ResNet and improve the performance of the FSNN. Although the accumulation in APP causes non-spike operation, the AAP could be removed and only the FSNN remains in the test phase. In other words, our model could act as an FSNN inference in practical application. This not only keeps the lower energy consumption but also makes our model easy to deploy in the neuromorphic hardware. Moreover, for some cases where the non-spike operations are available, the AAP could also be retained in test inference and further improve feature discrimination by introducing a small number of non-spike MAC operations. It is notable that FSNN+AAP only brings a linear rise in the non-spike AC computation with the model depth, resulting in less computational consumption.

We evaluate DSNN on both the static ImageNet dataset (Deng et al., 2009) and the neuromorphic DVS Gesture dataset (Amir et al., 2017), CIFAR10-DVS dataset (Li et al., 2017), CIFAR100 (Krizhevsky et al., 2009). The experiment results are consistent with our analysis, indicating that the DST could improve the previous ResNet-based and Transformer-based FSNNs to higher performance by simply increasing the network's depth, and keeping efficient computation consumption simultaneously.

## 2 Related Work

### Spiking Neural Networks

To train deep SNNs, ANN to SNN conversion (ANN2SNN) (Hunsberger and Eliasmith, 2015; Cao et al., 2015; Rueckauer et al., 2017; Sengupta et al., 2019; Han et al., 2020; Han and Roy, 2020; Deng and Gu, 2021; Stockl and Maass, 2021; Li et al., 2021) and backpropagation with surrogate gradient (Neftici et al., 2019) are the two mainstream methods. For ANN2SNN, an ANN with ReLU activation is first trained, then converts the ANN to an SNN by replacing ReLU with spiking neurons and adding scaling operations like weight normalization and threshold balancing. Recently, several ANN2SNN methods (Han et al., 2020; Han and Roy, 2020; Deng and Gu, 2021; Li et al., 2021) have achieved near loss-less accuracy for VGG-16 and ResNet. However, the converted SNN needs longer time steps to rival the original ANN, and increases the SNN's computational consumption (Rueckauer et al., 2017). One of the backpropagation methods (Kim et al., 2020) computes the gradients of the timings of existing spikes with respect to the membrane potential at the spike timing (Comsa et al., 2020; Mostafa, 2017; Kheradpisheh and Masquelier, 2020; Zhou et al., 2021; Zhang and Li, 2020). Another kind of backpropagation method gets the gradient by unfolding the network over the simulation time-steps (Lee et al., 2016; Huh and Sejnowski, 2018; Wu et al., 2018; Shrestha and Orchard, 2018; Lee et al., 2020; Neftci et al., 2019). As the gradient with respect to the threshold-triggered firing is non-differentiable, the surrogate gradient is often used.

### Spiking Residual Networks

For ANN2SNN with ResNet, several methods made specific normalizations for conversion. The residual structure in ANN2SNN with scaled shortcuts is applied in SNN to match the activations of the original ResNet (Hu et al., 2018). Previous ANN2SNN methods noticed the distinction between plain feedforward ANNs and residual ANNs, and made specific normalizations for conversion. Then, Spike-Norm (Sengupta et al., 2019) is proposed to balance the threshold of the Spiking Neural Model and verified their method by converting ResNet to SNNs. Moreover, existing backpropagation-based methods use nearly the same structure as ResNet. Several custom surrogate methods (Sengupta et al., 2019) are evaluated on shallow ResNets. Threshold-dependent batch normalization (td-BN) is proposed to replace naive batch normalization (BN) (Ioffe and Szegedy, 2015) and successfully trained Spiking ResNet-34/50 directly with surrogate gradient by adding td-BN in shortcuts. SEW ResNet is the first method to increase the SNNs to more than 100 layers, but its use of additive operations in residual connections leads to a none-spike structure in the deep layer of the network bringing higher computational consumption. (Deng et al., 2022) introduces the temporal efficient training approach to compensate for the loss of momentum in the gradient descent with SG so that the training process can converge into flatter minima with better generalizability. (Xiao et al., 2022) proposes online training through time for SNNs, which enables forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Recently, (Zhou et al., 2023) considers leveraging both self-attention capability and biological properties of
SNNs and proposes the Spiking Transformer (Spikformer).

## 3 Preliminaries

### Spiking Neuron Model

The spiking neuron is the fundamental computing unit of SNNs. The dynamics of all kinds of spiking neurons can be described as follow:

\[H_{t} =SN(V_{t-1},X_{t}), \tag{1}\] \[S_{t} =\Theta(H_{t}-V_{th}),\] (2) \[V_{t} =H_{t}\ \left(1-S_{t}\right)+V_{reset}\ S_{t}, \tag{3}\]

where \(X_{t}\) is the input at time-step \(t\), \(H_{t}\) and \(V_{t}\) denote the membrane potential after neuronal dynamics and after the trigger of a spike at time-step \(t\), respectively. \(V_{th}\) is the firing threshold, \(\Theta(x)\) is the Heaviside step function and is defined by \(\Theta(x)=1\) for \(x\geq 0\) and \(\Theta(x)=0\) for \(x<0\). \(S_{t}\) is the output spike at time-step \(t\), which equals 1 if there is a spike and 0 otherwise. \(V_{reset}\) denotes the reset potential. The function \(SN(\cdot)\) in Eq. (1) describes the neuronal dynamics and takes different forms for different spiking neuron models, which include the Integrate-and-Fire (IF) model (Eq. (4)) and Leaky Integrate-and-Fire (LIF) model (Eq. (5)):

\[H_{t} =V_{t-1}+X_{t}, \tag{4}\] \[H_{t} =V_{t-1}+\frac{1}{\tau}(X_{t}-(V_{t-1}-V_{reset})), \tag{5}\]

where \(\tau\) represents the membrane time constant. Eq. (2) and Eq. (3) describe the spike generation and resetting processes, which are the same for all kinds of spiking neuron models.

### Computational Consumption

With the sparsity of firing and the short simulation period, SNN can achieve the calculation with about the same number of synaptic operations (SyOPs) (Rueckauer et al., 2017) rather than FLOPs, The number of synaptic operations per layer of the network can be easily estimated for an ANN from the architecture of the convolutional and linear layers. For the ANN, a multiply-accumulate (MAC) computation takes place per synaptic operation. On the other hand, specialized SNN hardware would perform an accumulated computation (AC) per synaptic operation only upon the receipt of an incoming spike. Hence, the total number of AC operations occurring in the SNN would be represented by the dot-product of the average cumulative neural spike count for a particular layer and the corresponding number of synaptic operations. With the deepening of SNN and ANN, the relative energy ratio gradually approaches a fixed value, which could be calculated as follows:

\[\frac{E(\mathrm{SNN})}{E(\mathrm{ANN})}\approx T\cdot fr\cdot\frac{E_{ac}}{E_{ mac}}, \tag{6}\]

where \(T\) and \(fr\) represent the simulation time and the average firing rate.

Eq. (6) assumes the SNN only contains \(\{0,1\}\) spike AC operators. However, several SOTA SNNs employ many non-spike MAC and their computation consumption could not be estimated by Eq. (6) accurately. To estimate the computation consumption of different SNNs more accurately, we calculate the number of computations required in terms of AC and MAC synaptic operations, respectively. The main energy consumption of non-spike signals comes from the MAC between neurons. The contribution from one neuron

Figure 1: Residual blocks and Dual-stream blocks in Spiking ResNet (Fang et al., 2021) and Spike Transformer (Zhou et al., 2023). (a) the Spiking ResNet by replacing ReLU activation layers with spiking neurons. (b) The Spiking ResNet with Dual-stream training. (c) The input of the Spike Transformer with Spiking Self-Attention (SSA) is non-spike data due to residual addition between blocks, which brings additional multiply accumulative computation to SNN. (d) Spike Transformer with a dual-stream structure is divided into a spike propagation pathway and a plug-and-play auxiliary accumulation pathway (AAP), which could compensate for full spike propagation. Note AAP could be either retained or removed in model inference.
to another requires a MAC for each timestep, multiplying each non-spike activation with the respective weight before adding it to the internal sum. In contrast, a transmitted spike requires only an accumulation at the target neuron, adding weight to the potential, and where spikes may be quite sparse. Therefore, for any SNN network \(\mathcal{F}\), the theoretical computational consumption can be determined by the number of AC and MAC operations (\(O_{ac},O_{mac}\)):

\[\mathrm{E}(\mathcal{F})=T\cdot(fr\cdot E_{ac}\cdot O_{ac}+E_{mac}\cdot O_{mac}). \tag{7}\]

Moreover, we developed and open-sourced a tool to calculate the dynamic consumption of SNNs as **syops-counter1**, which can compute the theoretical amount of AC and MAC operations.

Footnote 1: github.com/iCGY96/syops-counter

### Spiking Residual Blocks

There are two main types of residual blocks for existing SNNs. The one replaces ReLU activation layers with spiking neurons, which constructs FSNN but is prone to vanishing gradient problems. The second uses the same addition operation as the ANN, but leads to additional computation consumption due to the appearance of non-spike signals.

Vanishing Gradient Problems of Spiking ResNet.Consider a Spiking ResNet with \(k\) sequential blocks to transmit \(s_{l}^{t}\), and the identity mapping condition is met by using the IF neurons for residual connection with \(0<V_{th}\leq 1\), then we have \(s_{l}^{t}=s_{l+1}^{t}=...=s_{l+k-1}^{t}=o_{l+k-1}^{t}\). The gradient of the output of the \((l+k-1)\)-th residual block with respect to the input of the \(l\)-th residual block could be calculated layer by layer:

\[\begin{split}\frac{\partial o_{l+k-1}^{t}}{\partial s_{l}^{t}}& =\prod_{i=0}^{k-1}\frac{\partial o_{l+i}^{t}}{\partial s_{l+i}^{t} }=\prod_{i=0}^{k-1}\Theta^{\prime}(s_{l+i}^{t}-V_{th})\\ &\rightarrow\begin{cases}0,\text{if}\;0<\Theta^{\prime}(s_{l}^{t} -V_{th})<1\\ 1,\text{if}\;\Theta^{\prime}(s_{l}^{t}-V_{th})=1\end{cases},\end{split} \tag{8}\]

where \(\Theta(x)\) is the Heaviside step function and \(\Theta^{\prime}(x)\) is defined by the surrogate gradient. The second equality hold as \(o_{l+i}^{t}=\mathrm{SN}(s_{l+i}^{t})\). In view of the fact that \(s_{l}^{t}\) could only take 0 or 1 with identity mapping, \(\Theta^{\prime}(s_{l}^{t}-V_{th})=1\) is not satisfied for commonly used surrogate functions mentioned in (Nefci et al., 2019). When using the common \(Sigmoid\)(Nefci et al., 2019) function as the Heaviside step function, the gradient vanishing problem would be prone be happen.

Spiking Residual Blocks with ADD.As illustrated in Figure 1(c), the residual block for SNN can be formulated with an ADD function (Fang et al., 2021; Zhou et al., 2023), which can implement identity mapping and overcome the vanishing and exploding gradient problems.

\[o_{l}^{t}=\mathrm{SN}(f_{l}(o_{l-1}^{t}))+o_{l-1}^{t}=s_{l}^{t}+o_{l-1}^{t}, \tag{9}\]

where \(s_{l}^{t}\) denotes the residual mapping learned as \(s_{l}^{t}=\mathrm{SN}(f_{l}(o_{l-1}^{t}))\). While this design brings performance improvements, it inevitably brings in non-spike data and thus MAC operations. In particular, for the ADD function, if both \(s_{l}^{t}\) and \(o_{l-1}^{t}\) are spike signals, its output \(o_{l}^{t}\) will be a non-spike signal whose value range is \(\{0,1,2\}\). As the depth of the network increases, the range of signals transmitted to the next layer of the network will also expand. Convolution requires much more computational overhead of multiplication and addition when dealing with these non-spiking signals as shown in Figure 1(b). In this case, the network will incur additional high computational consumption.

## 4 Dual-stream Training

### Dual-stream SNN

Basic Block.Instead of the residual Block containing only one path with respect to the input spike \(x\), we initialize the input to two consistent paths \(s_{0}^{t}=a_{0}^{t}=x\), where \(s_{l}^{t}\) represents the spike signal propagated between blocks, and \(a_{l}^{t}\) represents the spike accumulation carried out on the output of each block. As illustrated in Figure 1(b) and (d), the Dual-Stream Block can be formulated as:

\[\begin{split} o_{l}^{t}&=\begin{cases}\mathrm{SN}(f_ {l}(o_{l-1}^{t})+o_{l-1}^{t})=SN(s_{l}^{t}+o_{l-1}^{t})\\ g(\mathrm{SN}(f_{l}(o_{l-1}^{t})),o_{l-1}^{t})=g(s_{l}^{t},o_{l-1}^{t})\end{cases} \\ a_{l}^{t}&=s_{l}^{t}+a_{l-1}^{t},\end{split} \tag{11}\]

where \(g\) represents an element-wise function with two spikes tensors as inputs. Note that here we restrict \(g\) to be only the corresponding _logical operation function_ as shown in Table 1, so as to ensure that the input and output of \(g\) function are spike trains. Here, Eq.(10) is the full-spike propagation pathway, which could be either of two types of spiking ResNet. Eq.(11) denotes the plug-and-play Auxiliary Accumulation Pathway (AAP). During the inference phase, the auxiliary accumulation could be removed as needed.

Downsampling Block.Remarkably, when the input and output of one block have different dimensions, the shortcut is set as convolutional layers with stride \(>1\), rather than the identity connection, to perform downsampling. The Spiking ResNet utilize {Conv-BN} without ReLU in the shortcut. SEW ResNet (Fang et al., 2021) adds an SN in shortcut as shown in Figure 2(b). Figure 2(b) shows the downsampling of auxiliary accumulation, that the overhead of multiply accumulation in Dual-stream Blocks mainly comes from the downsampling of the spike accumulation signal. Fortunately, the number of downsampling in a network is always fixed, so the MAC-based downsampling operation of the spike
accumulation pathway will not increase with the increase of network depth. Therefore, the increased computational overhead of DSNN with the increase of network depth mainly comes from the accumulation calculation.

**Training.** For backpropagation, the gradient could be back-propagated to these spiking neurons through the auxiliary accumulation to prevent the vanishing gradient caused by deeper layers. Therefore, in the training phase, the outputs of both auxiliary accumulation and spike propagation are used as the final output and the gradient is calculated according to a consistent objective function \(\mathcal{L}_{c}\):

\[\mathcal{L}(x,y)=\mathcal{L}_{c}(\mathcal{O}_{s},y)+\mathcal{L}_{c}(\mathcal{O }_{a},y), \tag{12}\]

where \(\mathcal{O}_{s}\) and \(\mathcal{O}_{a}\) are the final outputs of auxiliary accumulation and spike propagation respectively.

### Identity Mapping

As stated in (He et al., 2016), satisfying identity mapping is crucial to training a deep network.

**Auxiliary Accumulation Pathway.** For auxiliary accumulation of Eq.(11), identity mapping is achieved by when \(s_{l}^{t}\equiv 0\), which can be implemented by setting the weights and the bias of the last BN layer in \(f_{l}\) to zero.

The Spiking Neurons Residuals.The first spike propagation of Eq.(10) based on Spiking ResNet could implement identity mapping by partial spiking neurons. When \(f_{l}(o_{l}^{t-1})\equiv 0\), \(o_{l}^{t}=\mathrm{SN}(o_{l}^{t-1})\neq o_{l}^{t-1}\). To transmit \(o_{l}^{t-1}\) and make \(\mathrm{SN}(o_{l}^{t-1})=o_{l}^{t-1}\), the last spiking neuron (SN) in the \(l\)-th residual block needs to fire a spike after receiving a spike and keep silent after receiving no spike at time-step \(t\). It works for IF neurons described by Eq. (4). Specifically, we can set \(0<V_{th}\leq 1\) and \(V_{t-1}=0\) to ensure that \(o_{t}=1\) leads to \(H_{t}\geq V_{th}\), and \(o_{t}=0\) leads to \(H_{t}<V_{th}\). Hence, we replaced all the spiking neurons at the residual connections with IF neurons.

The Element-wise Logical Residuals.For the second spike propagation of Eq.(10), different element-wise functions \(g\) in Table 1 satisfy identity mapping. Specifically, for \(\mathrm{IAND}\), \(\mathrm{OR}\) and \(\mathrm{XOR}\) as element-wise functions \(g\), identity mapping could be achieved by setting \(s_{l}^{t}\equiv 0\). Then \(o_{l}^{t}=g(s_{l}^{t},o_{l-1}^{t})=g(\mathrm{SN}(0),o_{l-1}^{t})=g(0,o_{l-1}^{ t})=o_{l-1}^{t}\). This is consistent with the conditions for auxiliary accumulation Eq.(11) to achieve identity mapping and is applicable to all neuron models. In contrast, for \(\mathrm{AND}\) as the element-wise function \(g\), \(s_{l}^{t}\) should be ones to get identity mapping. Then \(o_{l}^{t}=1\wedge o_{l-1}^{t}=o_{l-1}^{t}\). Although the input parameters of spiking neuron models can be adjusted to practice identity mapping, it is conflicted with the auxiliary accumulation pathway for identity mapping. This is not conducive to maintaining the consistency of signal propagation between the two pathways, thus affecting the effects of training and final recognition. Meanwhile, it is hard to control some spiking neuron models with complex neuronal dynamics to generate spikes at a specified time step.

### Vanishing Gradient Problem

Auxiliary Accumulation Pathway.The gradient for the auxiliary accumulation pathway is calculated as \(\frac{\partial a_{l+k-l}^{t}}{\partial a_{l}^{t}}=\frac{\partial a_{l}^{t}}{ \partial a_{l}^{t}}=1\) with identity mapping. Since the above gradient is a constant, the auxiliary accumulation path of Eq. (11) could also overcome the vanishing gradient problems.

The Spiking Neurons Residuals.Moreover, consider a Spiking ResNet with AAP, the gradient of the output of the \((l+k-1)\)-th residual block with respect to the input of the \(l\)-th residual block with identity mapping could be calculated:

\[\begin{split}\frac{\partial o_{l+k-1}^{t}}{\partial s_{l}^{t}}& +\frac{\partial a_{l+k-1}^{t}}{\partial a_{l}^{t}}=\prod_{i=0}^{k- 1}\frac{\partial o_{l+i}^{t}}{\partial s_{l+i}^{t}}+\frac{\partial a_{l}^{t }}{\partial a_{l}^{t}}\\ &=\prod_{i=0}^{k-1}\Theta^{\prime}(s_{l+i}^{t}-V_{th})+\frac{ \partial a_{l}^{t}}{\partial a_{l}^{t}}\\ &\rightarrow\begin{cases}1,\text{if}\;0<\Theta^{\prime}(s_{l}^{t }-V_{th})<1\\ 2,\text{if}\;\Theta^{\prime}(s_{l}^{t}-V_{th})=1\end{cases},\end{split} \tag{13}\]

Since the above gradient is a constant, the Spiking ResNet with AAP could alleviate the vanishing gradient problems.

\begin{table}
\begin{tabular}{c l} \hline \hline Operator & \(g(x_{l}^{t},s_{l}^{t})\) \\ \hline \(\mathrm{AND}\) & \(s_{l}^{t}\wedge x_{l}^{t}=s_{l}^{t}\cdot x_{l}^{t}\) \\ \(\mathrm{IAND}\) & \((\neg s_{l}^{t})\wedge x_{l}^{t}\) = \((1-s_{l}^{t})\cdot x_{l}^{t}\) \\ \(\mathrm{OR}\) & \(s_{l}^{t}\lor x_{l}^{t}=s_{l}^{t}+x_{l}^{t}-(s_{l}^{t}\cdot x_{l}^{t})\) \\ \(\mathrm{XOR}\) & \(s_{l}^{t}\oplus x_{l}^{t}=s_{l}^{t}\cdot(1-x_{l}^{t})+x_{l}^{t}\cdot(1-s_{l}^{ t})\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: List of element-wise functions \(g\).

Figure 2: The SEW Block (Fang et al., 2021) and its Downsample blocks with a dual-stream structure.
The Element-wise Logical Residuals.When the identity mapping is implemented for the spiking propagation path of Eq.(10), the gradient of the output of the \((l+k)\)-th dual-stream block with respect to the input of the \(l\)-th DSNN block could be calculated layer by layer:

\[\frac{\partial{o_{l+k-1}^{t}}}{\partial{x_{l}^{t}}} =\prod_{i=0}^{k}\frac{\partial{g(s_{l+i}^{t};x_{l+i}^{t})}}{ \partial{x_{l+i}^{t}}} \tag{14}\] \[=\begin{cases}\prod_{i=0}^{k}\frac{\partial{(1+\tau_{i+i}^{t})}}{ \partial{x_{l+i}^{t}}},\text{if }\ g=\mathrm{AND}\\ \prod_{i=0}^{k}\frac{\partial{((1-0)\cdot x_{l+i}^{t})}}{\partial{x_{l+i}^{t}} },\text{if }\ g=\mathrm{IAND}\\ \prod_{i=0}^{k}\frac{\partial{((1+0)\cdot x_{l+i}^{t})}}{\partial{x_{l+i}^{t}} },\text{if }\ g=\mathrm{OR}\\ \prod_{i=0}^{k}\frac{\partial{((1+0)\cdot x_{l+i}^{t})}}{\partial{x_{l+i}^{t} }},\text{if }\ g=\mathrm{XOR}\end{cases}=1.\]

The second equality holds as identity mapping is achieved by setting \(s_{l+i}^{t}\equiv 1\) for \(g=\mathrm{AND}\), and \(s_{l+i}^{t}\equiv 0\) for \(g=\mathrm{IAND}/\mathrm{OR}/\mathrm{XOR}\). Since the gradient in Eq. (14) is a constant, the spiking propagation path of Eq.(10) overcomes the vanishing gradient problems.

## 5 Experiments

### Computational Consumption

To estimate the computational consumption of different SNNs, we calculate the number of computations required in terms of AC and MAC synaptic operations, respectively. Moreover, (Rueckauer et al., 2017) integrates batch normalization (BN) layers into the weights of the preceding layer with loss-less conversion. Therefore, we ignore BN operations when calculating the number of MAC operations, resulting in a more efficient inference consumption. See Appendix A for details of the fusion process of convolution with BN. To quantitatively estimate energy consumption, we evaluate the computational consumption based on the number of AC and MAC operations and the data for various operations in \(45nm\) technology (Horowitz, 2014), where \(\mathrm{E_{MAC}}\) and \(\mathrm{E_{AC}}\) are \(4.6pJ\) and \(0.9pJ\) respectively. Here, we calculate the **D**ynamic **C**ansumption (DC) of the SNN by Eq.(8) based on its spike firing rate on the target dataset. Moreover, we use the **E**stimated **C**onsumption (EC) to estimate the theoretical consumption range from \([0,100\%]\) spike firing rate, that is

\[\mathrm{E}(\mathcal{F})\in[T\cdot E_{mac}\cdot O_{mac},T\cdot(E_{ac}\cdot O_{ ac}+E_{mac}\cdot O_{mac})]. \tag{15}\]

### ImageNet Classification

We validate the effectiveness of our Dual-stream Training method on image classification of ImageNet (Deng et al., 2009) dataset. The IF neuron model is adopted for the static ImageNet dataset. For a fair comparison, all of our training parameters are consistent with SEW ResNet (Fang et al., 2021). As shown in Table 2, two variations of our DST are evaluated: "FSNN (DST)" means the FSNN is trained by the proposed DST and AAP is removed in the test phase, and "DSNN" means the APP is retained to improve the discrimination of features. "FSNN-18" and "DSNN-18" represents the SNN is designed based on ResNet-18, and so on. Three types of methods are compared respectively: "A2S" represents ANN2SNN methods, "FSNN" and "MPSNN" means Full Spike Neural Networks and Mixed-Precision Spike Neural Networks respectively. These notations keep the same in the below.

As shown in Table 2, we can obtain 3 following key findings:

First, the SOTA ANN2SNN methods (Li et al., 2021; Hu et al., 2018) achieve higher accuracies than FSNN as well as other SNNs trained from scratch, but they use 64 and 87.5 times as many time-steps as FSNN respectively, which means that they require more computational consumption. Since most of these methods do not provide the trained models and their DCs are not available, we only used EC to evaluate the consumption. From Table 2, these models with larger time steps also have larger computational consumption. (Meng et al., 2022) also achieves good performance with ResNet-18, but its SNN acquisition process is relatively complex. It needs ANN to pre-train the model first and then retrain the SNN. Although its time steps are less than other ANN2SNN methods, it is still 12.5 times that of DSNN. Due to ANN2SNN being a different type of method from ours, the comparisons are listed just for reference.

Second, for full-spike neural networks, FSNN (DST) outperforms Spiking ResNet (Zheng et al., 2021; Deng et al., 2022) even with lower computational consumption. The performance of FSNN (DST) has a major advantage over other FSNNs and also improves with the increasing depth of the network. It indicates that the proposed DST is indeed helpful to train FSNN.

Finally, the performance of DSNN is further improved when AAP is added to FSNN at the inference phase. At the same time, the DSNN offers superior performance compared to the mixed-precision SEW ResNet (Fang et al., 2021). Note that the performance gap between SEW ResNet-34 and SEW ResNet-50 is not big, and the computational consumption of SEW ResNet-34 is higher than SEW ResNet-50. This phenomenon comes from that ResNet-34 and ResNet-50 use BasicBlock (He et al., 2016) and Bottleneck (He et al., 2016) as blocks respectively. As shown in Figure 1 (b), the first-layer convolution of each block is regarded as a MAC computation operation due to the non-spike data generated by ADD function. However, the first-layer convolution of BasicBlock is much larger than the synaptic operation of Bottleneck, which causes the computational consumption of SEW ResNet-34 to be greater than that of
ResNet-50. In contrast, DSNN ensures that the input and output of each block are spike data through the dual-stream mechanism, so that the theoretical computational consumption increases linearly with the increase of network depth.

### DVS Classification

Dvs Gesture.We also compare our method with SEW ResNet-7B-Net (Fang et al., 2021) on the DVS Gesture dataset (Amir et al., 2017), which contains 11 hand gestures from 29 subjects under 3 illumination conditions. We use the similar network structure 7B-Net in (Fang et al., 2021). As shown in Table 3, FSNN-7 (DST) with \(\mathrm{IAND}\) obtains better performance than other FSNN methods, such as SEW with \(\mathrm{IAND}\) and td-BN, demonstrating the DST is effective to train FSNN. In addition, even though SEW utilizes \(\mathrm{IAND}\) as \(g(\cdot)\) which brings non-spike operations, our FSNN-7 (DST) still achieves comparable performance, and only requires a tenth of the computational consumption of SEW.

Cifar10-Dvs.We also evaluate Spiking ResNet models on the CIFAR10-DVS dataset (Li et al., 2017), which is obtained by recording the moving images of the CIFAR-10 dataset on an LCD monitor by a DVS camera. We use Wide-7B-DSNN which is a similar network structure Wide-7B-Net in (Fang et al., 2021). As shown in Table 3, FSNN-7 (DST) achieves better performance and lower computational consumption than the previous Spiking ResNet (Zheng et al., 2021) and SEW ResNet (Fang et al., 2021).

### Further Analysis

Computational ConsumptionHere, we analyze the computational consumption advantages of DSNN. First, the networks in most ANN2SNN methods are full-spike, where few MAC operations are from batch normalization and conversion of images to spikes. However, their computational consumption is still very large due to the large time steps. DSNN and FSNN has fewer time steps than ANN2SNN methods, which means that its theoretical maximum consumption is much smaller than ANN2SNN as shown in Table 2. Second, the DSNN has lower EC and DC than Spiking ResNet based on addition (SEW ResNet), and achieves better performance. The additive-based SEW ResNet will increase its AC and MAC as the network gets deeper, which will bring about a multifold increase in com

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline Network & Methods & Acc@1 & \(T\) & EC(mJ) & \(O_{ac}\)(G) & \(O_{mac}\)(G) & DC(mJ) \\ \hline PreAct-ResNet-18 (Meng et al., 2022) & A2S & 67.74 & 50 & \([1.43,77.84]\) & - & - & - \\ Spiking ResNet-34 (Rathi et al., 2020) & A2S & 61.48 & 250 & \([7.31,805.79]\) & - & - & - \\ Spiking ResNet-34 (Li et al., 2021) & A2S & 74.61 & 256 & \([7.45,825.29]\) & - & - & - \\ \hline Spiking ResNet-34 with td-BN (Zheng et al., 2021) & FSNN & 63.72 & 6 & \([0.69,19.85]\) & 5.34 & 0.15 & 5.50 \({}^{\dagger}\) \\ Spiking ResNet-50 with td-BN (Zheng et al., 2021) & FSNN & 64.88 & 6 & \([1.10,22.59]\) & 6.01 & 0.24 & 6.52 \({}^{\dagger}\) \\ Spiking ResNet-34 (Deng et al., 2022) & FSNN & 64.79 & 6 & \([0.69,19.85]\) & 5.34 & 0.15 & 5.50 \({}^{\dagger}\) \\ FSNN-18 (DST) & FSNN & 62.16 & 4 & \([0.55,4.31]\) & 1.69 & 0.12 & 2.07 \\ FSNN-34 (DST) & FSNN & 66.45 & 4 & \([0.55,7.64]\) & 3.42 & 0.12 & 3.63 \\ FSNN-50 (DST) & FSNN & 67.69 & 4 & \([0.55,10.42]\) & 3.14 & 0.12 & 3.38 \\ FSNN-101 (DST) & FSNN & 68.38 & 4 & \([0.55,20.64]\) & 4.42 & 0.12 & 4.53 \\ \hline SEW ResNet-18 (ADD) (Fang et al., 2021) & MPSNN & 63.18 & 4 & \([12.65,16.40]\) & 0.51 & 2.75 & 13.11 \\ SEW ResNet-34 (ADD) (Fang et al., 2021) & MPSNN & 67.04 & 4 & \([29.72,36.78]\) & 0.86 & 6.46 & 30.50 \({}^{\dagger}\) \\ SEW ResNet-50 (ADD) (Fang et al., 2021) & MPSNN & 67.78 & 4 & \([23.64,33.50]\) & 2.01 & 5.14 & 25.45 \\ SEW ResNet-34 (ADD) (Deng et al., 2022) & MPSNN & 68.00 & 4 & \([29.72,36.78]\) & 0.86 & 6.46 & 30.50 \({}^{\dagger}\) \\ SEW ResNet-101 (ADD) (Fang et al., 2021) & MPSNN & 68.76 & 4 & \([39.88,59.97]\) & 3.07 & 8.67 & 42.65 \\ DSNN-18 & MPSNN & 63.46 & 4 & \([0.92,4.67]\) & 1.69 & 0.20 & 2.44 \\ DSNN-34 & MPSNN & 67.52 & 4 & \([0.92,8.00]\) & 3.42 & 0.20 & 4.00 \\ DSNN-50 & MPSNN & 69.56 & 4 & \([6.30,16.17]\) & 3.20 & 1.37 & 9.18 \\ DSNN-101 & MPSNN & 71.12 & 4 & \([6.30,26.39]\) & 4.48 & 1.37 & 10.33 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison with previous Spiking ResNet on ImageNet. \(\dagger\) denotes the estimated dynamic consumption based on the spike firing rate provided in the corresponding paper. A2S represents ANN2SNN methods, FSNN and MPSNN mean Full Spike Neural Networks and Mixed-Precision Spike Neural Networks respectively. FSNN (DST) represent the FSNN is trained by the proposed DST and DSNN means APP is retained in the test phase.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Networks & \(g(\cdot)\) & DVS Gesture & CIFAR10-DVS & \(T\) \\  & & ACC/DC(mJ) & & \\ \hline SEW (Fang et al., 2021) & ADD & **97.92**/17.09 & 74.4/16.71 & 16 \\ \hline SEW (Fang et al., 2021) & IAND & 95.49/1.48 & - & 16 \\ td-BN (Zheng et al., 2021) & - & 96.87/- & 67.8/- & 40/10 \\ \hline FSNN-7 (DST) & \(\mathrm{AND}\) & 55.56/2.59 & 70.9/3.57 & 16 \\ FSNN-7 (DST) & OR & 96.18/1.04 & 74.8/3.51 & 16 \\ FSNN-7 (DST) & XOR & 96.53/1.19 & 74.0/3.41 & 16 \\ FSNN-7 (DST) & IAND & 97.57/1.10 & 73.14/6.65 & 16 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison with Spiking ResNet methods on DVS Gesture and CIFAR10-DVS. Once the ADD function is used as \(g(\cdot)\), it will bring non-spiking operation to SEW (Fang et al., 2021). The comparison with it is listed just for reference.
putational consumption. In contrast, the main MAC operation for DSNN comes from the downsampling process of accumulated signals in spike accumulation. Therefore, a deeper DSNN only increases the AC operation, and its MAC operation is constant as shown in Table 2. Finally, it also reveals for the first time the positive relationship between computational consumption and the performance of SNNs. In addition, the performance of DSNN increases gradually with the increase in computational consumption. It indicates the added computational consumption of deeper DSNN is meaningful.

Vanishing Gradient Problem.As mentioned in Section 3.3, Spiking ResNet suffers from a vanishing gradient problem. As shown in Figure 3, the performance of the Spiking ResNet gradually decreases as the number of network layers increases. Network depth did not bring additional gain to the original Spiking ResNet. With the addition of DST, the performance of the Spiking ResNet gradually increases with increasing network depth. Also, the performance is superior with the addition of AAP. As demonstrated in Eq.(13), DST could help alleviate the vanishing gradient problem of the Spiking ResNet.

Feature Enhancement.Here, we analyze the effects of auxiliary accumulation in terms of improving feature discrimination. Moreover, AAP increases the feature discriminant of forward as shown in Figure 4. Under fewer time steps, the spike feature is almost a binary feature, because its discrete property limits the discriminant of its feature. The discrimination of features from spike propagation is not enough, which affects the overall performance. On the other hand, we could increase the number of deep spike features by increasing time steps, so as to improve the discrimination by increasing the number of features, which often leads to too high computational consumption as shown in Table 2. More importantly, AAP separates the computation of AC operations from the MAC computation of recognition, allowing SNN to take full advantage of its low computational consumption.

Spike Transformer.We also test the role of DST on the Transformer on CIFAR100. The latest method Spikeformer Zhou et al. (2023) utilizes \(\mathrm{ADD}\) as its residual connection and achieve SOTA performance. However, the \(\mathrm{ADD}\) introduces additional computational consumption from non-spike data. As shown in Table 4, once the \(\mathrm{IAND}\) is replaced by \(\mathrm{IAND}\), the performance of the full-spike Spikeformer will drop. After introducing APP as shown in Figure 1, the Spikeformer with a full-spike signal achieves similar performance to the original mixed-precision Spikeformer. Notably, there is no downsampling in the Spikeformer, which further exploits the advantages of AAP while not introducing additional MAC operations. The small amount of MAC operations comes mainly from the image-to-spike conversion and the classifier operations.

## 6 Conclusion & Outlook

In this paper, we point out that the main contradiction of FSNNs comes from information loss of full spike propagation. Therefore, we propose the Auxiliary Accumulation Pathway with consistent identity mapping which is able to compensate for the loss of information forward and backward from full spike propagation, so as to keep computationally efficient and high-performance recognition simultaneously. The experiments on the ImageNet, DVS Gesture, and CIFAR10-DVS indicate that DST could improve the ResNet-based and Transformer-based SNNs trained from scratch in both accuracy and computation consumption. Looking forward, this SNN with a dual-stream structure may provide a reference for the design of neuromorphic hardware, which

\begin{table}
\begin{tabular}{l c c c} \hline \hline Networks & Method & Acc & DC(mJ) \\ \hline Spikformer (ADD) & MPSNN & 77.21 & 5.27 \\ \hline Spikformer (IAND) & FSNN & 75.54 & 0.82 \\ Spikformer (IAND) w/ DST & FSNN & 76.96 & 0.84 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Learning Spike Transformer Zhou et al. (2023) with DST. ADD will bring additional computational consumption from non-spike data. Spikformer (IAND) is the full-spike version of Spikformer, and Spikformer (IAND) w/ DST means the full-spike Spikformer is trained by our DST.

Figure 4: The the t-SNE Maaten and Hinton (2008) plots of embedding features on DVS Gesture.

Figure 3: Training Spiking ResNet from scratch on ImageNet with/without Dual-Stream Training.
could improve computational efficiency by separating spike and non-spike computation. In addition, the proposed dual-streams mechanism is similar to the dual-streams object recognition pathway in the human brain, which may provide a new potential direction for SNN structure design.
