STCSNN: High energy efficiency spike-train level spiking neural networks
  with spatio-temporal conversion
Brain-inspired spiking neuron networks (SNNs) have attracted widespread
research interest due to their low power features, high biological
plausibility, and strong spatiotemporal information processing capability.
Although adopting a surrogate gradient (SG) makes the non-differentiability SNN
trainable, achieving comparable accuracy for ANNs and keeping low-power
features simultaneously is still tricky. In this paper, we proposed an
energy-efficient spike-train level spiking neural network with spatio-temporal
conversion, which has low computational cost and high accuracy. In the STCSNN,
spatio-temporal conversion blocks (STCBs) are proposed to keep the low power
features of SNNs and improve accuracy. However, STCSNN cannot adopt
backpropagation algorithms directly due to the non-differentiability nature of
spike trains. We proposed a suitable learning rule for STCSNNs by deducing the
equivalent gradient of STCB. We evaluate the proposed STCSNN on static and
neuromorphic datasets, including Fashion-Mnist, Cifar10, Cifar100,
TinyImageNet, and DVS-Cifar10. The experiment results show that our proposed
STCSNN outperforms the state-of-the-art accuracy on nearly all datasets, using
fewer time steps and being highly energy-efficient.
Changqing Xu, Yi Liu, Yintang Yang
SLSSNN: High energy efficiency spike-train level spiking neural networks with spatio-temporal conversion

###### Abstract

Brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest due to their low power features, high biological plausibility, and strong spatiotemporal information processing capability. Although adopting a surrogate gradient (SG) makes the non-differentiability SNN trainable, achieving comparable accuracy for ANNs and keeping low-power features simultaneously is still tricky. In this paper, we proposed an energy-efficient spike-train level spiking neural network (SLSSNN) with low computational cost and high accuracy. In the SLSSNN, spatio-temporal conversion blocks (STCBs) are applied to replace the convolutional and ReLU layers to keep the low power features of SNNs and improve accuracy. However, SLSSNN cannot adopt backpropagation algorithms directly due to the non-differentiability nature of spike trains. We proposed a suitable learning rule for SLSSNNs by deducing the equivalent gradient of STCB. We evaluate the proposed SLSSNN on static and neuromorphic datasets, including Fashion-Mnist, Cifar10, Cifar100, TinyImageNet, and DVS-Cifar10. The experiment results show that our proposed SLSSNN outperforms the state-of-the-art accuracy on nearly all datasets, using fewer time steps and being highly energy-efficient.

## 1 Introduction

Spiking neural networks (SNNs) have gained increasing attention recently due to their biologically-inspired computational principles, spatiotemporal information processing capability, and potential for achieving high energy efficiency in neuromorphic computing[1; 2]. However, the training of SNNs is still a significant challenge[3] since information in SNNs is transmitted and processed through non-differentiable spike trains. To tackle this problem, the surrogate gradient (SG) method[4; 5; 6] and the ANN-to-SNN conversion method[7; 8; 8] have been proposed. The ANN-to-SNN conversion method directly obtains the weights of an SNN from a corresponding ANN, relying on the relationship between the firing rates of the SNN and activations of the ANN[3]. The method enables the obtained
SNN to obtain comparable accuracy as ANNs. However, its latency is usually intolerably since only a large number of time steps can close the firing rates to activations of the ANN. The significant latency also eliminates the potential of SNNs for high energy efficiency. On the other hand, surrogate gradient (SG) makes the non-differentiability SNNs can be trained directly with low latency[4; 5; 6] but cannot achieve high performance comparable to leading ANNs. Overall, it's still a challenge for traditional SNNs to achieve high accuracy, low latency, and energy efficiency for real-world applications.

In this paper, we overcome both the low performance and high latency issues of traditional SNNs by introducing spatio-temporal conversion blocks (STCBs) to propose a high energy efficiency spike-train level spiking neural network. In traditional spiking convolutional neural networks (SCNNs), Integrate-and-Fire(IF) models or Leaky-Integrate-and-Fire(LIF) models are used to replace the ReLU function to make the information can be transmitted and processed in spiking form. Non-differentiable spikes make SNNs challenging to train but have excellent potential for high energy efficiency. Our basic idea is that we can utilize the advantages of spikes in energy consumption and keep the high performance of convolutional neural networks (CNNs) meanwhile. The SNN hardware can utilize a binary spike-based sparse processing over a fixed number of time steps via accumulate (AC) operations, consuming much lower power than multiply-accumulate (MAC) operations that dominate computations in ANNs[9]. Convolutional operations consist of a large number of MAC operations, making them one of the primary sources of power consumption for convolutional neural networks[9; 10]. So we proposed spatio-temporal conversion blocks based on IF models to encode the real value of information into spike trains before convolutional layers, which can help reduce the energy consumption of convolution operation significantly. To maintain the efficiency of information transmission between blocks, we use the current accumulator blocks (CABs) to decode the information after the convolutional layers. To effectively train our proposed SLSSNNs with low latency, we further study the error backpropagation of STCBs and deduce the equivalent gradient of STCBs. With these methods, we can train a high-performance, low-latency, and energy-efficiency SLSSNN.

The following summarizes our main contributions:

* We proposed high energy efficiency spike-train level spiking neural networks, which utilize the advantages of spikes in energy consumption and keep the high performance of convolutional neural networks (CNNs) meanwhile.
* We studied the error backpropagation of STCBs and proposed the corresponding equivalent gradient, which avoids the non-differentiability problem in SNN training and does not require the costly error backpropagation through the temporal domain.
* Our proposed SLSSNN achieves competitive or state-of-the-art(SOTA) SNN performance with low latency and energy efficiency on the Fashion-MNIST[11], CIFAR-10[12], CIFAR-100[12], TinyImageNet[13], and DVS-CIFAR10[14] datasets.

## 2 Related work

In recent years, SNNs have attracted widespread research interest and have developed rapidly. However, there are still lots of challenging problems that remain to be unsolved. Many works seek brain-inspired methods with biological plausibility to train SNNs[15; 16; 17]. However, these methods cannot obtain competitive performance and are hard to apply to complicated datasets[3]. Besides the brain-inspired method, SNN training has been carried out in two strategies: ANN-to-SNN conversion and direct training.

ANN-to-SNN conversionThe basic idea of ANN-to-SNN conversion is to convert a high-performance ANN to an SNN and adjust its parameters. This method can avoid training problems of SNNs due to the non-differentiability. But it usually needs high latency to make the firing rates closely approach the high-precision activation, which eliminates the potential of SNNs on energy efficiency. To solve this problem, various technologies are proposed to reduce the latency, such as spike-norm [18], robust normalization [19], and channel-wise normalization [20]. Recently, Kim et al. proposed a novel attention-guided ANN compression method to convert compressed deep SNN models that have high energy efficiency and reduced inference latency[9]. Li et al. proposed to apply adaptive threshold and layer-wise calibration to obtain high-performance SNNs[21]. However, The SNNs obtained by the ANN-to-SNN conversion method have more significant latency compared to the
SNNs trained directly. In addition, the ANN-to-SNN convert method is not suitable for neuromorphic datasets.

Direct trainingTo solve the non-differentiability problem of SNNs, Lee et al. proposed to use a surrogate gradient to replace the non-differentiable activation term.[22]. Compared with the ANN-to-SNN conversion method, direct training achieves high accuracy with fewer time steps but suffers more training costs[23]. Many studies use surrogate gradient (SG) to obtain high-performance SNNs on static and neuromorphic datasets[4; 5; 6; 24]. Wu et al. first proposed the STBP method, which significantly improves the performance of SNNs obtained by direct training [25]. Zhang et al. proposed TSSL-BP to improve temporal learning precision by breaking down error backpropagation across two types of inter-neuron and intra-neuron dependencies and achieve low-latency and high-accuracy SNNs [26]. Recently, Xu et al. proposed a novel training method based on backpropagation (BP) for SNNs with multi-threshold, which can reduce the latency of SNN to 1-2 time steps[1].

## 3 Methodology

### Spatio-temporal conversion block (STCB)

To reduce the energy consumption of convolution layers, we proposed spatio-temporal conversion blocks (STCBs) to replace the convolutional and ReLU layers. The STCB consists of a spike train encoding block (STEB), a convolutional layer, and a current accumulator block (CAB), which is shown in Fig.1. The STCB utilizes IF models with decaying threshold voltage to encode information into spike trains, making MAC operations of convolutional layers convert to AC operations. Due to the feature of Integrate-and-Fire (IF) models, the STEB also has the function of ReLU layers. In STCB, we treat the convolutional layer as a synaptic model, which receives spike and generate synaptic current responses. The CAB is proposed to accumulate synaptic current with time steps and the final accumulated synaptic current will be the output of the next STCB. In STCB, information is transformed into binary information in the form of spike trains to simplify the convolutional operation and is recovered back to real information after convolutional operations.

### Spike train encoding block(STEB) and Current accumulator block (ACB)

Spike train encoding block(STEB)As one of the main components of the STCBs, the spike train encoding block (STEB) utilizes the IF model[27] to encode information into spike trains. As Fig. 2(a) shows, the input data will be used as the initial membrane voltage of the neuron model at the zeroth time step. It is known that the IF model is one of the most widely applied models to describe neuronal dynamics in SNNs. In this paper, we apply the IF models with decaying threshold voltages, whose threshold voltages decay with time steps, to encode input data into spike trains. The IF model can be depicted as

\[I(t)=C_{m}\frac{dV_{m}(t)}{dt} \tag{1}\]

Figure 1: Spatio-temporal conversion block (STCB)
where \(C_{m}\) is the membrane capacitance, \(V_{m}(t)\) is membrane potential and \(I(t)\) denotes the membrane current. Since we only use input data as the initial membrane, the neuronal membrane potential of neuron \(i\) at time step \(t\) is

\[u_{i}[t]=\left\{\begin{array}{rl}I[t],&t=0\\ u_{i}[t-1]+u_{reset}[t],&t>0\end{array}\right. \tag{2}\]

where \(u_{reset}[t]\) denotes the reset function, which reduces the membrane potential by the \(V_{th}[t]\) after the neuron \(i\) fires at the \(t\) time step. \(u_{reset}[t]\) is equal to \(-s_{i}[t]V_{th}[t]\). \(s_{i}[t]\) is the output spike of neuron \(i\) and \(V_{th}[t]\) is the threshold voltage at time step \(t\), which is given by

\[s_{i}[t]=\left\{\begin{array}{rl}0,&u_{i}[t]<V_{th}[t]\\ 1,&u_{i}[t]\geq V_{th}[t]\end{array}\right. \tag{3}\]

\[V_{th}[t]=\left\{\begin{array}{rl}V_{init},&t=0\\ \frac{V_{th}[t-1]}{\tau_{vth}},&t>0\end{array}\right. \tag{4}\]

where \(V_{init}\) is the initial threshold voltage and \(\tau_{vth}\) is the time constant of threshold voltages.

Current accumulator Block (ACB)As we mentioned in section 3.1, convolutional layer is treated as synaptic model. So spike train is converted into a series of synaptic currents after the convolutional operation. We use the current accumulator block to accumulate the synaptic currents into a final synaptic current to improve the transmission efficiency between blocks. As Fig. 2(b) shows, input data at time step \(t\) is multiplied by the threshold voltage at time step \(t\), and the result will be accumulated. The output of ACB \(I_{acc}[t]\) is calculated by

\[I_{acc}[t]=\left\{\begin{array}{rl}0,&t<T_{max}\\ i_{acc}[t],&t=T_{max}\end{array}\right. \tag{5}\]

where \(T_{max}\) is the maximum time step which is also equal to the length of the spike trains. \(i_{acc}[t]\) is the synaptic current at time step \(t\), which is given by

\[i_{acc}[t]=\left\{\begin{array}{rl}I_{syn}[t]V_{th}[t],&t=0\\ i_{acc}[t-1]+I_{syn}[t]V_{th}[t],&t>0\end{array}\right. \tag{6}\]

where \(I_{syn}[t]\) is the synaptic current at time step \(t\), and \(V_{th}[t]\) is the threshold voltage at time step \(t\) which can be obtained by Eq. 4.

### Error Backpropagation of STCB

The information is transmitted in the form of spike trains in STCB, while transmitted in the form of the accumulated synaptic current between STCBs, which makes error backpropagation of STCB

Figure 2: The structure of STEB and CAB. (a) STEB (b) CAB
complex. To solve this issue, we proposed the equivalent gradient of STCBs, which can simplify the error backpropagation of STEB and reduce the computation. As we mentioned in section 3.2, data will be first encoded into spike trains by STEB in the forward pass process. Then the spike trains will convert into a series of synaptic currents by the convolutional layer. Finally, these synaptic currents will be accumulated by CAB, and the final accumulated synaptic current will be used as the output of STCB. As Fig. 3 shows, the equivalent forward pass 1 of STCB has a different order of operations but the same output. STEB + CAB is a process of encoding and decoding. If we ignore the error of this process, STEB + CAB is equal to ReLU. So we can use the backward Pass of ReLU + Cov to replace the one of STCB, which can simplify the error backpropagation of STCB and reduce the computation significantly.

### Network Architecture

Using the STCB mentioned in the previous section, we propose an SLSSNN that accurately classifies static and neuromorphic datasets with a low computational cost and energy consumption. In this section, we chose VGG and ResNet architecture to show how to build an SLSSNN. Fig. 4(a) and 4(b) show the diagram of VGG and ResNet based on STCBs. For the first convolutional block of VGG architecture, we keep a convolutional layer with 3\(\times\)3 filters as the first layer, which helps to obtain enough features from the original data. In other convolution blocks, we just need to use the STCB to replace the convolutional layers. For ResNet architecture, we also keep a convolutional layer with 3\(\times\)3 filters as the first layer. Due to the feature of IF models, the STEB also has the function of ReLU layers. So we use the STCB to replace the convolutional layer + ReLU function in residual blocks.

Figure 4: Diagram of VGG and ResNet based on STCB. (a) VGG, (b) ResNet. Conv refers to the convolutional layer, \(N\times N\) Cov/STCB means the Convolutional layer or STCB with \(N\times N\) filters, BN means Batch Norm, AP is the average pooling layer, AAP is the adaptive average pool, and FC refers to the fully connected layers.

Figure 3: Forward and backward pass of STCB
Experiments and results

In this section, we first evaluate the proposed SLSSNN on four static datasets, Fashion-MNIST[11], CIFAR-10[12], CIFAR-100[12], TinyImageNet[13], and one and neuromorphic dataset, DVS-CIFAR10[14] with different structures of SNNs. And we compared our method with several previously reported state-of-the-art results with the same or similar networks, including different SNNs trained by BP-based methods and converted SNNs. We also study the influence of initial threshold voltages and the length of the spike trains to show the tradeoff between accuracy and latency. Finally, we count the number of spikes for different datasets and analyze energy consumption of SLSSNNs.

### Experiment settings

All reported experiments below are conducted on an NVIDIA Tesla V100 GPU. The implementation of our proposed method is on the Pytorch framework [28]. The experimented SNNs are based on the network structure described in Sec. 3.4. Only 8 time steps are used to demonstrate the proposed ultra low-latency spiking neural network. No refractory period is used. SGD[29] is applied as the optimizer. If not otherwise specified, the accuracy in this paper refers to the best results obtained by repeating the experiments five times. Please refer to the Supplementary Materials for experiment details.

### Comparison with existing works

The comparison results on FashionMnist, CIFAR-10, CIFAR-100, TinyImageNet, and DVS-CIFAR10 are shown in Tab. 1.

For the FashionMnist dataset, we use ResNet18[30] as the network structure. Tab. 1 shows that the proposed SLSSNN method outperforms all other methods on FashionMnist with 8 time steps, based on 5 runs of experiments. Although some direct training methods such as [31] use smaller time steps than ours, our method can achieve better performances. For the FashionMnist dataset, which is a simple dataset, our proposed method still obtained 0.51%-5.19% accuracy improvement.

For the Cifar-10 and Cifar-100 datasets, we use VGG16[9] and ResNet18[30] as the network architectures. On Cifar-10 and Cifar-100, our proposed SLSSNN achieves the highest accuracy above all existing approaches based on 5 runs of experiments. For Cifar-10, Our proposed SLSSNN based on VGG16 and ResNet18 architecture obtains \(93.32\%\) and \(94.08\%\) test accuracy with only 8 time steps, respectively. Though our proposed SLSSNN based on VGG16 architecture obtain \(93.32\%\) accuracy, which is a little lower than [32] on Cifar-100. We only train the network in 100 epochs, while [32] needs 200-300 epochs. In addition, Our proposed SLSSNN based on ResNet18 can obtain high accuracy than other methods.

For the TinyImageNet dataset, we still used VGG16[9] and ResNet18[30] as the network architectures. Compared to the ANN-to-SNN methods, which apply the same VGG16 architecture, our method obtains \(4.22\%\)-\(6.44\%\) accuracy improvement with fewer time steps. If we apply a more powerful architecture ResNet18, SLSSNN can obtain \(59.38\%\) accuracy, which improves the accuracy by \(1.58\%\)-\(12.59\%\) compared with other methods.

W also test our SLSSNN on the neuromorphic DVS-CIFAR10. we adopt the VGG-11[8] and VGG16[9] architectures and conduct 5 runs of experiments for each architecture in 50 epochs. As Tab. 1 shows, the proposed method outperforms other SOTA methods with low latency using VGG11 and VGG16.

### Performance analysis

Since the initial threshold voltage and the length of the spike trains directly affect the loss of information during the encoding and decoding process in STCB, we study the influence of initial threshold voltages and length of spike trains on the accuracy of SNNs using FashionMnist, Cifar10, Cifar100, TinyImageNet, and DVS-Cifar10.
#### 4.3.1 Influence of initial threshold voltages

We chose different voltages (0.1, 0.2, 0.4, 0.6, 0.8, 1, 2, 4, 8mV) as the initial threshold voltage of SLSSNNs. For static datasets, FashionMnist, Cifar10, Cifar100, and TinyImageNet, we use ResNet18 as the network structure. For the neuromorphic dataset DVS-Cifar10, VGG16 is applied. As Fig. 5(a) shows, there is a significant accuracy drop when the initial threshold voltage is less than 0.8 mV. SLSSNN can obtain a reasonable accuracy for the static datasets when the initial threshold voltage is larger than 1. We further study the distribution of the input of SLSSNNs and find that the small initial threshold voltage may cause serious information loss during the encoding process in STCB. For FashionMnist, Cifar10, Cifar100, and TinyImageNet datasets, the input of STCB is in the range of [-1, 1]. When the initial threshold voltage is too small, such as 0.1, the part of the input larger than 0.1 will be encoded into one spike, which may cause profound information loss. For DVS-Cifar10, the input of STCB is in the range of [-2, 2]. So the initial threshold voltage is less than 2 will make a significant accuracy drop.

\begin{table}
\begin{tabular}{c c c c c c c c} \hline Dataset & Model & Methods & Architecture & Time Steps & mean(\%) & stddev(\%) & Best(\%) \\ \hline \multirow{4}{*}{Fashion MNIST} & Zhang and Li[6] & SNN & 12cmv1,12cmv1 & 400 & 90.0 & 0.14 & 90.13 \\  & Cheng et al.[33] & SNN & 2cmv1,12cmv1 & 20 & 92.07 & - & - \\  & Xu et al [1] & SNN & 2cmv1,12cmv1 & 2 & - & - & 93.80 \\  & Fang, et al [34] & SNN & 2cmv,112cmv1 & 8 & - & - & 94.38 \\  & Kazuma Suekac et al [31] & SNN & ResNet18 & 1 & - & - & 94.81 \\  & Our model & SNN & ResNet18 & 8 & 95.05 & 0.26 & 95.32 \\ \hline \multirow{4}{*}{Cifar10} & Kim et al.[17] & ANN-to-SNN & VGG16 & 150 & - & - & 89.2 \\  & Zhang and Li[26] & SNN & 5cmv, 21cmv1 & 5 & - & - & 91.41 \\  & Rathi, et al[8] & ANN-to-SNN & ResNet20 & 250 & - & - & 92.22 \\  & Kundu, et al[9] & ANN-to-SNN & VGG16 & 10 & - & - & 92.53 \\  & Our model & SNN & VGG16 & 8 & 93.08 & 0.25 & 93.32 \\  & Our model & SNN & ResNet18 & 8 & 93.81 & 0.30 & 94.08 \\ \hline \multirow{4}{*}{Cifar100} & Shen et al.[35] & RNN & ResNet20 & 1 & - & - & 69.38 \\  & Kim et al.[7] & ANN-to-SNN & VGG16 & 200 & - & - & 62.3 \\  & Rathi et al.[8] & ANN-to-SNN & VGG11 & 125 & - & - & 67.87 \\  & Kazuma Suekac et al [31] & SNN & ResNet18 & 1 & - & - & 71.23 \\  & Kim, Li et al.[32] & SNN & NAS & 5 & 73.04 & - & 73.40 \\  & Our model & SNN & VGG16 & 8 & 71.89 & 0.55 & 72.05 \\  & Our model & SNN & ResNet18 & 8 & 74.05 & 0.28 & 74.31 \\ \hline \multirow{4}{*}{TinyImageNet} & Na et al.[36] & SNN & NAS & 8 & - & - & 46.79 \\  & Kim et al.[7] & ANN-to-SNN & VGG16 & 200 & - & - & 51.9 \\  & Kundu et al.[9] & ANN-to-SNN & VGG16 & 10 & - & - & 54.1 \\  & Kim, Li et al.[32] & SNN & NAS & 5 & - & - & 54.6 \\  & Kim and Panda[37] & SNN & VGG11 & 30 & - & - & 57.8 \\  & Our model & SNN & VGG16 & 8 & 58.14 & 0.21 & 58.34 \\  & Our model & SNN & ResNet18 & 8 & 59.10 & 0.25 & 59.38 \\ \hline \multirow{4}{*}{DVS-Cifar10} & Wu et al.[38] & SNN & VGG7 & 50 & - & - & 62.5 \\  & Wu, et al.[39] & SNN & 7cmv,21cmv1 & 20 & - & - & 65.59 \\  & Zheng, et al.[40] & SNN & ResNet19 & 10 & - & - & 67.80 \\  & Xu, et al.[4] & SNN & 6cmv,112cmv1 & 5 & - & - & 73.8 \\  & Fang, et al.[34] & SNN & 7cmv,21cmv1 & 20 & - & - & 74.80 \\  & Meng, et al.[3] & SNN & VGG11 & 20 & 75.03 & - & 75.42 \\  & Our model & SNN & VGG11 & 8 & 76.0 & 1.11 & 77.5 \\  & Our model & SNN & VGG16 & 8 & 77.5 & 0.53 & 77.9 \\ \hline \end{tabular}
\end{table}
Table 1: Compare with exciting work on different datasets.

Figure 5: Performance analysis. (a) Initial threshold voltages (b) length of spike trains
#### 4.3.2 Influence of the length of spike trains

We chose different lengths of spike trains (2, 4, 6, 8, 12, 16) to study the influence on the accuracy of SLSSNNs. The network architecture and datasets are the same as section 4.3.1. The initial threshold voltages are set as 0.8 and 2 for the static datasets (FashionMnist, Cifar10, Cifar100, and TinyImageNet datasets) and the neuromorphic dataset (TinyImageNet), respectively. Fig. 5(b) is the accuracy of SLSSNN with different lengths of spike trains on different datasets. When the length of spike trains is less than 4, the SLSSNN cannot work properly. For a fixed initial threshold voltage, the length of spike trains determines the encoding and decoding precision of STCB. So too short spike trains may cause the smaller input information will be lost, which causes SLSSNN accuracy to drop significantly, especially for the deeper network. Fig. 5(b) also shows that the accuracy of SLSSNN is not significantly increasing with the increase of the length of the spike train when the length is larger than 8. Longer spike trains mean more energy consumption. Therefore a suitable initial threshold voltage and length of the spike train can make a better tradeoff between accuracy and energy.

### Energy consumption analysis

In this section, we first study the fire rate and the number of spikes of each convolutional layer of ResNets18 and VGG16. Then we use FLOPs count to capture the computation energy to show the high energy efficiency of SLSSNNs.

#### 4.4.1 Fire rate and Number of spikes

We use Cifar10 and DVS-Cifar10 datasets to show the average fire rate and the number of spikes of each convolutional layer of ResNets18 and VGG16, respectively. As Fig. 6(a) and 6(b)shows, the

Figure 6: Average fire rate. (a) ResNet18 (Cifar10) (b)VGG16 (DVS-Cifar10)

Figure 7: Average number of spikes. (a) ResNet18 (Cifar10) (b)VGG16 (DVS-Cifar10)
average fire rate increases with the increasing spike train's length. Fig. 6(a) depicts the benefits of the short length of spike trains, achieving significantly lower spike activity for almost all convolutional layers. Fig. 7(a) and 7(b) show the average number of spikes of ResNet18 and VGG16. Because the number of spikes is determined by the fire rate and the length of spike trains, the number of each convolutional layer increases rapidly with the increase of the length of spike trains. The good news is that we do not need a very long spike train to obtain a competitive result.

#### 4.4.2 Floating Point Operations (FLOPs) \(\&\) computation Energy

In this section, we refer [9] to calculate the FLOPs and computation energy of SLSSNN. The detail is in the supplementary materials. For static datasets, we chose ResNet18 as the network architecture. For the neuromorphic dataset, we apply VGG18 as the network architecture. Fig. 8(a) and 8(b) shows the average FLOPs of ResNet18 and VGG16. Compared with ANN, SLSSNNs have fewer FLOPs when the length of the spike train is less than 6. Though SLSSNNs have more FLOPs when the spike train is longer than 6, the accumulation (AC) operation rather than the multiply-and-accumulate (MAC) operation is dominant due to the binary spiking information, which helps SLSSNN keep high energy efficiency. As Fig. 9 shows that the SLSSNNs have significant energy reduction. The energy efficiency can reach up to \(11.76\times\sim 36.79\times\), when the length of spike train is 8. Even if the length of the spike train is 16, SLSSNN still has \(6.54\times\sim 15.17\times\) energy reduction.

Figure 8: FLOPs of ResNet18 and VGG16. (a) Static Datasets (ResNet18) (b)Neuromorphic dataset (VGG16)

Figure 9: Comparison of computation energy
## 5 Discussions and Conclusion

This paper proposes a high energy efficiency spike-train level spiking neural network with spatio temporal conversion. Based on the idea of spatio temporal conversion, spatio-temporal conversion blocks (STCBs) are proposed to replace the convolutional and ReLU layers to keep the low power features of SNNs and improve the accuracy of SNNs. We chose ResNet18 and VGG16 as network architecture to evaluate the proposed SLSSNN on static and neuromorphic datasets, including Fashion-Mnist, Cifar10, Cifar100, TinyImageNet, and DVS-Cifar10. The experiment results show that our proposed SLSSNN outperforms the state-of-the-art accuracy on nearly all datasets, using fewer time steps and being highly energy-efficient. As for limitations, the proposed SLSSNNs may suffer from a certain degree of performance drop when the latency is extremely low(e.g., the length of the spike train is less than 4) since the SLSSNNs require enough spikes to transmit information between layers.
