MT-SNN: Enhance Spiking Neural Network with Multiple Thresholds
Spiking neural networks (SNNs), as a biology-inspired method mimicking the
spiking nature of brain neurons, is a promising energy-efficient alternative to
the traditional artificial neural networks (ANNs). The energy saving of SNNs is
mainly from multiplication free property brought by binarized intermediate
activations. In this paper, we proposed a Multiple Threshold (MT) approach to
alleviate the precision loss brought by the binarized activations, such that
SNNs can reach higher accuracy at fewer steps. We evaluate the approach on
CIFAR10, CIFAR100 and DVS-CIFAR10, and demonstrate that MT can promote SNNs
extensively, especially at early steps. For example, With MT,
Parametric-Leaky-Integrate-Fire(PLIF) based VGG net can even outperform the ANN
counterpart with 1 step.
Xiaoting Wang, Yanxiang Zhang, Yongzhe Zhang
# MT-SNN: Enhance Spiking Neural Network with Multiple Thresholds

###### Abstract

Spiking neural networks (SNNs), as a biology-inspired method mimicking the spiking nature of brain neurons, is a promising energy-efficient alternative to the traditional artificial neural networks (ANNs). The energy saving of SNNs is mainly from multiplication free property brought by binarized intermediate activations. In this paper, we proposed a Multiple Threshold (MT) approach to alleviate the precision loss brought by the binarized activations, such that SNNs can reach higher accuracy at fewer steps. We evaluate the approach on CIFAR10, CIFAR100 and DVS-CIFAR10, and demonstrate that MT can promote SNNs extensively, especially at early steps. For example, With MT, Parametric-Leaky-Integrate-Fire(PLIF) based VGG net can even outperform the ANN counterpart with 1 step.

## 1 Introduction

The development of artificial neural networks (ANNs) especially deep learning in recent years brought remarkable successes in lots of domains like computer vision [1][2], natural language processing [3][4][5], multi-modal[6][7] etc. However, computational costs of ANNs are too high for even one single application. For example, a standard convolution neural network to classify ImageNet-1k dataset will expend about 250 watts[8], while by contrast, only 20 watts are required by human brain to process various tasks simultaneously[9].

Spiking Neural Networks (SNNs) are considered as the next generation neural models for its closer mechanism to the biological neurons in the brain[10]. Recently SNNs attract increasing attention due to their abilities such as temporal information processing, energy efficiency [8] and high biological plausibility [11]. By leveraging binary spike signals, SNNs can achieve low-energy consumption by multiplication-free inference and avoiding computing zero values of inputs or activation [8]. Neuromorphic hadwares such as TrueNorth[12], Lohi[13] and Tianjic[14] demonstrated that SNNs is capable of saving energy by orders of magnitude.

However, even though with these promising inherent properties, SNNs are not widely adopted by the main-stream machine learning problems. We argue three obvious obstacles should account for this. Firstly, quality degradation caused by information loss is unavoidable by replacing the float activations with binary spikes. Previously the problem is severe due to lacking efficient learning algorithms, surrogate gradient[15] fills the gap to a large extend. But it still exists when limiting time steps. Secondly, many popular problems are not with temporal information, pre-processing is necessary to make them solvable by SNNs. However, it's inefficient if a problem is converted from single step to multiple steps. For example, in static image classification tasks, the static image is duplicated into multiple copies to adapt the SNNs, multiple steps indeed boost the quality while reducing the efficiency of training and inference. Thirdly, SNNs are constraint by both software and hardware, modern popular ML frameworks such as Pytorch [16],
Jax [17] and Tensorflow [18] don't provide efficient and general instructions or high-level functions to accelerate the convolution with 0/1 spike activations [19]. And current popular devices like TPU, GPU are not specifically optimized for multiplication free operations.

In this work, we propose a multiple threshold (MT) algorithm at Leak-Integrate-Fire(LIF) cell to partly recover the information loss brought by replacing floating-point activations with 0/1 spikes. The difference between one single threshold (ST) and MT can be seen in Fig 1 and Fig 2. And we find the algorithm is effective in enhancing the quality of SNNs by both achieving better accuracy at same time steps and reaching similar accuracy with fewer time steps. The latter one reduces the burden of training SNNs on current hardwares. Furthermore, the trained model can still be deployed on neuromorphic hardwares without violating the nature of spikes.

Our contributions can be summarized as follows:

* We propose the multiple threshold (MT) algorithm as an augmentation to partly reserve precision in spike activations. MT can be viewed as a widget used in the fire stage of LIF cells such that can be deployed in SNNs of various architectures. we demonstrate MT is a robust way to improve quality of SNNs.
* We evaluate the MT algorithm on Parametric Leak Integrate Fire (PLIF) based VGG and ResNet on CIFAR-10, CIFAR-100 and DVS-CIFAR10 datasets, Results show that MT is highly effective and versatile. Our implemented SNNs with MT can exceed state-of-the-art accuracy of similar architectures, using fewer steps.

## 2 Related Works

### Learning Algorithms

After the emergence of SNNs, how to train the models effectively stays as an active area. The earliest learning algorithms are based on biological plausible local learning rules, like Hebbian learning [20] and Spike-Timing-Dependent Plasticity

Figure 1: Spikes over Single Threshold (ST), \(\mu\) is membrane and \(o\) is output spike.

Figure 2: Spikes over Multiple Thresholds (MT), \(\mu\) is membrane and \(o\) is output spike.
[21]. However, such methods are only suitable for shallow SNNs, and the performance is far below methods mentioned below.

ANN-To-SNN methods are capable of obtaining high-quality SNNs by leveraging the knowledge from well-trained ANNs and converting the ReLU activations to spike activations based on rate coding[22][23]. However, achieving near lossless conversion requires a considerable amount of time steps (\(>200\)) to accumulate the spikes, which may significantly increase the latency.

Backpropagation with surrogate gradient is widely used in recent researches to achieve high-quality deep models [24][25][26]. The utilization of surrogate gradient function enables end-to-end backpropagation in SNN training, thus training knowledge of ANNs can be transferred to SNNs. [27][15] systematically studied the remarkable robustness of the surrogate gradient based algorithms and demonstrated that SNNs trained with surrogate gradient can achieve competitive performance with ANNs. Backpropagation with surrogate functions requires much fewer steps compared to ANN-To-SNN methods as it's not dependent of rate coding [28].

### Model architecture

Backpropagation with surrogate gradient unlocked the model architecture migration from ANNs to SNNs. [28] initially applied direct training of various sized deep convolutional neural networks on both static and neuromorphic datasets with normalization and rate coding. [29] explored ResNet with time-dependent batch normalization and validated the architecture on ImageNet for the first time. [30][31] proposed spike-element-wise ResNet and membrane based ResNet respectively which pushed the depth of the architecture to more than 100 layers. [32] introduced attention mechanisms into SNNs and achieved comparable and even better performance compared to the ANN counterpart on large-scale datasets. Furthermore, [33] proposed Spiking Transformer by combining self-attention with spiking neural network. There will be more works on such model migrations in the foreseeable future.

### Information Loss

Information loss in SNNs attracts increasing attentions recently. In [34], Soft Reset is leveraged to reserve the information in the residual potential, and Membrane Potential Rectifier is proposed to alleviate the precision loss in quantization. [35] leveraged a re-parameterization technique to decouple training and inference stage, and enhance the presentation capacity b learning real-valued spikes during training and transferring the capacity to model parameters by not sharing convolution kernels. The MT approach in this paper falls into the same domain with [34][35], which provides another option to reduce information loss in spike activations.

## 3 Preliminary

### Parametric Leaky Integrate and Fire Model

Parametric Leaky Integrate and Fire (PLIF) neuron model[36] is adopted as the basic unit in the model architectures in this paper. PLIF includes the following discrete time equations:

\[H[t]=f(V[t-1],X[t]), \tag{1}\] \[S[t]=\Theta(H[t]-V_{th}),\] (2) \[V[t]=H[t](1-S[t])+V_{reset}S[t] \tag{3}\]

Where X[t] is the input current at time-step t from the previous layer, H[t] and V[t] represent the membrane potential after neuronal dynamics and spike trigger respectively. \(V_{th}\) is the firing threshold. \(\Theta(x)\) is the Heaviside step function which is defined by

\[\Theta(x)=\left\{\begin{array}{cc}1&x\geq 0\\ 0&x<0\end{array}\right. \tag{4}\]

\(V_{reset}\) denotes the reset potential. The function in Eq 1 describes the neuronal dynamics, which can represent different model with different forms. Eq 5 and Eq 6 describe the Integrate-and-Fire(IF) and Leaky Integrate-and-Fire(LIF) respectively.

\[H[t]=V[t-1]+X[t] \tag{5}\]
\[H[t]=V[t-1]+\frac{1}{\tau}(X[t]-(V[t-1]-V_{reset})) \tag{6}\]

In Eq 6, the \(\tau\) is the membrane time constant, [36] made it learnable with Eq 7.

\[\tau=1+exp(-a)\in(1,+\infty) \tag{7}\]

Where \(a\) is a learnable variable.

Furthermore, when \(V_{reset}=0\), the PLIF neuron becomes

\[H[t]=(1-\frac{1}{\tau})V_{t-1}+\frac{1}{\tau}X_{t} \tag{8}\]

which is analogous to the input gate and forget gate in Long Short-Term Memory(LSTM) networks [37]. In this paper, We set \(V_{reset}=0\) and implement the PLIF model with formula in Eq 8.

### Surrogate Function

The surrogate function used in this paper is the popular rectangular function proposed by [28], given by Eq below.

\[\frac{\partial S[t]}{\partial H[t]}=\frac{1}{a}sign(|H[t]-V_{th}|\leq\frac{a }{2}) \tag{9}\]

Where a is a hyper-parameter and usually set to 1.

### Outputs

Based on whether to fire at the last layer, there are naturally two kinds of outputs:

1. Output the membrane of the last layer directly without firing such that the loss can be computed on float values as the ANNs, which is widely used by [19][29][38] etc.
2. Output the spikes across time steps. This method inherently loss some precision especially with fewer time steps. A voting layer is proposed by [36] to alleviate the problem.

The loss functions are computed on the mean output across time steps of the model, classic classification losses such as Softmax or Mean Squared Error(MSE) can all be applied to the training of SNNs.

In practices, we found that the two outputs perform quite different on various datasets with various number of classes, which is illustrated in the experiment section.

## 4 Multi-Thresholds SNN

It's commonly observed that the quality of SNNs can be boosted by increasing time steps. Fig 3 below demonstrated this phenomenon. In Fig 3, the yellow line is the corresponding ANN with the same architecture of SNNs, and the red line is the SNNs with ST. It's observed that SNNs with ST require at least 2 steps to recover the same accuracy of its ANN counterpart.

We argue that compared to the ANNs, the precision of middle outputs of SNNs is lost in replacing float values with spikes and compensated on multiple time steps.

To overcome the precision loss problem in the intermediate activations in SNNs, we propose a Multi-Threshold (MT) approach to partly preserve the precision, the idea is illustrated in Fig 1 and Fig 2. In Fig 1, one threshold in red line split the input membrane into two values 0 and 1, in this process, the information about how much the membrane is greater / smaller than the threshold is completely lost. In Fig 2, two auxiliary thresholds are added, one is smaller in blue line and the other one is larger in yellow line. Two more series of spikes are obtained based on the auxiliary thresholds, the result that adds up the three sequences of spikes is a more accurate representation of the original membrane.

To formulate the approach, a series of parameters are introduced in, namely \([\Delta_{1},\Delta_{2},...,\Delta_{n}]\), and we apply each \(\Delta\) to Eq 2:
\[S[t]_{\Delta_{i}}=\Theta(H[t]-V_{th}-\Delta_{i}),i\in[1,2,...,n] \tag{10}\]

And Eq 3 is still used to update the membrane.

After obtaining all spikes for each threshold, the final outputs are obtained by

\[S[t]_{sum}=S[t]+\Sigma S[t]_{\Delta_{i}} \tag{11}\]

However, after applying multiple thresholds, from Eq 11, the outputs have other possible values other than 0/1, which breaks the binary property of SNNs, making the network not multiplication-free anymore. To keep the multiplication-free property, convolution layer of next layer can be involved in to perform an equivalent operation in Eq 12, right side of which is still multiplication-free.

\[Conv(S[t]_{sum})=Conv(S[t])+\Sigma Conv(S[t]_{\Delta_{i}}) \tag{12}\]

Generally, we can deploy left side of Eq 12 on current popular hardwares like TPU, GPU etc and right side on neuromorphic hardwares with multiplication-free properties.

## 5 Experiments

In this section, experimental results for the proposed method are reported on static datasets CIFAR10, CIFAR100 and neuromorphic datasets CIFAR10-DVS. ResNet and VGG network architectures are implemented to demonstrated the versatility of the method. The code is implemented with Tensorflow and all experiments are conducted on TPU.

### Network Architectures

ResNet and VGG networks are implemented for experiments with various configurations, the configurations of VGG-like architectures are listed in Table 1. The convolutions between pooling layers are viewed as a stage, same number of filters are used for convolutions in the same stage. The models end with two fully connected layers, the former layer is a hidden layer, while the latter layer is for output.

The VGG-8 model is the model architecture[36] for CIFAR10 dataset, 92% parameters of the model are from the first dense layer because of the large size of output of the last pooling layer. So VGG-9 is proposed to deepen the network as well as reduce the model size.

Each convolution layer is followed by a batch normalization layer and a PLIF layer to fire spikes, the first Conv-BN-PLIF is viewed as the encoder to convert image pixels into spikes.

Similarly, the configuration of ResNet architectures is listed in Table 2. The residual connections happened at the membrane level following [31]. After convolution layers, a pooling layer with large size is applied to convert the output image into 1-D, after which a fully-connected layer is appended as the output.

### Static Datasets Classification

CIFAR10 and CIFAR100 are the static datasets evaluated in this paper. The datasets contains 50K training and 10K test images with 32x32 pixels. Random horizontal flip, random rotation by 15 degree and random height, width shift by 10% fraction are used as the data augmentation approaches.

\begin{table}
\begin{tabular}{c c c c c c c} \hline Model Name & Dataset & \(N_{stages}\) & \(N_{conv}\) & Conv Filters & \(N_{fc}\) & First FC & \(N_{params}\) \\ \hline
*VGG-8 & CIFAR10(100) & 2 & 3,3 & 256,256 & 2 & 2048 & 36.72M \\ VGG-9 & CIFAR10(100) & 3 & 2,2,3 & 256,512,512 & 2 & 1024 & 19.72M \\ VGG-12 & DVS-CIFAR10 & 4 & 2,2,3,3,3 & 128,128,128,128,128 & 2 & 512 & 2.88M \\ \hline \end{tabular}
\end{table}
Table 1: Configurations of VGG architectures.

\begin{table}
\begin{tabular}{c c c c c c} \hline Model Name & Dataset & \(N_{layers}\) & \(N_{stages}\) & \(N_{filters}\) & \(N_{params}\) \\ \hline ResNet20 & CIFAR10(100) & 20 & 3 & 64,128,256 & 4.66M \\ \hline \end{tabular}
\end{table}
Table 2: Configurations of ResNet architectures.
In training, SGD optimizer is adopted with 0.9 momentum, the learning rate is initially set to 0.1 and decay by 0.1 after 100 epochs, after 500 epochs, the peak accuracy is reported.

The accuracy comparison on CIFAR10, CIFAR100 between SNNs with ST, SNNs with MT and the ANN counterparts are illustrated in Fig 3. It's observed that on both CIFAR10 and CIFAR100 datasets, all of the VGG-8, VGG-9 adn ResNet-20 models, all the time steps from 1 to 5, the accuracy of SNNs with MT outperforms SNNs with ST. And for all combinations except the ResNet-20 on CIFAR100, SNNs with MT can achieve close or even better accuracy at \(time\_step=1\).

The comparison between our results and previous state-of-the-art results of similar model architectures is listed in Table 3. The best accuracy of our proposed VGG-9 network are 94.74% and 75.53% respectively, with a 0.49% and 1.29% absolute accuracy improvement compared to the existed state-of-the-art results of simialr model architectures. Furthermore, the VGG-9 model can reach 94.71% and 74.79% accuracy when step is 2, which are already better than the previous state of the art results. Our implemented ResNet-20 network is not as good as the VGG-9 network, but still, on CIFAR10, the accuracy of MT-ResNet-20 can achieve 94.71% accuracy at \(time\_step=4\) and exceed the previous state of the art at \(time\_step=3\).

All the results on CIFAR10/CIFAR100 are from models trained by taking membrane as the output (See (A) of Sec 3.3), as we found that SNNs taking membrane as output is superior to models taking spikes (See (B) of Sec 3.3) as output on CIFAR10. And specifically for CIFAR100, it's difficult for model to converge if taking spikes and voting layer as output makes the model.

### Neuromorphic datasets classification

VGG-12 in Table 1 is adopted on DVS-Cifar10 dataset to verify the effectiveness of MT. The DVS-CIFAR10 dataset is processed in the same way as [36], namely converting the events to frames by accumulating events into T slices evenly, thus different frame datasets are obtained for different step T. No data augmentation approaches are involved in the experiments.

From Fig 4, it's observed that the accuracy is improved significantly at early stage especially when step is less than 4. The comparison of our proposed model with previous similar models are listed in Table 3, in which our final model accuracy is 76.3%, which is superior to the previous models.

However, when the number of steps is large, the gap between ST and MT is small, also, in the experiments on DVS-CIFAR10, we found that the accuracy shakes significantly even with same configuration. We argue that this is mainly caused by the fewer examples in the dataset. As currently large-scale DVS dataset is difficult to be acquired, we leave more explorations on DVS datasets as future work.

Figure 3: Performance comparison of ST and MT on CIFAR10 / CIFAR100 with various steps.
### Ablation Study

Extensive ablation studies are conducted to evaluate different MT settings. Firstly we study which part of the model needs MT, in (a)(b) of Fig 5, we apply MT on convolution layers only and both convolution and fully-connected layers, and evaluate the results on different steps. From (a), it's observed that applying MT on fully-connected layers improves accuracy at early steps. However, from (b), applying MT on fully-connected layers doesn't make significant changes on accuracy. We argue the deeper reason is that the fully-connected layers occupies 92% parameters of VGG-8 while 42.7% of VGG-9.

Secondly different deltas are explored on VGG-9 with \(step=1\), from (c) of Fig 5, it's observed that using negative deltas only performs more significant effect over positive deltas only, while combination of both positive and negative delta performs the best. And the accuracy curve is smoother when absolute delta is larger than 0.3, indicating no more information will be recovered when delta is too large.

Figure 4: Performance comparison of ST and MT on DVS-CIFAR10 with various steps.

\begin{table}
\begin{tabular}{c c c c} \hline Dataset & Name & Model & Steps & Accuracy(\%) \\ \hline \multirow{8}{*}{CIFAR10 / CIFAR100} & \multirow{8}{*}{ResNet-19} & 2 & 92.34 / - \\  & & & 4 & 92.92 / - \\  & & & 6 & 93.16 / - \\ \cline{2-4}  & \multirow{2}{*}{VGG-8} & 8 & 93.50 / - \\ \cline{2-4}  & & & 5 & 93.01 / 71.19 \\ \cline{2-4}  & \multirow{2}{*}{InfLoR-SNN[34]} & ResNet-20 & 5 & 93.01 / 71.19 \\  & & VGG-16 & 5 & 94.06 / 71.56 \\ \cline{2-4}  & \multirow{2}{*}{Real Spike[35]} & ResNet-20 & 5 & 93.01 / 66.60 \\  & & VGG-16 & 5 & 92.90 / 70.62 \\ \cline{2-4}  & \multirow{2}{*}{DSpike[19]} & \multirow{2}{*}{ResNet-18} & 2 & 93.13 / 71.68 \\  & & & 4 & 93.66 / 73.35 \\  & & & 6 & **94.25 / 74.24** \\ \cline{2-4}  & \multirow{2}{*}{Ours} & \multirow{2}{*}{MT-VGG-9} & 1 & 93.47 / 73.67 \\  & & & 2 & 94.71 / 74.79 \\  & & & 3 & 94.54 / 75.13 \\  & & & 4 & 94.51 / **75.53** \\  & & & 5 & **94.74 / 75.29** \\ \cline{2-4}  & \multirow{2}{*}{Ours} & \multirow{2}{*}{MT-ResNet-20} & 1 & 93.50 / 70.74 \\  & & & 2 & 93.84 / 72.05 \\  & & & 3 & 94.55 / 73.23 \\  & & & 4 & **94.71 / 73.75** \\  & & & 5 & 94.44 / 73.45 \\ \hline \multirow{8}{*}{DVS-CIFAR10} & \multirow{2}{*}{STDP-tdBN[29]} & ResNet-17/19 & 10 & 67.80 \\  & PLIF[36] & Conv+FC & 20 & 74.80 \\ \cline{1-1} \cline{2-4}  & SEW-ResNet[30] & SEW-ResNet & 16 & 74.40 \\ \cline{1-1} \cline{2-4}  & DSpike[19] & ResNet-18 & 10 & 75.40 \\ \cline{1-1} \cline{2-4}  & MS-ResNet20[31] & ResNet-20 & - & 75.56 \\ \cline{1-1} \cline{2-4}  & \multirow{2}{*}{Ours} & ST-VGG-12 & 5 & 75.90 \\ \cline{1-1} \cline{2-4}  & & MT-VGG-12 & 5 & **76.30** \\ \hline \end{tabular}
\end{table}
Table 3: Performance comparison of MT with SoTA methods.
## 6 Conclusion

In this paper, multiple threshold approach is proposed to augment the output of spiking layers, thus relieving the precision loss of SNNs comparing to float-based neural networks. We show that SNNs with multiple thresholds outperform state-of-the-art models with similar architectures, and tend to reach higher accuracy at fewer steps, thus providing us an option to make easier trade off between the number of steps and the quality.
