Gaining the Sparse Rewards by Exploring Lottery Tickets in Spiking
  Neural Network
Deploying energy-efficient deep learning algorithms on computational-limited
devices, such as robots, is still a pressing issue for real-world applications.
Spiking Neural Networks (SNNs), a novel brain-inspired algorithm, offer a
promising solution due to their low-latency and low-energy properties over
traditional Artificial Neural Networks (ANNs). Despite their advantages, the
dense structure of deep SNNs can still result in extra energy consumption. The
Lottery Ticket Hypothesis (LTH) posits that within dense neural networks, there
exist winning Lottery Tickets (LTs), namely sub-networks, that can be obtained
without compromising performance. Inspired by this, this paper delves into the
spiking-based LTs (SLTs), examining their unique properties and potential for
extreme efficiency. Then, two significant sparse \textbf{\textit{Rewards}} are
gained through comprehensive explorations and meticulous experiments on SLTs
across various dense structures. Moreover, a sparse algorithm tailored for
spiking transformer structure, which incorporates convolution operations into
the Patch Embedding Projection (ConvPEP) module, has been proposed to achieve
Multi-level Sparsity (MultiSp). MultiSp refers to (1) Patch number sparsity;
(2) ConvPEP weights sparsity and binarization; and (3) ConvPEP activation layer
binarization. Extensive experiments demonstrate that our method achieves
extreme sparsity with only a slight performance decrease, paving the way for
deploying energy-efficient neural networks in robotics and beyond.
Hao Cheng, Jiahang Cao, Erjia Xiao, Mengshu Sun, Renjing Xu
# Gaining the Sparse Rewards by Exploring Binary Lottery Tickets in Spiking Neural Networks

###### Abstract

Spiking Neural Network (SNN) as a brain-inspired strategy receives lots of attention because of the high-sparsity and low-power properties derived from its inherent spiking information state. To further improve the efficiency of SNN, some works declare that the Lottery Tickets (LTs) Hypothesis, which indicates that the Artificial Neural Network (ANN) contains a subnetwork without sacrificing the performance of the original network, also exists in SNN. However, the spiking information handled by SNN has a natural similarity and affinity with binarization in sparsification. Therefore, to further explore SNN efficiency, this paper focuses on _(1) the presence or absence of LTs in the binary SNN, and (2) whether the spiking mechanism is a superior strategy in terms of handling binary information compared to simple model binarization_. To certify these consumptions, a sparse training method is proposed to find Binary Weights Spiking Lottery Tickets (BinW-SLT) under different network structures. Through comprehensive evaluations, we show that BinW-SLT could attain up to \(+5.86\%\) and \(+3.17\%\) improvement on CIFAR-10 and CIFAR-100 compared with binary LTs, as well as achieve 1.86x and 8.92x energy saving compared with full-precision SNN and ANN.

Hao Cheng\({}^{1}\), Jiahang Cao\({}^{1}\), Erjia Xiao\({}^{1}\), Pu Zhao\({}^{2}\), Mengshu Sun\({}^{3}\),

Jiaxu Wang\({}^{1}\), Jize Zhang\({}^{4}\), Xue Lin\({}^{2}\), Bhavya Kailkhura\({}^{5}\), Kaidi Xu\({}^{6}\), Renjing Xu\({}^{1}\)\({}^{1}\) The Hong Kong University of Science and Technology (Guangzhou)

\({}^{2}\) Northeastern University, \({}^{3}\) Beijing University of Technology

\({}^{4}\) The Hong Kong University of Science and Technology

\({}^{5}\) Lawrence Livermore National Laboratory, \({}^{6}\) Drexel University

Spiking Neural Networks, Lottery Tickets Hypothesis, Binary Neural Network

## 1 Introduction

Although many deep learning algorithms have performed well on different tasks in the current AI research field, some practical problems still need to be resolved. Significantly, the redundancy of model structure and computational burden greatly prevents the democratization of AI. Model compression has been widely combined with various existing AI algorithms as a very effective method. Model pruning [1] and quantization [2] as two representative compressing methods could make the original network pursue more lightweight structures and facilitate their implementations on resource-constrained application systems, such as smartphone or Field Programmable Gate Arrays (FPGA).

About model pruning, initially, all of them need to go through 1) original model training, 2) model pruning, and 3) post-fine-tuning to find a sub-model according to various pruning rules, which could be categorized as irregular pruning [1] and regular pruning [3; 4; 5]. And the regular pruning could be further classified as filter pruning [3], column pruning [4], and block-based pruning [5]. Recently, ANNs are been proven to have specific sparser but not sacrificing original performance subnetworks named LTs Hypothesis [6]. This finding makes sparse training [7; 8; 9] possible, which makes it more convenient and efficient to find sub-networks without the need for complex pretraining, pruning, and retraining collocation techniques. Model quantization [10] could quantize the original 32-bit model toward the 16-bit to 2-bit state. Among them, converting 32-bit toward 2-bit is the extreme situation of quantization that could be termed as the Binary Neural Network (BNN) [10]. While adopting quantization or binarization to obtain a sparse model, the corresponding processing objects can be weights and activations. However, weights quantizing or binarizing can achieve a smaller performance loss than handling both. Furthermore, about combining LTs with binarization, the Multi-prize Lottery Tickets Hypothesis (MPTs) [9] certifies that the lottery tickets or subnetworks remain in BNN.

SNN, as the most promising third-generation neural network, has received lots of attention due to its distinctive properties of high biological plausibility, temporal information processing capability, inherent binary (spiking) information processing superiority, and energy saving. For training SNN, there are three main approaches, including (1) ANN to SNN conversions [11], (2) direct training [12], and (3) local training [13]. Although it is more computationally and energy efficient than traditional ANN, further improving its sparsity and efficiency is important to be more scalable in resource-limited hardware scenarios. There has already been some work on SNN pruning [14] and binarization [15; 16]. Additionally, Kim et al. [17] claimed that the LTs also exists in SNN.
However, the existence or non-existence of LTs in the binary case of SNN is the pending issue of the most interest to this paper. Furthermore, because of the inherent advantage of SNN in processing spiking information that could be assumed as a special binary form, the specific performance of binarized spiking LTs is also a point of great curiosity for this paper.

Therefore, two important issues to be investigated are how to discover Binary Weights Lottery Ticket (BinW-SLT) more expeditiously and how to analyze the properties of BinW-SLT comprehensively. We propose a new sparse training method that could efficiently find BinW-SLT without model training to solve these two issues. With this training method, we can prove that the LTs exists in the binary SNN case. Furthermore, compared to ANN with simultaneous binary weights and activations, BinW-SLT shows their advantage in processing binary information and attains up to \(+5.86\%\) and \(+3.17\%\) improvement in CIFAR-10 and CIFAR-100 compared with traditional submodel of BNN with binary weights and activations (BNN-BinAct) [6, 10]. BinW-SLT also maintains a better computational energy advantage of over 1.86x and 8.92x compared to full-precision SNN and ANN.

## 2 Methods and Experiments

### Binary Weights Spking Lottery Tickets (BinW-SLT)

**SNN Fundamentals:** Different from ANN, SNN specializes in the processing of spiking information. In this paper, we adopt the widely used Leaky Integrate-and-Fire (LIF) model [18], which is suitable to characterize the dynamic process of spike generation and can be defined as:

\[\tau\frac{\mathrm{d}V(t)}{\mathrm{d}t}=-(V(t)-V_{reset})+I(t) \tag{1}\]

where \(I(t)\) represents the input synaptic current at time \(t\) to charge up to produce a membrane potential \(V(t),\tau\) is the time constant. When the membrane potential exceeds the threshold \(V_{th}\), the neuron will trigger a spike and reset its membrane potential to a value \(V_{reset}\) (\(V_{reset}<V_{th}\)). The LIF neuron achieves a balance between computing cost and biological plausibility. In practice, the dynamics must be discretized to facilitate reasoning and training. The discretized version of the LIF model can be described as:

\[U[n] =e^{\frac{1}{\tau}}V[n-1]+(1-e^{\frac{1}{\lambda}})I[n] \tag{2}\] \[S[n] =\Theta(U[n]-V_{th})\] (3) \[V[n] =U[n](1-S[n])+V_{reset}S[n] \tag{4}\]

Where \(n\) is the discrete timestep, \(U[n]\) is the membrane potential before reset, \(S[n]\) denotes the output spike, which equals 1 when there is a spike and 0 otherwise, \(\Theta(x)\) is the Heaviside step function, \(V[n]\) represents the membrane potential after triggering a spike.

**Finding Methods:** To explore BinW-SLT, we refer to effective MPTs [9] method that could find binary Spiking LTs without suffering any weight training. The main work we must do is transfer MPTs to the binary state. This transferring process contains two issues that need special attention: _(1) SNN do not need binarize activation since the LIF as the method of processing spiking information is already a special case of binarization; 2) Compared with ANN, SNN has more parameters, such as timestep \(T\), and decay rate \(\lambda\), that will affect final performance and need to be considered._ Therefore, to make the original MPTs satisfy these two issues, the optimizing formula could be rewritten as:

\[min_{\alpha}||g(LIF(x;\alpha(M\odot sign(w))))-f(x;W^{*})|| \tag{5}\]

where \(\alpha\in\mathbb{R}\) is the Gain term necessary to perform binary subnetworks well. \(f(x;W^{*})\) is the target original network with optimized weights \(W^{*}\) that we wish to approximate.

The specific BinW-SLT finding measure is present in Algorithm 1. The optimizing objects still score \(s\) and mask \(M\). However, compared with MPTs, some updates that adapt to SNN are applied. In the input, Step-1 to Step-5 are the initialization of different parameters. The best spiking LTs choose from Step-6 to Step-13, including updating and sorting pruning scores \(s\) and mask \(M\) updating processes. Finally, the sparse BinW-SLT \(g(LIF(x;\alpha(M\odot sign(w))))\) with best performance is screened out.

```
1:Input: SNN \(g(LIF(x;))\) with 2-bit spiking activation; Loss function \(L\); Training data \(\{(x^{(i)},y^{(i)})\}_{j=1}^{N}\); Dataset size \(N\); SNN parameters \(N_{snn}\).
2:Initialize SNN: Pruning rate \(r_{p}\); Timestep \(t\); Decay rate \(\lambda\).
3:Initialize spiking LTs Parameters: SNN weights \(w\); Pruning scores \(s\); Layerwise masks \(M\in\{0,1\}\).
4:Initialize spiking LTs Weights: \(sign(w)\)
5:Initialize Gain Term: \(\alpha\leftarrow||M\odot w||/||M||\)
6:for k = 1 to \(N_{epochs}\)do
7:for\(r_{p}\), \(t\) and \(\lambda\)do
8:\(s\gets s-\eta\nabla_{s}\ell(L(\alpha\cdot M\odot sign(w)))\)
9:\(Proj_{[0,1]}\leftarrow\) Sorting \(s\) according to \(r_{p}N_{snn}\)
10:\(M\gets M\odot Proj_{[0,1]},\alpha\leftarrow||M\odot\textbf{w}||_{1}/||M||_{1}\)
11:endfor
12:endfor
13:Output: Return \(g(LIF(x;\alpha(M\odot sign(w))))\)
```

**Algorithm 1** Finding BinW-SLT

### Experiments

**Experimental Setting:** This paper uses various SNN structures as our basic models under two popular datasets, CIFAR-10 and CIFAR-100. In Cifar-10, Conv-4 with 4 convolutional layers, VGG-9, VGG-11, and ResNet-19 based SNN are utilized here. In CIFAR-100, we choose VGG-11 and ResNet-19 SNNs to explore our BinW-SLT. About training SNN, Adam optimizer with a learning rate of \(0.1\) and surrogate gradient [12] are adopted. In finding the BinW-SLT process, we adopt commonly used LIF neurons with
timestep \(T=4\) and decay rate \(\lambda=0.99\). Additionally, to explore the relationship between different SNN parameters and the performance of BinW-SLT, \(T=1,2,6,8\) and \(\lambda=0.8,0.7,0.6,0.5,0.4,0.3,0.2\) are further adopted here. All conducting experiments are implemented based on Pytorch and SpikingJelly [19].

**General Performance of BinW-SLT:** According to the sub-figure (a) in Fig. 2, in CIFAR-10, among most pruning rate regions, BinW-SLT could attain better accuracy compared with BNN with Binary Activation (BNN-BinAct). With respect to the pruning rate\(=40\%\) in Table 1, compared to BNN-BinAct, in CIFAR-10, BinW-SLT could achieve at most \(+5.61\%\), \(+5.79\%\), \(+4.68\%\) and \(+5.86\%\) improvement on CONV-4, VGG-9, VGG-11 and ResNet-19. About CIFAR-100 in Table 2, BinW-SLT could attain \(+3.17\%\) and \(+3.31\%\) enhancement on VGG-11 and ResNet-19. We also adopt the full-precision original ANN and original SNN as our baselines. For the performance of CIFAR-10 in Fig. 2, from low to high pruning rate, BinW-SLT could even exceed full-precision SNN in most cases and approaches full-precision ANN to the greatest extent. Specifically, compared with full-precision SNN/ANN, for the CONV-4, VGG-9, VGG-11 and ResNet-19 in CIFAR-10, BinW-SLT with T=4 could generate \(+0.24\%\), \(-0.95\%\), \(+0.05\%\), \(-1.35\%\), \(+0.56\%\), \(-0.34\%\) and \(+0.04\%\), \(-0.23\%\) modification when the pruning rate is \(40\%\). For CIFAR-100 in Table 2, compared with SNN, BinW-SLT also maintains a relatively low-performance decrease, and the gaps in VGG-11 and ResNet-19 are only \(-1.2\%\) and \(-0.54\%\). Additionally, to have a comprehensive analysis, we also apply the same corresponding model with Timestep \(T=1\), BinW-SLT (T=1), as a fairer comparing object in the sub-figure (a) of Fig. 2. In this case, consistent with the performance scenarios analyzed above, the BinW-SLT in each structure still exceeds the BNN-BinAct and approaches the full-precision SNN and ANN.

According to the results in Fig. 2, Table 1 and Table 2, we could get a straightforward concept: _In the case of extreme sparsity that involves both pruning and binarization, a spiking mechanism, like LIF, can better preserve the original information of activations than 0-1 processing of activation values._ This can be proved by the better performance of BinW-SLT compared with BNN-BinAcT under different pruning rates, from \(r_{p}=20\%\) to \(r_{p}=90\%\), and timesteps, \(t=4\) or \(1\). To give a more intuitive analysis, we draw Fig. 1 to illustrate the activation map of full-precision ANN, BNN-BinAct, and BinW-SLT from left to right. The BinW-SLT could obtain a higher-level summary from the ANN activation map (Left). Compared with BNN-BinAct, which performs the naive binary operation on the ANN, the activation map of BinW-SLT is sparser and more representative of the original information.

**The Effect of Timestep and Decay Rate:** In Fig.2 (b) and (c), we explore the effect of SNN parameters timestep \(t\) and decay rate \(\lambda\) on the performance of BinW-SLT. The experiments adopt CONV-4, VGG-9, VGG-11, and ResNet-19 with \(r_{p}=40\%,50\%,60\%\) as our basic structures, and \(t=4\) and \(\lambda=0.99\) as the main parameters. During exploration, while respectively keeping other parameters constant, we select \(T=1,2,4,6,8\) and \(\lambda=0.99,0.8,0.7,0.6,0.5,0.4,0.3,0.2\) as our main research objects.

According to Fig. 2 (b), on the basis that the performance of BinW-SLT is better than BNN-BinAct, the change of timestep \(t\) could lead up to \(+3.3\%\) (CONV-4), \(+1.22\%\) (VGG-9), \(+3.48\%\) (VGG-11) and \(+5.86\%\) (ResNet-19) increase compared to the full-precision SNN. This increase also confirms the theoretical mechanism of \(t\) in SNN that could be assumed as a multiple processing method of the same input representation.

And in Fig. 2 (c), following the changes in decay rate from \(0.99\) to \(0.2\), BinW-SLT still consistently outperforms BNN-BinAct and also maintains a performance advantage over full-precision SNN under certain value combinations of \(r_{p}\) and \(\lambda\). Concretely, BinW-SLT could attain at most

\begin{table}
\begin{tabular}{c c c} \hline \hline
**Architecture** & **Method** & **CIFAR-10 Acc(\%)** \\ \hline \multirow{3}{*}{_CONV-4_ (2.43M)_} & ANN/SNN & 84.60/83.41 \\  & _BNN-BinAct_ & _79.21_ \\  & **BinW-SLT** & **83.65 (+5.61)** \\ \hline \multirow{3}{*}{_VGG-9_ (2.26M)_} & ANN/SNN & 86.70/85.30 \\  & _BNN-BinAct_ & _80.68_ \\  & **BinW-SLT** & **85.35 (+5.79)** \\ \hline \multirow{3}{*}{_VGG-11_ (5.27M)_} & ANN/SNN & 88.10/87.20 \\  & _BNN-BinAct_ & _83.84_ \\ \cline{1-1}  & **BinW-SLT** & **87.76 (+4.68)** \\ \hline \multirow{3}{*}{_ResNet-19_ (12.63M)_} & ANN/SNN & 92.05/91.78 \\ \cline{1-1}  & _BNN-BinAct_ & _86.74_ \\ \cline{1-1}  & **BinW-SLT** & **91.82 (+5.86)** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation results on CIFAR-10. BinW-SLT is mainly compared with BNN-BinAct.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Architecture** & **Method** & \begin{tabular}{c} **OPs** \\ **(G)** \\ \end{tabular} & \begin{tabular}{c} **Power** \\ **(mJ)** \\ \end{tabular} & 
\begin{tabular}{c} **CIFAR-100** \\ **Acc(\%)** \\ \end{tabular} \\ \hline \multirow{3}{*}{_VGG-11_ (5.27M)_} & ANN & 0.209 & 0.966 & 68.64 \\  & SNN & 0.140 & 0.131 & 62.94 \\  & _BNN-BinAct_ & _0.209_ & _0.966_ & _58.57_ \\  & **BinW-SLT** & **0.132** & **0.127** & **61.74 (+3.17)** \\ \hline \multirow{3}{*}{_ResNet-19_ (12.63M)_} & ANN & 2.223 & 10.225 & 86.74 \\  & SNN & 1.778 & 1.609 & 64.27 \\ \cline{1-1}  & _BNN-BinAct_ & _2.223_ & _10.225_ & _60.60_ \\ \cline{1-1}  & **BinW-SLT** & **1.689** & **1.528** & **63.73 (+3.31)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation results on CIFAR-100. BinW-SLT is mainly compared with BNN-BinAct. OPs (G) and Power (mJ) are corresponding to computing and power consumption.

Figure 1: Activation map of the 128 last layer neuron representation of (a) ANN, (b) BNN-BinAct (c) BinW-SLT.
\(+2.69\%\) (CONV-4), \(+1.23\%\) (VGG-9), \(+1.91\%\) (VGG-11) and \(+4.33\%\) (ResNet-19) performance fluctuation. Additionally, regardless of the various \(p_{r}\), the effect of \(\lambda\) on the performance of BinW-SLT is related to the particular model size. In a word, a smaller model could also tolerate a smaller decay rate \(\lambda\). Initially, CONV-4, VGG-9, VGG-11, and ResNet-19 experience some degree of accuracy rise or stabilization. However, by decreasing \(r_{\text{p}}\), the accuracy would also translate to the decrease under different \(\lambda\). When the model size is small to large, the corresponding \(\lambda\) are also the increasing values \(0.3\), \(0.4\), \(0.6\), and \(0.7\).

**The Effect of Fine Tuning:** Since the finding of BinW-SLT does not adopt weight training, BinW-SLT could also be assumed as a unique initialization strategy for the precisovery of sparse SLTs, and its performance could be further improved according to further fine-tuning. In the right subfigure of Fig. 3, fine-tuned BinW-SLT at different \(p_{r}\) could produce a tangible improvement under the original trend.

**Theoretical Power Consumption:** To better understand the effects of the BinW-SLT on energy using, we estimate the theoretical power consumption on neuromorphic chips based on previous studies [20]. We use OPs as a metric to judge computational consumption. For ANNs, OPs correspond to floating point operations (FLOPs), while SNNs perform synaptic operations (SOPs), which is defined as [21]:

\[SOPs(l)=fr\times T\times FLOPs(l) \tag{6}\]

where \(l\) is a block/layer, \(fr\) is the firing rate of \(l\) and \(T\) is the timestep. After assuming BinW-SLT is implemented on a 45nm hardware [21] with each FLOP and SOP are \(4.6pJ\) and \(0.9pJ\), respectively, then we can get the theoretical energy consumption calculated by:

\[E_{SNN} =E_{flop}\times\mathrm{FLOP}^{1}_{\mathrm{SConv}}\] \[+E_{sop}\times\left(\sum_{n=2}^{N}\mathrm{SOP}^{n}_{\mathrm{SConv }}+\sum_{m=1}^{M}\mathrm{SOP}^{m}_{\mathrm{SFC}}\right) \tag{7}\]

where \(N\) and \(M\) denotes the number of spike convolutional (\(SConv\)) and fully connected (\(SFC\)) layers. We first sum up the SOPs of all \(SConv\) layers (except the first layer), and \(SFC\) layers and multiply by \(E_{flop}\). For the first convolutional layer of SNN, we calculate the energy consumption utilizing FLOPs due to the spike encoding operation performed here. We illustrate the results of our BinW-SLT, BNN-BinAct, full-precision ANN, and SNN in Table 2. Also, as exemplified in Fig. 3, BinW-SLT benefits from the energy advantage of up to 1.86x and 8.92x compared to standard SNN and ANN.

## 3 Conclusion

The main focus of this paper is the Binary Weights Spiking Lottery Ticket, abbreviated as BinW-SLT. The analysis of BinW-SLT certifies that (1) the presence of LTs under binary weights SNN; (2) the spiking mechanism could generate a higher-level binary summary and produce binary information that is superior to simple 0-1 processing of activation; (3) As a special kind of initialization, the performance of BinW-SLT has the potential to be further enhanced by fine-tuning. Additionally, we introduce theoretical energy consumption, and the results indicate that our BinW-SLT model can achieve energy savings of up to 1.86x and 8.92x compared to full-precision SNN and ANN, respectively.

Figure 3: Left: Comparison of normalized compute energy computed using (a) ANN, (b) SNN without any pruning, and (c) SNN with Spiking Lottery Ticket; Right: BinW-SLTs and their Fine-Tuning (FT) performance under different pruning rates.

Figure 2: The comparison of CIFAR-10 performance under the modification of (a) Pruning Rate; (b) Timestep; (c) Decay Rate.
## Acknowledgement

BK's work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE- AC52-07NA27344 and was supported by the LLNL-LDRD Program under Project No. 22-DR-009.
