Resource Constrained Model Compression via Minimax Optimization for
  Spiking Neural Networks
Brain-inspired Spiking Neural Networks (SNNs) have the characteristics of
event-driven and high energy-efficient, which are different from traditional
Artificial Neural Networks (ANNs) when deployed on edge devices such as
neuromorphic chips. Most previous work focuses on SNNs training strategies to
improve model performance and brings larger and deeper network architectures.
It is difficult to deploy these complex networks on resource-limited edge
devices directly. To meet such demand, people compress SNNs very cautiously to
balance the performance and the computation efficiency. Existing compression
methods either iteratively pruned SNNs using weights norm magnitude or
formulated the problem as a sparse learning optimization. We propose an
improved end-to-end Minimax optimization method for this sparse learning
problem to better balance the model performance and the computation efficiency.
We also demonstrate that jointly applying compression and finetuning on SNNs is
better than sequentially, especially for extreme compression ratios. The
compressed SNN models achieved state-of-the-art (SOTA) performance on various
benchmark datasets and architectures. Our code is available at
https://github.com/chenjallen/Resource-Constrained-Compression-on-SNN.
Jue Chen, Huan Yuan, Jianchao Tan, Bin Chen, Chengru Song, Di Zhang
# Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networks

###### Abstract.

Brain-inspired Spiking Neural Networks (SNNs) have the characteristics of event-driven and high energy-efficient, which are different from traditional Artificial Neural Networks (ANNs) when deployed on edge devices such as neuromorphic chips. Most previous work focuses on SNNs training strategies to improve model performance and brings larger and deeper network architectures. It's difficult to deploy these complex networks on resource-limited edge devices directly. To meet such demand, people compress SNNs very cautiously to balance the performance and the computation efficiency. Existing compression methods either iteratively pruned SNNs using weights norm magnitude or formulated the problem as a sparse learning optimization. We propose an improved end-to-end Minimax optimization method for this sparse learning problem to better balance the model performance and the computation efficiency. We also demonstrate that jointly applying compression and finetuning on SNNs is better than sequentially, especially for extreme compression ratios. The compressed SNN models achieved state-of-the-art (SOTA) performance on various benchmark datasets and architectures. Our code is available at _[https://github.com/chenjallen/Resource-Constrained-Compression-on-SNN_](https://github.com/chenjallen/Resource-Constrained-Compression-on-SNN_).

Spiking Neural Networks, Model Compression, Sparse Pruning, Minimax Optimization 1
Footnote 1: The corresponding author.

2023

Spiking Neural Networks, Model Compression, Sparse Pruning, Minimax Optimization 2023

## 1. Introduction

As the third generation of neural networks (Wang et al., 2017), Spiking Neural Networks (SNNs) has been widely concerned in recent years. SNNs are the core of brain heuristic intelligence research, which have high biological interpretability and strong Spatio-temporal information processing ability (Shen et al., 2018). In addition, due to the inherent asynchrony and sparsity of spiking training, these types of networks can maintain relatively good performance as well as low power consumption, especially when combined with neuromorphic chips (Shen et al., 2018; Chen et al., 2019). With the development of efficient deep SNN training strategies, some useful network architectures are built, such as Spiking ResNet (Shen et al., 2018; Chen et al., 2019; Chen et al., 2019) and SEW ResNet (He et al., 2019) to improve the performance of SNNs. The parameters and computational energy of SNN models rapidly increase, while the computational resources of edge devices are usually limited. For example, SpiNNaker demonstrated to run networks with up to 250,000 neurons and 80 million synapses on a 48-chip board (Shen et al., 2018), which is still unable to run those more advanced SNNs. Thus, it is of great significance to compress SNNs before deploying them in real scenarios, which reduces computing costs, saves storage resources, and helps researchers exploit more benefits from high energy savings. Model compression was proposed to reduce the model size and improve the inference efficiency of the DNNs (Chen et al., 2019). Weights pruning (Chen et al., 2019) is one of the widely used techniques for compressing the model size by zeroing out the individual weight of the convolutional kernel or fully connected weights matrix. Filter pruning (Wang et al., 2017; Chen et al., 2019; Chen et al., 2019) is another kind of pruning technique that prunes entire filters (or nodes for fully connected layers) and their corresponding weights. In this way, the entire filters can be removed and the original DNN can be transformed to be a thinner network, thus achieving speedup on general hardware.

Recently, researchers have carried out several works on SNNs pruning methods and made considerable progress. In GPSNN (Chen et al., 2019), a two-stage growing-pruning algorithm was used to compress fully
connected SNN so that the network could achieve better performance. In (Kumar et al., 2017), the non-critical synapses of SNNs were regularly pruned during Spike Timing Dependent Plasticity (STDP) training process based on a preset threshold. A soft pruning method has been considered to reduce the number of SNN weight updating during network training (Kumar et al., 2017). Recently, ADMM optimization combined with Spatio-temporal backpropagation (STBP) training was used to compress SNNs (Kumar et al., 2018). An attention-guided compression technique presented in (Song et al., 2019), used two steps to generate compressed deep SNN that could reduce spiking activity. Recent work (Chen et al., 2020) performs pruning on the temporal dimension of SNNs to reduce time steps for inference. Grad Rewiring (Gord et al., 2019) is a joint learning algorithm of SNN connection and weight, which can strengthen the exploration of network structures. Most existing SNNs pruning work has either focused on shallow structures or has only attempted to prune networks at low sparsity. Besides, A very recent work proposed a dynamic pruning framework to prune SNNs based on temporal lottery ticket hyperthesis (Song et al., 2020), which handles the weights pruning of the deep SNN structures.

In this paper, we present an end-to-end framework of weights pruning to compress the SNNs with a given resource budget. Unlike most resource-constrained compression methods which treat the resource consumption function as a black box (Kumar et al., 2019), we directly use the resource consumption to formulate a constrained optimization. The key idea is to use learnable parameters to control the lower bound of the sparsities. This introduces a sparsity constraint so that the resource constraint will only depend on the sparsity parameters. The constrained problem can be transformed into a Minimax optimization. Since the sparsity and resource constraints are not differentiable, the Minimax problem cannot be directly solved by gradient-based methods. In this work, we use the difference of convex function (DC) (Song et al., 2019) sparsity reformulation and straight-through estimator (STE) (Beng et al., 2019) to build a gradient-based algorithm to effectively optimize the compression problem.

We summarize the contributions as below:

* We propose an end-to-end Minimax optimization method to successfully compress the SNNs, as shown in Figure 1. DC sparsity reformulation (Song et al., 2019) and STE (Beng et al., 2019) are key components in this Minimax reformulation. Our compression procedure is end-to-end joint training of compression and fine-tuning on SNNs.
* We formulate the resource-constrained SNNs compression problem into a constrained optimization problem where the SNNs weights and resource consumption are linked with learnable sparsity parameters.
* The algorithm is gradient-based and easy to train. Evaluations of SNNs pruning on the public benchmark tasks show that our method is effective to compress SNNs and achieves state-of-the-art (SOTA) performance.

## 2. Related Work

We review the related work from three aspects: the set of work in SNNs; the set of work in model compression and the specific model compression for SNNs.

### Spiking Neural Networks

Different from ANNs, SNNs have a temporal dimension inherently, which uses sparse binary spike event sequences to represent information. Therefore, they contribute to more energy savings in specialized neuromorphic hardware (Kumar et al., 2018). The information is transmitted among neurons via synapses. When the membrane potential exceeds a certain threshold caused by accumulating received spikes, the neuron fires a spike to the next layer. In this study, we employed the Leaky Integrate-and-Fire (LIF) neuron (Leaky Integrate and-Fire, 1990), which is one of the most widely used neurons due to its effectiveness. The most common form of the LIF neuron is described as:

\[\tau_{m}\frac{\mathrm{d}V_{m}(t)}{\mathrm{d}t}=-\left(V_{m}(t)-V_{\mathrm{ rest}}\right)+X_{t} \tag{1}\]

where \(V_{m}(t)\) represents the membrane potential of the neuron at time \(t\), \(X_{t}\) represents the input from the presynaptic neuron. \(\tau_{m}\) is the membrane time as a constant value, that controls the decay and \(V_{\mathrm{rest}}\) is the resting potential after firing. A spike will fire if \(V_{t}\) exceeds the threshold \(V_{\mathrm{th}}\). As claimed in previous works (Kumar et al., 2018) and (Gord et al., 2019), We convert the above continuous differential equation into a discrete version:

\[H_{t+1} =V_{t}+\frac{1}{\tau_{m}}\left(-\left(V_{t}-V_{\mathrm{rest}}\right) +X_{t}\right) \tag{3}\] \[S_{t+1} =\Theta\left(m_{t+1}-V_{\mathrm{th}}\right)\] (4) \[V_{t+1} =S_{t+1}V_{\mathrm{rest}}+\left(1-S_{t+1}\right)H_{t+1} \tag{2}\]

where \(H_{t}\) and \(V_{t}\) denote the value of membrane potential after neural dynamics and after generating a spike at time step \(t\), respectively. \(S_{t}\) denotes the spike output at time step \(t\). \(\Theta(\cdot)\) is the Heaviside step function which is defined as \(\Theta(x)=1\) for \(x>=0\) and \(\Theta(x)=0\) for \(x<0\).

As we can see, the integration and firing behavior of neurons will result in the non-differentiability of the transfer function. So it is difficult to apply standard backpropagation in the training phase (Beng et al., 2019). To obtain a high-performance SNN, researchers have proposed various training methods (Beng et al., 2019; Chen et al., 2020; Chen et al., 2020; Chen et al., 2020; Chen et al., 2020). Recently, some works focus on supervised learning based on backpropagation algorithms, where they use a surrogate gradient function to approximate the gradient of non-differentiable spike activity (Kumar et al., 2018; Chen et al., 2020; Chen et al., 2020; Chen et al., 2020; Chen et al., 2020). These surrogate gradient methods provide an effective solution for training SNNs with deeper architecture (Xu et al., 2019), such as VGGNet (Wang et al., 2019) and ResNet (Krizhevsky et al., 2015) families. Therefore, we adopt a backpropagation algorithm based on surrogate gradient (Song et al., 2019) as the basic method for our SNNs training.

Figure 1. Our whole pipeline. The resource-constrained Minimax Optimization method can compress SNNs into a light-weight model with different sparsity levels.
### Model Compression

There are different techniques for model compression, such as pruning (Zhou et al., 2017; Chen et al., 2017; Chen et al., 2018; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019), quantization (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019), low-rank factorization (Chen et al., 2017; Chen et al., 2019), etc. Pruning utilizes the sparsity of weights tensors to achieve model compression. Weights pruning (Zhou et al., 2017; Chen et al., 2019) is effective to remove the single elements by zeroing out the individual weight. Moreover, structured pruning (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019) prunes the weights according to specific structures to achieve more speedup on general hardware. Filter pruning (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019) is the most broadly used structured pruning, which prunes all the weights associated with certain filters. Most filter pruning works prune the channels with low magnitude weights (Chen et al., 2017; Chen et al., 2019), or estimate the importance of channels for pruning (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019). Yang _et al._(Yang et al., 2019) pre-compute a lookup table to estimate the latency of a convolution layer with different input/output channels, and use a greedy search strategy to gradually reduce the number of filters until the given resource budget is achieved. He _et al._(He et al., 2020) adopt reinforcement learning to search the number of pruned filters for each layer. The classification accuracy is used as the reward, and the number of pruned filters is taken as the action. Recently, more approaches (He et al., 2020; Chen et al., 2019; Chen et al., 2019) consider the model compression as a constrained problem (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019). Furthermore, resource consumption is used to restrict the action space. These methods are successfully applied to fully supervised tasks such as image classification and object detection. In this paper, we proposed an end-to-end optimization method to solve a resource-constrained compression problem on SNNs, we demonstrate the problem formulation from the unstructured weights pruning perspective.

### Model Compression for SNN

To reduce the energy consumption of SNNs, some approaches focus on the compression of SNN models recently, such as connection pruning (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019) and model quantization (Chen et al., 2017; Chen et al., 2019; Chen et al., 2019; Chen et al., 2019). Deng _et al._(Deng et al., 2019) defined the connection pruning and weight quantization as a constrained optimization problem and used Spatio-temporal backpropagation (STBP) and alternating direction method of multipliers (ADMM) to solve it. Chen _et al._(Chen et al., 2017) formulated the gradient rewiring (Grad R) algorithm which redefines the gradient of SNNs to a new synaptic parameter and joint learning SNNs connection and weight. In addition, Kim _et al._(Kim et al., 2019) performed the connection pruning toward deeper SNNs (\(\geq 16\) layers) and combined the Iterative Magnitude Pruning (Chen et al., 2019) and Early-Bird (Evely and Ferrara, 2019) tickets to obtain smaller SNNs. Chen _et al._(Chen et al., 2019) proposed a dynamic pruning algorithm based on nonlinear reparameterization mapping from spine size to SNN weights. To compare with these unstructured weights pruning works for SNNs, we adopt the Minimax optimization method to jointly optimize the global sparsity of SNNs and weights parameters. We handle all layers' weights parameters globally with one sparsity parameter to solve the unstructured pruning problem.

## 3. Formulation

### Resource-Constrained Optimization

The ideal scenario of SNN compression is that given a resource budget \(R_{\text{budget}}\) (based on a certain metric, e.g., Parameters, Flops, latency, or energy), the compression method can return a compressed model which satisfies the given budget and maintains the accuracy as well as possible. The whole process should be automatic, i.e., there is no need to manually set the sparsity of each layer. In this paper, we directly formulate such a compression scheme for a constrained optimization problem:

(5a) \[\min_{\mathcal{W},\mathbf{s}} \mathcal{L}(\mathcal{W})\] (5c) \[\text{s. t.} R(s)\leq R_{\text{budget}},\] (5d) \[\sum_{i}\mathbf{I}(\mathcal{W}_{i}=0)\geq s\] (5e)

where \(R(s)\) evaluates a general resource consumption (e.g., Flops or latency) based on the number of (nonzero) weights for each layer. It does not need to be differentiable. For example, when representing latency, it can be computed by a latency table as in Yang et al. (Yang et al., 2019). \(\mathcal{L}\) is the standard training loss.

\(\mathbf{I}(\cdot)\) is the indicator function that returns \(1\) if the argument is satisfied and \(0\) otherwise. \(\mathbf{s}\) is a learnable scalar variable to control the lower bound of the sparsity of weight parameters vector \(\mathcal{W}\) of the whole network. This formulation holds because the resource function \(R\) monotonically decreases with respect to the increasing sparsities, i.e., the more weights are pruned, the smaller the resource consumption we have. Note that we mainly focus on the unstructured pruning (or called weights sparsification) for SNN in the main text, thus \(\mathcal{W}_{i}\) in the above equation stands for **each element of weight parameters vector \(\mathcal{W}\)** of the whole network.

## 4. Optimization

In the previous section, we have already formulated the resource-constrained pruning as a constrained optimization (5). In this section, we first do some reformulation to make it more convenient to solve. Then we propose a gradient-based algorithm to solve the resource-constrained pruning in an end-to-end way.

### Minimax Reformulation

The sparsity constraint (\(\mathbf{c}_{\text{C}}\)) is non-convex and the non-continuous indicator function makes it more difficult. Common approaches to deal with this constraint include \(\ell_{1}\)-norm relaxation (Zhou et al., 2017) and \(\ell_{0}\)-norm projection (Chen et al., 2019). The ideas of \(\ell_{1}\)-norm relaxation have been applied in DNN compression (Wang et al., 2019; Chen et al., 2019; Chen et al., 2019). However, the \(\ell_{1}\)-norm can only approximate the sparsity constraint, so it is not equivalent to the sparsity constraint and there is no guarantee to bound the real sparsity by restricting the \(\ell_{1}\)-norm.

Figure 2. The dynamics of LIF neurons, as similarly described in (Chen et al., 2019). When the membrane potential exceeds a threshold value, the neuron will fire a spike to the next layer and resets.
Let \(\|\mathbf{u}\|_{\mathrm{s,2}}\) be the bottom-(s, 2) "norm", which denotes the \(\ell_{2}\)-norm of the sub vector composed of bottom-\(s\) elements in magnitude. Then we have an _equivalent_ reformulation for the sparsity constraint (5c):

\[\|\mathcal{W}\|_{\mathrm{s,2}}=0\Leftrightarrow\sum_{i}\mathbf{I}(\mathcal{W}_ {i}=0)\geq s, \tag{6}\]

Equation (6) is proved by Tono _et al._Tono et al. (2017), where the left-hand side is called DC (difference of convex functions) representation of the \(\ell_{0}\)-constraint. By this transformation, the sparsity constraint becomes an equality constraint of a continuous function. Compared to the original \(\ell_{0}\)-norm constraint, it can be written as a "soft" constraint and avoid being stuck in the bad local optimum of the constraint set.

By introducing dual variables \(y\) and \(z\), we derive the minimax reformulation of problem (5):

\[\min_{\mathcal{W},s}\max_{y,z\geq 0}\mathcal{L}(\mathcal{W})+\underbrace{y\| \mathcal{W}\|_{\mathrm{f}=1}^{2}}_{\mathrm{sparsity\ loss}\cdot\mathcal{S}(y,s \mathcal{W})}+\underbrace{z(R(s)-R_{\mathrm{budget}})}_{\mathrm{resource\ loss}}. \tag{7}\]

Where we introduce the sparsity loss \(\mathcal{S}(y,s,\mathcal{W}):=y\|\mathcal{W}\|_{\mathrm{f}=1}^{2}\), and resource loss \(z(R(s)-R_{\mathrm{budget}})\) to substitute the original constraints. Figure 3 shows an illustration of the reformulated minimax problem. It is easy to verify that (7)\(\rightarrow\)\(\ \
### Implementation details

In our works, we validate the compression method on six SNN models, including the shallow SNNs (e.g. 2 FC, 6 Conv and 2 FC) and the deep SNNs (e.g. VGG16 (Wang et al., 2017), ResNet19 (He et al., 2017), SEW ResNet18 (He et al., 2017), VGGSNN (He et al., 2017)). We compare the performance of our method with previous SNN compression methods on static MNIST, CIFAR10, CIFAR100, ImageNet11K datasets and neuromorphic CIFAR10-DVS dataset which is converted from the static image dataset by using a DVS camera. Experiments are conducted on NVIDIA V100 GPUs and we use Spikingfelly (Srivastava et al., 2015) framework to implement SNNs.

Similar to the previous SNN work (Chen et al., 2017), we use a shallow SNN with 2 FC layers on the MINST dataset, and a model with 6 convolution layers and 2 FC layers for the CIFAR10 dataset. These two shallow SNNs are trained with Adam optimizer with a learning rate 1e-4. The timestep is set to 8. Other hyperparameters of baseline are the same as [6](e.g. batch size, learning rate). What's more, we use deep SNNs, VGG16, ResNet19, and VGGSNN. The training method follows the previous SNN work (He et al., 2017). We train deep SNNs by SGD optimizer with momentum 0.9 and weight decay 5e-4. The learning rate is set to 0.05 for baseline training and cosine decay to 0. The timestep is set to 5 and the batch size is set to 32. The default number of epochs is set to 300. As for training SEW ResNet18 on ImageNet, we follow all the training setting of SEW ResNet (He et al., 2017). The base learning is 0.1 with cosine annealing scheduler. The number of epochs is 320, the batch size is set to 32, and the timestep is set to 4. In all datasets, the learning rate of \(y\) is set to 0.1, while the default learning rate of \(z\) is set to \(10^{5}\). We count the number of zero and nonzero values of the whole weights in SNN and compute the percentage of zero values to be the sparsity.

Training methodologyAccording to previous SNNs work settings respectively, we first train SNNs models to get pre-trained baseline models. Then our compression training stage starts with pre-trained models. Before pruning the SNN models, we set a budget list which values are some compression ratios from 0.5 to 0.005 (e.g. [0.25, 0.1, 0.05, 0.01, 0.005]). Furthermore, the value of the budget list can be connectivity, parameter size, latency times, FLOPs, and so on. For connection pruning, the values in the budget list are connectivity ratios. For structure pruning, the values in the budget list are FLOPs ratios in our work. During our compression training, if the model connectivity meets the current target compression ratio, we pause the pruning and then fine-tune the snapshot at this ratio until achieving the maximum fine-tuning epochs. After the

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Pruning Method** & **Dataset** & **Arch.** & **Top-1 Acc. (\%)** & **Acc. Loss (\%)** & **Sparsity (\%)** \\ \hline \multirow{3}{*}{ADMM-based (Srivastava et al., 2015)} & \multirow{3}{*}{MNIST} & \multirow{3}{*}{LeNet-5 like} & \multirow{3}{*}{99.07} & +0.03 & 50.00 \\  & & & & -0.43 & 60.00 \\  & & & & -2.23 & 75.00 \\ \hline \multirow{3}{*}{Deep R (Chen et al., 2017)} & \multirow{3}{*}{MNIST} & \multirow{3}{*}{2 FC} & \multirow{3}{*}{98.92} & -0.36 & 62.86 \\  & & & & -0.56 & 86.70 \\  & & & & -2.47 & 95.82 \\  & & & & -9.67 & 98.90 \\ \hline \multirow{3}{*}{Grad R (Chen et al., 2017)} & \multirow{3}{*}{MNIST} & \multirow{3}{*}{2 FC} & \multirow{3}{*}{98.92} & -0.33 & 74.29 \\  & & & & -0.43 & 82.06 \\  & & & & -2.02 & 94.37 \\  & & & & -3.55 & 96.94 \\  & & & & -8.08 & 98.62 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{MNIST} & \multirow{3}{*}{2 FC} & \multirow{3}{*}{**98.91**} & **-0.06** & **75.00** \\  & & & & **-0.16** & **85.00** \\  & & & & **-1.23** & **95.00** \\  & & & & **-2.70** & **97.00** \\  & & & & & **-7.34** & **98.70** \\ \hline \hline \multirow{3}{*}{ADMM-based (Srivastava et al., 2015)} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{7 Conv, 2 FC} & \multirow{3}{*}{89.53} & -0.38 & 50.00 \\  & & & & -2.16 & 75.00 \\  & & & & -3.85 & 90.00 \\ \hline \multirow{3}{*}{Deep R (Chen et al., 2017)} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{6 Conv, 2 FC} & \multirow{3}{*}{92.84} & -1.98 & 94.76 \\  & & & & -2.56 & 98.05 \\  & & & & -3.53 & 98.96 \\ \hline \multirow{3}{*}{Grad R (Chen et al., 2017)} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{6 Conv, 2 FC} & \multirow{3}{*}{92.84} & -0.30 & 71.59 \\  & & & & -0.34 & 87.96 \\  & & & & -0.81 & 94.92 \\  & & & & -1.47 & 97.65 \\  & & & & -3.52 & 99.27 \\ \hline \multirow{3}{*}{STDS (Chen et al., 2017)} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{6 Conv, 2 FC} & \multirow{3}{*}{92.84} & -0.35 & 97.77 \\  & & & & -2.63 & 99.25 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{6 Conv, 2 FC} & \multirow{3}{*}{**92.88**} & **+0.84** & **75.00** \\  & & & & **+0.52** & **88.04** \\  & & & & **+0.41** & **95.07** \\  & & & & **-0.13** & **97.71** \\  & & & & **-1.38** & **98.84** \\  & & & & **-2.56** & **99.31** \\ \hline \hline \end{tabular}
\end{table}
Table 1. Performance comparison between our method and previous works on MNIST and CIFAR10 datasets.
fine-tuning process at the current compression rate finishes, our method removes the current ratio from the budget list and then continues the compression training automatically to achieve the next compression ratio. Compression and fine-tuning are jointly performed in the one-stage training process of SNNs. The number of epochs in fine-tuning phase for \(i_{th}\) compression ratio in the budget list is set to the same value or scheduled as \(\frac{1}{S^{t}}\frac{T_{\text{epoch}}-C_{\text{epoch}}}{\sum\limits_{j=1}^{N} \frac{1}{S^{t}}}\).

where, \(S^{t}\) is the \(i_{th}\) compression ratio. \(T_{\text{epoch}}\) is the total number of epochs, \(C_{\text{epoch}}\) is the number of already used epochs.

### Quantitative Experiments

_Connection pruning and fine-tuning jointly._ We use the connection of the SNNs model as the budget compression ratios for our Minimax optimization method. During compression training, the connection pruning and fine-tuning are trained jointly. Our joint compression method not only reduces the training time and simplifies the tedious fine-tuning process for different compression ratio, but also help the model under smaller ratios get better performance. Therefore, for one SNN model, we can achieve state-of-the-art (SOTA) performance of different ratios in one compression training process, which is different from previous work, since they only can get one compression ratio per training process [38]. As shown in Table 1 and 2, we summed up the results of our compression

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Pruning Method** & **Dataset** & **Arch.** & **Top-1 Acc. (\%)** & **Acc. (\%)** & **Sparsity (\%)** \\ \hline \multirow{3}{*}{IMP [38]} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{VGG16} & \multirow{3}{*}{-} & 92.66 & 68.30 \\  & & & & 92.54 & 89.91 \\  & & & & 92.38 & 95.69 \\  & & & & 91.81 & 98.13 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{VGG16} & \multirow{3}{*}{**93.25**} & **93.27** & **69.02** \\  & & & & **93.26** & **90.09** \\  & & & & **92.76** & **95.70** \\  & & & & **92.00** & **98.13** \\ \hline \hline \multirow{3}{*}{Grad R [6]} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{93.22} & 92.68 & 76.90 \\  & & & & 91.91 & 94.25 \\  & & & & 91.12 & 97.56 \\ \hline \multirow{3}{*}{IMP [38]} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{93.22} & 93.50 & 76.20 \\  & & & & 93.46 & 94.29 \\  & & & & 93.18 & 97.54 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{CIFAR10} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{**94.85**} & **94.84** & **80.05** \\  & & & & **94.36** & **95.07** \\  & & & & **93.81** & **97.07** \\ \hline \hline \multirow{3}{*}{Grad R [6]} & \multirow{3}{*}{CIFAR100} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{71.34} & 69.36 & 77.03 \\  & & & & 67.47 & 94.92 \\  & & & & 67.31 & 97.65 \\ \hline \multirow{3}{*}{IMP [38]} & \multirow{3}{*}{CIFAR100} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{71.34} & 71.45 & 76.20 \\  & & & & 69.05 & 97.54 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{CIFAR100} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{**74.71**} & **73.05** & **79.99** \\  & & & & **72.67** & **95.19** \\  & & & & **70.80** & **97.31** \\ \hline \hline \multirow{3}{*}{Grad R [6]} & \multirow{3}{*}{ImageNet} & \multirow{3}{*}{SEW ResNet18} & \multirow{3}{*}{63.22} & 60.65 & 50.94 \\  & & & & 24.62 & 53.65 \\ \hline \multirow{3}{*}{ADMM [10]} & \multirow{3}{*}{ImageNet} & \multirow{3}{*}{SEW ResNet18} & \multirow{3}{*}{63.22} & 59.48 & 82.58 \\  & & & & 55.85 & 88.84 \\ \hline \multirow{3}{*}{STDS [8]} & \multirow{3}{*}{ImageNet} & \multirow{3}{*}{SEW ResNet18} & \multirow{3}{*}{63.22} & 61.30 & 82.58 \\  & & & & 59.93 & 88.84 \\  & & & & 58.06 & 93.24 \\  & & & & 56.28 & 95.30 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{ImageNet} & \multirow{3}{*}{SEW ResNet18} & \multirow{3}{*}{**63.25**} & **61.42** & **82.50** \\  & & & & **60.51** & **88.84** \\  & & & & **58.12** & **93.20** \\  & & & & **56.46** & **94.39** \\ \hline \hline \end{tabular}
\end{table}
Table 2. Performance comparison between our method and previous works on larger models.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & **Arch.** & **Top-1 Acc. (\%)** & **Acc. (\%)** & **Sparsity (\%)** \\ \hline \multirow{3}{*}{IMP [38]} & \multirow{3}{*}{VGGNN} & \multirow{3}{*}{82.80} & 81.10 & 76.20 \\  & & & 81.50 & 86.57 \\  & & & 80.10 & 89.91 \\  & & & 78.60 & 94.28 \\ \hline \multirow{3}{*}{**Ours**} & \multirow{3}{*}{VGGNN} & \multirow{3}{*}{82.80} & **82.40** & **85.18** \\  & & & **81.90** & **90.14** \\  & & & **81.20** & **93.14** \\  & & & **80.10** & **95.16** \\ \hline \hline \end{tabular}
\end{table}
Table 3. Performance of our method and previous work IMP on CIFAR10-DVS dataset.
experiments which were obtained by jointly pruning connection and fine-tuning the model.

Comparisons to previous work.We compare our method with previous works in Table 1, 2 and 3. The results of two shallow SNN models on MNIST and CIFAR10 are shown in Table 1. We compare our connection pruning and fine-tuning jointly training method with previous research, including ADMM-based (Deng et al., 2017), Deep R (Chen et al., 2018), Grad R (Chen et al., 2018), and STDS (Chen et al., 2018). It is observed that our approach outperforms previous pruning methods in terms of connection sparsity and accuracy on the two benchmark datasets. Furthermore, we have smaller \(\Delta\)Acc degradations under all comparable compression ratios.

We present the comparisons of deep SNN models, VGG16 (Vgg et al., 2016), ResNet19 (He et al., 2016) and SEW ResNet18 (He et al., 2016) in Table 2. We reproduce the baseline based on the previous work (Wang et al., 2018), and the performance is much higher than the reported result in (Wang et al., 2018): 1.63% higher on CIFAR10 and 3.37% higher on CIFAR100 for ResNet19 (He et al., 2016) SNN model, therefore, we compare absolute values of accuracy on deep SNNs. As shown in Table 2, our method is comparable to other approaches. For the VGG16 model, our methods have the highest accuracy at all connection sparsity ratios compared with Grad R (Chen et al., 2018) and IMP (Wang et al., 2018). For ResNet19 (He et al., 2016), when the connection sparsity ratio is less than 97%, our method significantly outperforms other methods. When the connection sparsity is higher than 97%, we still achieve better accuracy compared with other works, but the connection sparsity is slightly smaller than Grad R (Chen et al., 2018) and IMP (Wang et al., 2018). It is worth mentioning that the accuracy of our method can even be further improved compared to the baseline on all datasets when the sparsity is nearly 80%. Even on the large-scale dataset like ImageNet, our pruning method has also achieved competitive performance compared with the state-of-the-art.

We also validated on neuromorphic datasets that have been less involved in previous work. To the best of our knowledge, our work is the first work to compress SNNs on the temporal CIFAR10-DVS dataset. Under the same structure and settings, we implemented the IMP method (Wang et al., 2018) and conducted our experiments on the CIFAR10-DVS dataset. As shown in Table 3, our method significantly improves the accuracy of VGGSNN (Chen et al., 2018) model with different sparsity ratios. In summary, experiments have shown that our compression optimization method can theoretically handle any type of SNN model.

## 6. Ablation Studies

Comparison with sequential method.We compare our end-to-end Minimax optimization method on both sequentially and jointly training, for pruning and fine-tuning on the CIFAR10 dataset. For the sequential compression method, we first prune the SNNs models and save the pruned model snapshots during pruning training. We then fine-tune each of these snapshots for another 256 epochs for 6 Conv, 2 FC SNN models, which means need to fine-tune extra \(n\times 256\) epochs, the \(n\) is the length of the budget list. As shown in Table 4, the compression ratios (column b) in which pruning and fine-tuning are trained jointly have better accuracy when compression ratios are smaller than 5%. Sequentially trained compression ratios (column a) which connect 11.95% have a slight advantage because trained more epochs than joint ratios with a similar connection. However, as the connection turn smaller, the joint method achieves better results and even obtains an 8.72% accuracy advantage at a 0.64 connection ratio. What's more, the joint method in Table 4 was trained 700 epochs in total, which is far less than the train epochs numbers of the sequential method.

Comparison of fine-tuning options.We compare the final accuracy between applying with a cosine annealing scheduler and without any learning rate scheduler when fine-tuning on the CIFAR10 dataset. For using a cosine annealing scheduler, in each stage of achieving the new compression ratio in the resource budget list, the number of the fine-tuning epoch is reset to 300 and the initial learning rate is changed to 0.001. As shown in Table 4, the accuracy significantly increased when we use the cosine annealing scheduler in the fine-tuning phase.
Training trend plots.We plot the changing trend of sparsity value \(s\) in Figure 4, which demonstrates that our Minimax optimization algorithm can converge well gradually.

Per-layer Sparsity vs. Global Sparsity.We compare the difference between pruning using per-layer sparsity and our pruning using global sparsity. Our optimization is based on global sparsity \(s\), which sorts the weight values of tensors of all layers together during pruning and coordinates the total sparsity of the whole model during optimization. On the contrary, the method using per-layer sparsity sorts the weights on each layer separately, each with a sparsity variable to control. For both the per-layer sparsity method and global sparsity method, our compression target is the layers whose connectivity is more than 1e4. Figure 5 shows the difference in connectivity sparsity between the per-layer sparsity method and our global sparsity method on 6 Conv, 2 FC SNN model, the 6 source convolution layers have the same connectivity in the baseline model. In Figure 5, we can see the connectivity after pruning using the global sparsity method show more obvious diversities between these 6 convolution layers. However, the per-layer sparsity method results in almost the same pruned connectivity for these 6 convolution layers. In Table 6, we can see that our global sparsity method has better pruning performance at each connectivity level.