Towards Zero Memory Footprint Spiking Neural Network Training
Biologically-inspired Spiking Neural Networks (SNNs), processing information
using discrete-time events known as spikes rather than continuous values, have
garnered significant attention due to their hardware-friendly and
energy-efficient characteristics. However, the training of SNNs necessitates a
considerably large memory footprint, given the additional storage requirements
for spikes or events, leading to a complex structure and dynamic setup. In this
paper, to address memory constraint in SNN training, we introduce an innovative
framework, characterized by a remarkably low memory footprint. We \textbf{(i)}
design a reversible SNN node that retains a high level of accuracy. Our design
is able to achieve a $\mathbf{58.65\times}$ reduction in memory usage compared
to the current SNN node. We \textbf{(ii)} propose a unique algorithm to
streamline the backpropagation process of our reversible SNN node. This
significantly trims the backward Floating Point Operations Per Second (FLOPs),
thereby accelerating the training process in comparison to current reversible
layer backpropagation method. By using our algorithm, the training time is able
to be curtailed by $\mathbf{23.8\%}$ relative to existing reversible layer
architectures.
Bin Lei, Sheng Lin, Pei-Hung Lin, Chunhua Liao, Caiwen Ding
# Towards Zero Memory Footprint Spiking Neural Network Training

###### Abstract

Biologically-inspired Spiking Neural Networks (SNNs), processing information using discrete-time events known as spikes rather than continuous values, have garnered significant attention due to their hardware-friendly and energy-efficient characteristics. However, the training of SNNs necessitates a considerably large memory footprint, given the additional storage requirements for spikes or events, leading to a complex structure and dynamic setup. In this paper, to address memory constraint in SNN training, we introduce an innovative framework, characterized by a remarkably low memory footprint. We **(i)** design a reversible SNN node that retains a high level of accuracy. Our design is able to achieve a \(\mathbf{58.65\times}\) reduction in memory usage compared to the current SNN node. We **(ii)** propose a unique algorithm to streamline the backpropagation process of our reversible SNN node. This significantly trims the backward Floating Point Operations Per Second (FLOPs), thereby accelerating the training process in comparison to current reversible layer backpropagation method. By using our algorithm, the training time is able to be curtailed by \(\mathbf{23.8\%}\) relative to existing reversible layer architectures.

## 1 Introduction

As a bio-inspired neuromorphic computing representative, Spiking Neural Network (SNN) has attracted considerable attention, in contrast to the high computational complexity and energy consumption of traditional Deep Neural Networks (DNNs) [28; 2; 25; 10]. SNN processes information using discrete-time events known as spikes rather than continuous values, offering extremely hardware-friendly and energy-efficient characteristics. For instance, in a robot navigation task using Intel's Loihi [1], SNN could achieve a 276\(\times\) reduction in energy compared to a conventional DNN approach. Work [24] shows that DNN consumes 111 mJ and 1035 mJ per sample on MNIST and CIFAR-10, respectively, while SNN consumes only 0.66 mJ and 102 mJ, i.e., 168\(\times\) and 10\(\times\) energy reduction.

Despite their numerous advantages, one major bottleneck in the deployment of SNNs has been memory consumption. The memory complexity of a traditional DNN with a depth of \(L\) is \(\mathcal{O}(L)\). But for SNN of the same depth \(L\), there are several timesteps \(T\) involved in the computation. Consequently, the memory complexity of SNNs escalates to \(\mathcal{O}(L*T)\). For instance, the memory requirement during the DNN training process of ResNet19 is 0.6 GB, but for the SNN with the same architecture could reach about 12.34 GB (-20 \(\times\)) when time-step equals 10. This presents a significant challenge for their applicability to resource-constrained systems, such as IoT-Edge devices [21].

To address the issue of high memory usage of SNN, researchers have proposed several methods, including Quantization [21], Weight Sparsification [22; 12], the Lottery Ticket Hypothesis [13], Knowledge Distillation [9], Efficient Checkpointing [27], and so on. In this paper, we introduce a novel reversible SNN node that drastically compresses the memory footprint of the SNN node inside the entire network. Our method achieves state-of-the-art results in terms of SNN memory
savings. It achieves this by recalculating all the intermediate states on-the-fly, rather than storing them during the backward propagation process. To further enhance the efficiency of our approach, we also present a new algorithm for the backpropagation process of our reversible SNN node, which significantly reduces the training time compared with the original reversible layer backpropagation method. Remarkably, our method maintains the same level of accuracy throughout the process.

As a result of our innovations, we reduce the memory complexity of the SNN node from \(\mathcal{O}(n^{2})\) to \(\mathcal{O}(1)\), while maintaining comparable accuracy to traditional SNN node. Moreover, our method reduces the FLOPs needed for the backpropagation by a factor of 23% compared to existing reversible layer backpropagation method, thus accelerating the training process. Collectively, these advances pave the way for more efficient and scalable SNN implementations, enabling the deployment of these biologically inspired networks across a wider range of applications and hardware platforms.

## 2 Background And Related Works

### Spiking Neural Network

Spiking neural network (SNN) uses sparse binary spikes over multiple time steps to deal with visual input in an event-driven manner [1; 3]. We use SNN with the popular Leaky Integrate and Fire (LIF) spiking neuron. The forward pass is formulated as follows.

\[v[t]=\alpha v[t{-}1]+\sum_{i}w_{i}s_{i}[t]-\vartheta o[t-1] \tag{1a}\] \[o[t]=h(v[t]-\vartheta)\] (1b) \[h(x)=\begin{cases}0,&\text{if }x<0\\ 1,&\text{otherwise}\end{cases} \tag{1c}\]

where \(t\) denotes time step. In Eq. (1a), \(v[t]\) is the dynamics of the neuron's membrane potential after the trigger of a spike at time step t. The sequence \(s_{i}[t]\in\{0,1\}\) represents the \(i\)-th input spike train, consisting solely of 0s and 1s, while \(w_{i}\) is the corresponding weight. In Eq. (1b), \(o[t]\in\{0,1\}\) is the neuron's output spike train. In Eq. (1c), \(h(x)\) is the Heaviside step function to generate the outputs.

In the backward pass, we adopt Backpropagation Through Time (BPTT) to train SNNs. The BPTT for SNNs using a surrogate gradient [19], which is formulated as follows.

\[\delta_{l}[t]=\epsilon_{l+1}[t]w_{l+1} \tag{2a}\] \[\epsilon_{l}[t]=\delta_{l}[t]\phi_{l}[t]+\alpha\epsilon_{l}[t]\] (2b) \[\frac{\partial L}{\partial w_{l}}=\sum_{t=0}^{T-1}\epsilon_{l}[t] \cdot[s_{l}[t]]^{\intercal} \tag{2c}\]

Denote \(L\) as the loss, in Eq. (2a), \(\delta_{l}[t]{=}\frac{\partial L}{\partial\alpha_{l}[t]}\) is the error signal at layer \(l\) time step \(t\) which is propagated using above formulations. In Eq. (2b), \(\epsilon_{l}[t]{=}\frac{\partial L}{\partial w_{l}[t]}\), \(\phi_{l}[t]{=}\frac{\partial o_{l}[t]}{\partial w_{l}[t]}{=}\frac{\partial u _{l}(v_{l}[t]-\vartheta)}{\partial v_{l}[t]}\), where we follow the gradient surrogate function in [7] to approximate the derivative of \(u(x)\), such that \(\frac{\partial u(x)}{\partial x}\approx\frac{1}{1+\pi^{x}x^{2}}\). Eq. (2c) calculates the gradient of \(l^{th}\) layer weight \(w_{l}\).

### Reversible Layer

A reversible layer refers to a family of neural network architectures that are based on the Non-linear Independent Components Estimation [4; 5]. In traditional training methods, the activation values of each layer are stored in memory, which leads to a dramatic increase in memory requirements as the network depth increases. However, reversible transformation technology allows us to store only the final output of the network and to recompute the discarded ones when needed. This approach significantly reduces memory requirements, making it possible to train deeper neural networks and more complex models under limited memory conditions, thus potentially unlocking new insights and improving performance across a wide range of tasks. The reversible transformation has been used in
different kinds of neural networks, such as CNN [8], Graph neural networks (GNN) [15], Recurrent Neural Networks (RNN) [17] and some transformers [18]. Furthermore, studies have demonstrated the effectiveness of reversible layers in different types of tasks, including image segmentation [20], natural language processing [14], compression [16], and denoising [11].

## 3 Reversible SNN Algorithm

### Reversible SNN Memory Analysis

During the training process of SNN networks, the activation values occupy the main memory storage space. The SNN activation value memory analysis schematic diagram is shown in Fig.1.

In this figure, we use the VGG-13 architecture [26] with ten timesteps as an example. The percentage values represent the memory footprint ratio of each part in the entire network. The left diagram is the original SNN where the activation values of \(X\) account for 90.9% of the memory usage, and the output potentials of each neuron occupy 9.1% of the memory. The right diagram is our designed reversible SNN, which only requires saving the final \(X_{output}\) values and the output potentials of each neuron, without storing all intermediate values, thus significantly saving memory. The intermediate activation values will be regained during the backpropagation process through our inverse calculation equation. In this example, our method is able to save 90.21% of the memory used for activation values. The exact amount of memory saved by our method will be shown in Experiment part.

### Reversible SNN Forward Calculation

Our forward algorithm is in the upper section of Fig.2.

The various input states \(S=(X,V)\) of each neuron are evenly divided into two groups along the last dimension. Namely: \(S=[S_{1},S_{2}]\).

**P**: Calculate the first part of output \(Y_{1}\):

\[M_{1}^{t}=V_{1}^{t-1}+\frac{1}{\tau}\cdot\left(X_{1}^{t}-V_{1}^{t-1}\right) \quad\text{(\ref{eq:SNNNN})} Y_{1}^{t}=H\left(M_{1}^{t}-V_{th}\right)+\beta\cdot X_{2}^{t} \tag{4}\]

\(M_{1}^{t}\) is the membrane potential of the first half neuron at time \(t\). \(V_{2}^{t-1}\) is the input potential of the second half neuron at time \(t-1\). \(\tau\) is the time constant. \(X_{2}^{t}\) is the input to the second half neuron at time \(t\). \(V_{th}\) is the threshold voltage of the neurons. \(H()\) is the Heaviside step function. \(\beta\) is a scaling factor for the input. \(\beta\cdot X_{2}^{t}\) will help \(Y_{1}^{t}\) to collect information about the second half of the input in the next step. Then calculate first part of output voltage \(V_{1}^{t}\):

\[V_{1}^{t}=\left(1-Y_{1}^{t}\right)\odot M_{1}^{t}+Y_{1}^{t}\cdot V_{res}+ \alpha\cdot V_{1}^{t-1} \tag{5}\]

\(V_{1}^{t}\) is the output potential of the first half neuron at time \(t\). \(V_{res}\) is the reset voltage of the neurons. \(\alpha\) is a scaling factor for the membrane potential.

Figure 1: Activation value Memory Comparison between the original SNN network and our reversible SNN network, using VGG13 with a timestep of ten as an example.

: SNN node,

: Save to memory,

: NOT save to memory.
\(\mathfrak{C}\): Use the first part of output \(Y_{1}\) to calculate the second part \(Y_{2}\):

\[M_{2}^{t}=V_{2}^{t-1}+\frac{1}{\tau}\left(Y_{1}^{t}-V_{2}^{t-1}\right) \tag{6}\] \[Y_{2}^{t}=H\left(M_{2}^{t}-V_{th}\right)+\beta\cdot X_{1}^{t} \tag{7}\]

\(M_{2}^{t}\) is the membrane potential of the second half neuron at time \(t\). \(Y_{2}^{t}\) is the output of the second half neuron at time \(t\). calculate the second part of output voltage \(V_{2}^{t}\):

\[V_{2}^{t}=\left(1-Y_{2}^{t}\right)\odot M_{2}^{t}+Y_{2}^{t}\cdot V_{res}+ \alpha\cdot V_{2}^{t-1} \tag{8}\]

\(V_{2}^{t}\) is the output potential of the second half neuron at time \(t\).

\(\mathfrak{C}\): For all the output states \(S_{output}=([Y_{1},Y_{2}],[V_{1}^{t},V_{2}^{t}])\), combine them by the last dimension.

### Reversible SNN Inverse Calculation

The purpose of the inverse calculation is to use the output results to obtain the unsaved input values. i.e. Use \(Y\) and \(V_{output}\) to calculate \(X\) and \(V\). Our inverse algorithm is in the lower section of Fig.2.

\(\mathfrak{1}\): For all the output states \(S_{output}=(Y,V_{output})\), divide them into two groups by the last dimension in the same way as in the first step of forward calculation, namely: \(S_{output}=[\hat{S}_{output}1;S_{output}2]\)

\(\mathfrak{2}\): Calculate \(V_{1}^{t}\) by combine Eq.6 and Eq.8, simplify:

\[V_{2}^{t-1}=\frac{V_{2}^{t}-(1-Y_{2})\cdot\frac{1}{\tau}\odot Y_{1}-Y_{2} \cdot V_{reset}}{(1-Y_{2})\cdot(1-\frac{1}{\tau})+\alpha} \tag{9}\]

Calculate \(X_{1}^{t}\) by combine Eq.6 and 7, simplify:

\[X_{1}^{t}=\left(Y_{2}^{t}-H\left(M_{2}^{t}-V_{th}\right)\right)\div\beta \tag{10}\]

\(\mathfrak{3}\): Calculate \(V_{1}^{t}\) by combine Eq.3 and Eq.5, simplify:

\[V_{1}^{t-1}=\frac{V_{1}^{t}-(1-Y_{1})\cdot\frac{1}{\tau}\odot X_{1}^{t}-Y_{1} \cdot V_{reset}}{(1-Y_{1})\cdot(1-\frac{1}{\tau})+\alpha} \tag{11}\]

Figure 2: This reversibility demo use \(2\times 2\) toy Input as an example and shows our forward and inverse calculations. : The origin of the equations in the inverse process.
Calculate \(X_{1}^{t}\) by combine Eq.3 and 4, simplify:

\[X_{2}^{t}=\left(Y_{1}^{t}-H\left(M_{1}^{t}-V_{th}\right)\right)\div\beta \tag{12}\]

\(\mathfrak{x}\): For all the input states \(S=([X_{1},X_{2}],[V_{1}^{t-1},V_{2}^{t-1}])\), combine them by the last dimension.

## 4 Inverse Gradient Calculation

Although our reversible architecture significantly reduces memory usage, it does extend computation time for two primary reasons: (i) It necessitates the recalculation of the activation values that weren't originally stored. (ii) Many of the previous reversible layer architectures have inherited the backpropagation method from checkpointing [30; 6]. This method requires using the recalculated intermediate activation values to rerun the forward equation, thereby constructing a forward computational graph. This graph is then used to derive the corresponding gradients. This step of rerunning the forward equation introduces additional computational overhead, which extends the overall computation time.

This scenario is prevalent across all existing architectures of reversible layers, including Reversible GNN [15], Reversible RNN [17], Reversible Transformers [18], and so on. To reduce the training time, we have designed a new algorithm, called the inverse gradient calculation method, which is able to substantially decrease the number of FLOPs during the backpropagation process compared to the original reversible architecture. Our design is shown in Fig.3.

The left diagram illustrates the original forward and backward processes. The middle diagram depicts the original calculation process for reversible layers. It contains four steps:

1. The input \(X\) pass the forward function to compute the output \(Y\), without storing the input data to conserve memory.
2. For each layer \(n\): The output \(X^{n}\) of this layer pass the inverse function to compute the input \(X^{n-1}\) of this layer. This process starts with the final output \(Y\).
3. For each layer \(n\): The input \(X^{n-1}\) passes through the forward function again to reconstruct the forward computational graph, which facilitates gradient computation.
4. For each layer \(n\): Compute the gradient \(\frac{\partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}\) based on the forward computational graph.

The right diagram is our design, it contains three steps:

1. The input \(X\) pass the forward function to compute the output \(Y\), without storing the input data to conserve memory.
2. For each layer \(n\): The output \(X^{n}\) of this layer pass the inverse function to compute the input \(X^{n-1}\) of this layer and construct an inverse computational graph.
3. For each layer \(n\): Compute the gradient \(\frac{\partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}\) based on the inverse computational graph.

Below is the specific calculation formula of the \(\frac{\partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}\) based on the inverse computation graph, and the derivation process is in the Appendix.

\[\frac{\partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}=\frac{\theta}{2+\left( \pi\cdot\theta\cdot\left(M_{1}^{t}-V_{th}\right)\right)^{2}}\cdot\frac{1}{\tau }\odot\left(1+\frac{\theta}{2+\left(\pi\cdot\theta\cdot\left(M_{2}^{t}-V_{th} \right)\right)^{2}}\cdot\frac{1}{\tau}\right)+\beta \tag{13}\]

\[\frac{\partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}=\frac{\theta}{2+\left( \pi\cdot\theta\cdot\left(M_{2}^{t}-V_{th}\right)\right)^{2}}+\beta \tag{14}\]

All the variables in Eq.13 and Eq.14 have the same meaning as the variables in Eq.3-Eq.12 and \(\theta\) is an adjustable constant parameter.

The ability to perform computational graph inverse computation in my algorithm is based on that our forward function has symmetry with the inverse computation function.

For the original reversible network:

\[FLOPS_{backward}^{pri}=FLOPS_{inverse}+FLOPS_{forward}+FLOPS_{\frac{ \partial\mathbf{X^{n}}}{\partial\mathbf{X^{n-1}}}} \tag{15}\]

For our reversible network:

\[FLOPS_{backward}^{our}=FLOPS_{inverse}+FLOPS_{part\;of\;\frac{\partial \mathbf{X^{n-1}}}{\partial\mathbf{X^{n}}}} \tag{16}\]

Compared to the standard reversible network, our method reduces FLOPS by 23%. The FLOPS analysis is shown in Appendix and the detailed time measurement is shown in the Experiment part.

## 5 Experiment

We first compare our design with the SOTA SNN Memory-efficient methods for SNN training on the several datasets. Subsequently, we incorporate our reversible SNN node into different architectures across various datasets. The goal is twofold: Firstly, we wanted to demonstrate that compared to the SNN node currently in use, our reversible version is able to offer substantial memory savings. Secondly, we aim to show that, when compared to the existing reversible layer backpropagation method, our reversible SNN node backpropagation design is able to considerably reduce the time spent in the backpropagation process, thereby accelerating the training phase. In the final part, we conduct an ablation study to evaluate the influence of various parameters within our equations and the impact of the number of groups into which the input is divided on the performance of our model.

All experiments were conducted on a Quadro RTX6000 GPU equipped with 24GB of memory, using PyTorch 1.13.1 with CUDA 11.4, and an Intel(R) Xeon(R) Gold 6244 CPU running at 3.60GHz. To ensure that the values we obtain through inverse-calculation is the same as the original forward-calculation method, we use torch.allclose(rtol=\(1e^{-06}\), atol=\(1e^{-10}\)) to compare all the inverse calculated values with the origianl forward calculated values. All the results return true as they should be. Detailed hyperparameter settings for each experiment are provided in the Appendix.

### Comparison with the SOTA Methods

We conducted a comparison of our approach with the current SOTA methods in memory efficiency during the SNN training process on the CIFAR10 and CIFAR100 datasets. To verify the universality of our work, we apply our designed reversible SNN node to the current SOTA sparse training work for SNNs. A comparison was then made between these two methods on the Tiny-ImageNet dataset, the results are shown in Table 1.
Our approach (RevSNN) achieves a \(\mathbf{2.54\times}\) memory reduction on the CIFAR10 dataset and a \(\mathbf{3.13\times}\) memory reduction on the CIFAR100 dataset compared to the current SOTA SNN training memory-efficient method. At the same time, we also maintain a high level of accuracy. On the Tiny-ImageNet dataset, we only replaced the original SNN node with our designed reversible SNN node, keeping all other conditions consistent (RevND). As a result, the accuracy of our VGG-16 model structure is 0.28% points higher than that of the original dense model and saves \(\mathbf{1.87\times}\) more memory than the original work at 99% sparsity. On the ResNet-19 model, our accuracy is 0.31% points higher than the dense model, and saves \(\mathbf{2.06\times}\) more memory than the original work at 99% sparsity.

### Memory Consumption Evaluation

To investigate whether our newly designed reversible SNN node achieves the expected memory savings compared to the original Spiking Neural Node, we incorporated our node into a range of architectures including VGG-11, 13, 16, 19, and ResNet-19, 34, 50, 101. For the VGG architectures, we examined the corresponding memory usage for timesteps ranging from 1 to 20, while for the ResNet architectures, we scrutinized the memory usage for timesteps from 1 to 10. These tests were conducted on the CIFAR-10 dataset. For all the experiments, we keep the batch size = 128.

The SNN node memory comparison results is shown in Fig.4. For VGG architecture, even when employing the most memory-intensive VGG-19 architecture with a timestep of 20, the cumulative memory usage for all the reversible SNN nodes within the entire network remains below 200MB. In contrast, using conventional SNN nodes demands a substantial amount of memory, up to 9032MB. For ResNet architectures, the ResNet-101 architecture with a timestep of 10 needs about 28993MB using conventional SNN node, but only 1382MB using our reversible SNN node. As the number of model layers and the timestep value increase, the memory savings achieved by our reversible SNN node become more pronounced. Specifically, when utilizing the VGG-19 architecture with a timestep of 20, our reversible SNN node enjoys a \(\mathbf{58.65\times}\) memory reduction compared to the original SNN node. The specific data values are shown in the Appendix.

These experimental results align with our theoretical analysis in Section 3.1, further validating that our design is able to significantly reduce memory usage.

### Training time Evaluation

To investigate the time efficiency of our designed backpropagation architecture in comparison with the traditional reversible layer backpropagation method, we employ two sets of backpropagation architectures for our reversible SNN node. The first set utilizes the original reversible layer backpropagation method, while the second set incorporates our newly designed backpropagation architecture.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Method** & **Architecture** & **Time-steps** & **Accuracy** & **Memory(Glb)** \\ \hline \multirow{8}{*}{CIFAR10} & OITT [32] & VGG(6/8V8) & 6 & 93.52\% & 4 \\  & S2A-STSU [29] & ResNet-17 & 5 & 92.75\% & 27.93 \\  & IDE-LIF [33] & CIFARNet-F & 30 & 91.74\% & 2.8 \\  & Hybrid [23] & VGG-16 & 100 & 91.13\% & 9.36 \\  & Tandem [31] & CifarNet & 8 & 89.04\% & 4.2 \\  & Skipper [27] & VGG-5 & 100 & 87.44\% & 4.6 \\ \cline{2-6}  & **RevSNN(Ours)** & ResNet-18 & 4 & 91.87\% & **1.101** \\ \hline \multirow{8}{*}{CIFAR100} & IDE-LIF [33] & CIFARNet-F & 30 & 71.56\% & 3.51\% \\  & OITT [32] & VGG(8/8V5) & 6 & 71.05\% & 4.04\% \\  & S2A-STSU [29] & VGG-13 & 4 & 68.96\% & 31.05 \\  & Skipper [27] & VGG-5 & 100 & 66.48\% & 4.6 \\ \cline{2-6}  & **RevSNN(Ours)** & ResNet-18 & 4 & 71.13\% & **1.12** \\ \hline \multirow{8}{*}{Tiny-ImageNet} & ND(Dense) [12] & VGG-16 & 5 & 39.45\% & 3.99 \\  & ND(90\% Sparsity) [12] & VGG-16 & 5 & 39.12\% & 3.78 \\ \cline{1-1}  & ND(99\% sparsity) [12] & VGG-16 & 5 & 33.84\% & 3.76 \\ \cline{1-1} \cline{2-6}  & **RevSNN(Ours)** & VGG-16 & 5 & 39.73\% & **2.01** \\ \cline{1-1} \cline{2-6}  & ND(Dense) [12] & ResNet-19 & 5 & 50.32\% & 5.29 \\ \cline{1-1}  & ND(90\% Sparsity) [12] & ResNet-19 & 5 & 49.25\% & 5.11 \\ \cline{1-1}  & ND(99\% sparsity) [12] & ResNet-19 & 5 & 41.96\% & 5.09 \\ \cline{1-1} \cline{2-6}  & **RevSNN(Ours)** & ResNet-19 & 5 & 50.63\% & **2.47** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of our work with the SOTA methods in memory efficiency during the SNN training process. For all the works: Batch size = 128. *: They did not provide the memory data directly for training CIFAR100, we estimate it based on their memory usage for training CIFAR10 and their parameter data.
We employ VGG-11, VGG-13, VGG-16, and VGG-19 architectures with timesteps ranging from 1 to 10. We compare the time required for one iteration of training using the original SNN node, the reversible SNN node with the original reversible layer backpropagation method, and the reversible SNN node with our backpropagation architecture on the CIFAR-10 datasets. We perform all the experiments on an empty RTX6000 GPU and keep the batch size = 64. The reported times for each forward and backward pass are averages taken over all iterations within the first five epochs.

Fig.5 presents our measurement of the training time when the number of timesteps is set to 4, 6, and 8. The forward computation times for the three methods are virtually identical. The shortest backward processing time is exhibited by the original SNN node, primarily due to it records all intermediate values throughout the computation, thus eliminating the need for recalculations. Comparatively, among the two reversible SNN nodes, our backpropagation design achieves a \(\mathbf{20}\%-\mathbf{30}\%\) increase in speed over previous reversible layer backpropagation method during the backward process. As the network expands, the superiority of our backpropagation design becomes increasingly evident. Specifically, under the VGG-19 architecture with a timestep of 8, our designed node is able to save \(\mathbf{23.8}\%\) of the total training time compared to the reversible node using the original reversible layer backpropagation method. This aligns well with our previous theoretical predictions in Section 4. Data for the other timesteps is shown in the Appendix.

### Ablation Study

#### Effects of parameters \(\alpha\) and \(\beta\) in our equations

In Eq.4 and Eq.5, we have two parameters: \(\alpha\) and \(\beta\). The optimal setting for the parameter \(\beta\) is 1, as this maximizes the preservation of the original features of the data. We conduct experiments to

Figure 4: Memory comparison between normal SNN node and our reversible SNN node.

Figure 5: Training time analysis. Solid lines: Backward process’s duration; Dashed lines: Forward process’s duration; Red lines: Training time for the original SNN; Green lines: Training time for the reversible SNN using the original reversible layer backpropagation method; Blue lines: Training time for the reversible SNN employing our proposed backpropagation architecture.
assess the impact of the \(\alpha\) parameter on the model's performance. We vary the \(\alpha\) parameter from 0.05 to 0.8, and then employ architectures VGG-19, VGG-16, VGG-13, and VGG-11 to evaluate the accuracy on the CIFAR100 dataset. The results are shown on the left of Fig.6. We observe that varying \(\alpha\) within the range of 0.05 to 0.8 impacts the final accuracy by approximately 1%. Generally, the model exhibits optimal performance when \(\alpha\) is set between 0.1 to 0.2.

**Effects of number of groups for the various states**

In Section 3.2, We introduce a method of splitting various input states into two groups along the last dimension. Nonetheless, this method might encounter issues under specific circumstances. For instance, if the last dimension of a tensor is an odd number, it cannot be evenly divided into two groups. To address this, we enhance the original algorithm: we divide the various input states into \(n\) groups according to the number of elements \(n\) in the last dimension. Eq.3-5 is then executed sequentially for each group. This enhancement further improves the universality of our algorithm.

To evaluate the impact of the number of groups on the model, we modified part of the fully connected layers in the original ResNet-19, ResNet-18, VGG-16, VGG-13 network from 128 activations to 144 activations. This is to allow it to have a wider variety of factors. We then evaluate the model's performance with the number of groups set to 2, 3, 6, 12, 24, 48, 72, and 144 respectively on CIFAR100 dataset. The results are shown on the right of Fig.6. We observe that the training accuracy improves as the number of groups increases. When the number of groups approaches the number of elements \(n\) in the last dimension, the accuracy typically surpasses that of the original SNN node. This is attributed to a larger number of groups yielding a higher fidelity representation of the original data.

## 6 Conclusion and Discussion

This work addresses a fundamental bottleneck of current deep SNNs: their high GPU memory consumption. We have designed a novel reversible SNN node that is able to reduce memory complexity from \(\mathcal{O}(n^{2})\) to \(\mathcal{O}(1)\). Specifically, our reversible SNN node allows our SNN network to achieve \(\mathbf{2.54}\) times greater memory efficiency than the current SOTA SNN memory-efficient work on the CIFAR10 dataset, and \(\mathbf{3.13}\) times greater on the CIFAR100 dataset. Furthermore, in order to tackle the prolonged training time issue caused by the need for recalculating intermediate values during backpropagation within our designed reversible SNN node, we've innovated a new backpropagation approach specifically suited for reversible architectures. This innovative method, when compared to the original reversible layer architecture, achieves a substantial reduction in overall training time by \(\mathbf{23.7}\%\). As a result, we are able to train over-parameterized networks that significantly outperform current models on standard benchmarks while consuming less memory.

Figure 6: **Left Figure**: Test VGG-19,VGG-16,VGG-13,VGG-11 models on CIFAR100 dataset by using different \(\alpha\) settings. **Right Figure**: Change activations number from 128 to 144 for some fully connected layers inside ResNet-19, ResNet-18, VGG-16, VGG-13 and test model performance for different number of groups on CIFAR100. Rev.: Reversible SNN node. Ori.: Original SNN node. Mo.: Modified network (Change some fully connected layers).