Membrane Potential Batch Normalization for Spiking Neural Networks
As one of the energy-efficient alternatives of conventional neural networks
(CNNs), spiking neural networks (SNNs) have gained more and more interest
recently. To train the deep models, some effective batch normalization (BN)
techniques are proposed in SNNs. All these BNs are suggested to be used after
the convolution layer as usually doing in CNNs. However, the spiking neuron is
much more complex with the spatio-temporal dynamics. The regulated data flow
after the BN layer will be disturbed again by the membrane potential updating
operation before the firing function, i.e., the nonlinear activation.
Therefore, we advocate adding another BN layer before the firing function to
normalize the membrane potential again, called MPBN. To eliminate the induced
time cost of MPBN, we also propose a training-inference-decoupled
re-parameterization technique to fold the trained MPBN into the firing
threshold. With the re-parameterization technique, the MPBN will not introduce
any extra time burden in the inference. Furthermore, the MPBN can also adopt
the element-wised form, while these BNs after the convolution layer can only
use the channel-wised form. Experimental results show that the proposed MPBN
performs well on both popular non-spiking static and neuromorphic datasets. Our
code is open-sourced at \href{https://github.com/yfguo91/MPBN}{MPBN}.
Yufei Guo, Yuhan Zhang, Yuanpei Chen, Weihang Peng, Xiaode Liu, Liwen Zhang, Xuhui Huang, Zhe Ma
# Membrane Potential Batch Normalization for Spiking Neural Networks

###### Abstract

As one of the energy-efficient alternatives of conventional neural networks (CNNs), spiking neural networks (SNNs) have gained more and more interest recently. To train the deep models, some effective batch normalization (BN) techniques are proposed in SNNs. All these BNs are suggested to be used after the convolution layer as usually doing in CNNs. However, the spiking neuron is much more complex with the spatio-temporal dynamics. The regulated data flow after the BN layer will be disturbed again by the membrane potential updating operation before the firing function, i.e., the nonlinear activation. Therefore, we advocate adding another BN layer before the firing function to normalize the membrane potential again, called MPBN. To eliminate the induced time cost of MPBN, we also propose a training-inference-decoupled re-parameterization technique to fold the trained MPBN into the firing threshold. With the re-parameterization technique, the MPBN will not introduce any extra time burden in the inference. Furthermore, the MPBN can also adopt the element-wised form, while these BNs after the convolution layer can only use the channel-wised form. Experimental results show that the proposed MPBN performs well on both popular non-spiking static and neuromorphic datasets. Our code is open-sourced at MPBN.

## 1 Introduction

Emerged as a biology-inspired method, spiking neural networks (SNNs) have received much attention in artificial intelligence and neuroscience recently [17, 13, 57, 56, 47, 58, 59]. SNNs deal with binary event-driven spikes as their activations and therefore the multiplications of activations and weights can be substituted for additions or only keeping silents. Benefitting from such a computation paradigm, SNNs derive extreme energy efficiency and run efficiently when implemented on neuromorphic hardware [1, 42, 5].

Despite the SNN has achieved great success in diverse fields including pattern recognition [12, 21, 19, 14], object detection [30], language processing [55], robotics [9], and so on, its development is deeply inspired by the experience of convolutional neural networks (CNNs) in many aspects. However, the spiking neuron model along with the rich spatio-temporal dynamic makes SNNs much different from CNNs, and directly transferring some experience of CNNs to SNNs without any modifications may be not a good idea. As one of the famous techniques in CNNs, the batch normalization(BN) technique shows great advantages. It can reduce the gradient exploding/vanishing problem, flatten the loss landscape, and reduce the internal covariate shift, thus being widely used in CNNs. There are also some works trying to apply normalization approaches in the SNN field to help model convergence. For example, inspired by BN in CNNs, NeuNorm [51] was proposed to normalize the data along the channel dimension. Considering that the temporal dimension is also important in SNNs, threshold-dependent batch normalization (tdBN) [62] then extended the scope of BN to the additional temporal dimension. Subsequently, to better depict the differences of data flow distributions in different time dimensions, the temporal batch normalization through time (BNTT) [31], postsynaptic potential normalization (PSP-BN) [28], and temporal effective batch normalization (TEBN) [10] that regulate the data flows with multiple BNs on different time steps were proposed.

However, all these BNs proposed in SNNs are advised to be used after convolution layers as usually doing in CNNs. This ignores the fact that the nonlinear transformation in the SNN spiking neuron is much more complex than that of the ReLU neuron. In the spiking neuron, the data flow after the convolution layer will be first injected into the residual membrane potential (MP) coming from the previous time step to generate a new MP at the current time step. And then the neuron will fire a spike or keep silent
still based on whether or not the new MP is up to the firing threshold. Obviously, though the data flow has been normalized by the BN after the convolution layer, it will be disturbed again by the residual MP in the membrane potential updating process. Therefore, we advocate also adding a BN layer after MP updating to regulate the data flow once again, called MPBN. Furthermore, we also propose a training-inference-decoupled re-parameterization technique in SNNs to fold the trained MPBN into the firing threshold. Hence, the MPBN will not induce any extra burden in the inference but a trivial burden in the training. The MPBN can be extended to channel-wised MPBN and element-wised MPBN further, which is very different from that of CNNs where only channel-wised normalization can be folded into weights. The difference between our SNN with MPBN and the vanilla SNN is illustrated in Fig. 1. Our main contributions are as follows:

* We propose to add another BN layer after the membrane potential updating operation named MPBN to handle the data flow disturbance in the spiking neuron. The experiment shows that MPBN can flatten the loss landscape further, thus benefiting model convergence and task accuracy.
* We also propose a re-parameterization method to decouple the training-time SNN and the inference-time SNN. In specific, we propose a method to fold the trained MPBN parameter into the firing threshold. Therefore, MPBN can be seen as only training auxiliary manner free from burdens in the inference-time. This re-parameterization method is suitable for both channel-wised MPBN and element-wised MPBN.
* Extensive experiment results show that the SNN trained with the MPBN is highly effective compared with other state-of-the-art SNN models on both static and dynamic datasets, e.g., 96.47% top-1 accuracy and 79.51% top-1 accuracy are achieved on the CIFAR-10 and CIFAR-100 with only 2 time steps.

## 2 Related Work

### Learning of Spiking Neural Networks

There are three kinds of learning algorithms of SNNs, including unsupervised learning [43, 24], converting ANN to SNN (ANN2SNN) [46, 25, 26], and supervised learning [19, 38, 20]. Unsupervised learning adopts some biological mechanism to update the SNN model, i.e., the spike-timing-dependent plasticity (STDP) approach [39], thus being considered a biologically plausible method. However, STDP cannot help train large-scale networks yet, thus it is usually limited to small datasets and non-ideal performance. The ANN-SNN conversion approach [22, 37] obtains an SNN by reusing well-trained homogeneous ANN parameters and replacing the ReLU neuron with a spiking neuron. Since the ANN model is easier to train and reach high performance, the ANN-SNN conversion method provides an interesting way to generate an SNN in a short time with competitive performance. However, the converted SNN will lose the rich temporal dynamic behaviors and thus cannot handle neuromorphic datasets well. Supervised learning [11, 50, 18] adopts the surrogate gradient (SG) approach to train SNNs with error backpropagation. It can handle temporal data and provide decent performance with few time steps on the large-scale dataset, thus having received much attention recently. For a more detailed introduction, please refer to the recent SNN survey [17]. Our work falls under the supervised learning.

### Normalization in Spiking Neural Networks

The batch normalization technique was originally introduced as a kind of training auxiliary method by [29] in CNNs. It uses the weight-summed input over a mini-batch of training cases to compute a mean and variance and then uses them to regulate the summed input. This simple operation can derive many benefits. i) It reduces the internal covariate shift (ICS), thus accelerating the training of a deep neural network. ii) It makes the network insensitive to the scale of the gradients, thus a higher learning rate can be chosen to accelerate the training. iii) It makes the network suitable for more nonlinearities by preventing the network from getting stuck in the saturated modes. With these advantages, more kinds of BNs were proposed, including layer normalization [2], group normalization [52], instance normalization [48], and switchable normalization [40].

Figure 1: The difference between our SNN with MPBN and the vanilla SNN. We add another BN layer after membrane potential updating (MPU) operation in the training. The MPBN can be folded into the firing threshold and then the homogenous firing threshold will be transformed into different ones.
There are also some works that modify and apply normalization approaches in the SNN field. For example, NeuNorm [51] also normalizes the feature map along the channel dimension like BN in CNNs. Recently, some methods were proposed to normalize the feature map from both the channel dimension and temporal dimension to take care of the spatio-temporal characteristics of the SNN, such as the threshold-dependent batch normalization (tdBN) [62]. It extends the scope of BN to the additional temporal dimension by adopting a 3DBN-like normalization method in CNNs. Note that, the tdBN can be folded into the weights, thus inducing no burden in the inference time. Nevertheless, NeuNorm and tdBN still use the shared parameters along the temporal dimension. Some works argued that the distributions of data in different time steps vary wildly and that using shared parameters is not a good choice. Subsequently, the temporal batch normalization through time (BNTT) [31], postsynaptic potential normalization (PSP-BN) [28], and temporal effective batch normalization (TEBN) [10] were proposed. These BNs regulate the data flow utilizing different parameters through time steps. Though these BNs with different BN parameters on different time steps can train more well-performed SNN models, their parameters can not be folded into the weights, thus will increase the computations and running time in the inference.

Nevertheless, all these BNs in the SNN field are advised to be used after convolution layers. However, the data flow after the convolution layer will not be presented to the firing function directly but to the membrane potential updating function first. Hence, the data flow will be disturbed again before reaching the firing function. To this end, in this paper, we add another BN after the membrane potential updating function, called the MPBN to retain normalized data flow before the firing function.

## 3 Preliminary

### Leaky Integrate-and-Fire Model

Different from CNNs, SNNs use binary spikes to transmit information. In the paper, we use the widely used Leaky-Integrate-and-Fire (LIF) neuron model [41] to introduce the unique spatial-temporal dynamic of the spiking model. First, we introduce the notation rules used here as follows. Vectors or tensors are denoted by bold italic letters, i.e., \(\mathbf{x}\) and \(\mathbf{o}\) represent the input and output variables respectively. Matrices are denoted by bold capital letters. For instance, \(\mathbf{W}\) is the weight matrix. The constant is denoted by small letters.

In LIF, the membrane potential is updated by

\[\mathbf{u}^{(t+1),\text{pre}}=\tau\mathbf{u}^{(t)}+\mathbf{c}^{(t+1)},\text{ where }\mathbf{c}^{(t+1)}=\mathbf{W}\mathbf{x}^{(t+1)}, \tag{1}\]

where \(\mathbf{u}\) represents the membrane potential and \(\mathbf{u}^{(t+1),\text{pre}}\) is the updated membrane potential at time step \(t+1\), \(\mathbf{c}^{(t+1)}\) is the pre-synaptic input at time step \(t+1\), which is charged by weight-summed input spikes \(\mathbf{x}^{(t+1)}\), and \(\tau\) is a constant within \((0,1)\), which controls the leakage of the membrane potential. Then, when the updated membrane potential \(\mathbf{u}^{(t+1),\text{pre}}\) is up to the firing threshold \(V_{\text{th}}\), the LIF spiking neuron will fire a spike as bellow,

\[\mathbf{o}^{(t+1)}=\begin{cases}1&\text{if }\mathbf{u}^{(t+1),\text{pre}}>V_{ \text{th}}\\ 0&\text{otherwise}\end{cases}, \tag{2}\] \[\mathbf{u}^{(t+1)}=\mathbf{u}^{(t+1),\text{pre}}\cdot(1-\mathbf{o}^{(t+1)}).\]

After firing, the spike output \(\mathbf{o}^{(t+1)}\) at time step \(t+1\) will be transmitted to the next layer and become its input. At the same time, the updated membrane potential will be reset to zero and becomes \(\mathbf{u}^{(t+1)}\) to join the neuron processing at the next time step.

**The Classifier in the SNN.** In a classification model, the final output is used to compute the \(\operatorname{Softmax}\) and predict the desired class object. In an SNN model, if we also use LIF neurons at the output layer to fire spikes and use the number of spikes to compute the probability, too much information will be lost. Therefore, we only integrate the output and do not fire them across time, as doing in recent work [20, 21, 12].

\[\mathbf{o}_{\text{out}}=\frac{1}{T}\sum_{t=1}^{T}\mathbf{c}_{\text{out}}^{(t)}=\frac{ 1}{T}\sum_{t=1}^{T}\mathbf{W}\mathbf{x}^{(t)}. \tag{3}\]

Then, the cross-entropy loss is computed based on the true label and \(\operatorname{Softmax}(\mathbf{o}_{\text{out}})\).

### Batch Normalization in SNNs

Batch normalization can effectively reduce the internal covariate shift and alleviate the gradient vanishing or explosion problem for training networks, thus having been widely used in CNNs. Fortunately, BN can also be used in SNNs. Considering a spiking neuron with input \(\mathbf{c}=\{\mathbf{c}^{(1)},\mathbf{c}^{(2)},\dots,\mathbf{c}^{(t)},\dots\}\), where \(t\) is the time step, BN regulate the input at each time step as follows,

\[\tilde{\mathbf{c}}_{i}^{(t)}=\frac{\mathbf{c}_{i}^{(t)}-\mathbf{\mu}_{i}}{\sqrt{\mathbf{ \sigma}_{i}^{2}+\epsilon}}, \tag{4}\]

where \(\mathbf{c}_{i}^{(t)}\) is the input in \(i\)-th channel at \(t\)-th time step, \(\mathbf{\mu}_{i}\) and \(\mathbf{\sigma}_{i}\) are the mean and variance of input in channel dimension, and \(\epsilon\) is a small constant to avoid denominator being zero. To ensure BN can represent the identity transformation, the normalized vector \(\tilde{\mathbf{c}}_{i}^{(t)}\) is scaled and shifted in a learning manner as follows,

\[\operatorname{BN}(\mathbf{c}_{i}^{(t)})=\mathbf{\lambda}_{i}\tilde{\mathbf{c}}_{i}^{(t)}+ \mathbf{\beta}_{i}, \tag{5}\]

where \(\mathbf{\lambda}_{i}\) and \(\mathbf{\beta}_{i}\) are channel-wised learnable parameters.
## 4 Methodology

This section first introduces the specific form of membrane potential batch normalization. Then the re-parameterization technique that how to fold the MPBN into \(V_{\mathrm{th}}\) will be introduced in detail. Next, some key details for training the SNN and the pseudocode for the training and inference of our SNN will be given. Finally, we will provide plenty of ablation studies and the comparison of the loss landscape of the models with or without MPBN to show the effectiveness of the proposed method.

### Membrane Potential Batch Normalization

As abovementioned, we argue that though the data flow has been normalized by the BN after the convolution layer, it will be disturbed again by the membrane potential updating operation. To better depict this, we give the vanilla form of LIF neuron with BN first as follows,

\[\mathbf{u}^{(t+1),\text{pre}}=\tau\mathbf{u}^{(t)}+\mathrm{BN}(\mathbf{W}\mathbf{x}^{(t+1 )}), \tag{6}\]

where \(\tau\) is 0.25 in the paper following [21, 38, 4]. To regulate the disturbed data flow once again, We further embed another BN after the membrane potential updating operation, called MPBN. The LIF neuron with MPBN can be updated as

\[\tilde{\mathbf{u}}^{(t+1),\text{pre}}=\mathrm{MPBN}(\mathbf{u}^{(t+1),\text{pre}}). \tag{7}\]

Obviously, \(\mathbf{u}^{(t+1),\text{pre}}\) will be scaled and sifted, and some \(\mathbf{u}^{(t+1),\text{pre}}\) less than \(V_{\mathrm{th}}\) may be greater than \(V_{\mathrm{th}}\) with MPBN and vice versa. This is abhorrent with the biology and MPBN will cause some extra computation burden in the inference compared with the vanilla one. To solve this problem, we also propose a training-inference-decoupled re-parameterization technique here.

### Re-parameterization

With MPBN, the firing function will be updated as

\[\mathbf{o}^{(t+1)}=\begin{cases}1&\text{if }\mathrm{MPBN}(\mathbf{u}^{(t+1),\text{pre}})>V _{\mathrm{th}}\\ 0&\text{otherwise}\end{cases}. \tag{8}\]

If we unfold the MPBN, the above equation will be re-organized as

\[\mathbf{o}_{i}^{(t+1)}=\begin{cases}1&\text{if }\mathbf{\lambda}_{i}\frac{\mathbf{u}_{i}^{(t+1), \text{pre}}-\mathbf{\mu}_{i}}{\sqrt{\mathbf{\sigma}_{i}^{2}}}+\mathbf{\beta}_{i}>V_{ \mathrm{th}}\\ 0&\text{otherwise}\end{cases}. \tag{9}\]

By folding the MPBN to \(V_{\mathrm{th}}\), the firing function will be further updated as

\[\mathbf{o}_{i}^{(t+1)}=\begin{cases}1&\text{if }\mathbf{u}^{(t+1),\text{pre}}>( \mathbf{\tilde{V}}_{\mathrm{th}})_{i}\\ 0&\text{otherwise}\end{cases}, \tag{10}\] \[\text{where }(\mathbf{\tilde{V}}_{\mathrm{th}})_{i}=\frac{(V_{ \mathrm{th}}-\mathbf{\beta}_{i})\sqrt{\mathbf{\sigma}_{i}^{2}}}{\mathbf{\lambda}_{i}}+\bm {\mu}_{i}.\]

It can be seen that by absorbing some parameters from MPBN, \(V_{\mathrm{th}}\) will be transformed to another channel-wised \((\mathbf{\tilde{V}}_{\mathrm{th}})_{i}\). In this way, the extra computation burden caused by MPBN will be eliminated again in the inference time. Furthermore, the diversity of the spiking neuron will be improved with abundant firing parameters as the learnable firing threshold in other work [3, 49].

### Training Framework

In the paper, the spatial-temporal backpropagation (STBP) algorithm [51] is adopted to train the SNN mod
els. STBP treats the SNN model as a self-recurrent neural network thus enabling an error backpropagation mechanism following the same principles as in CNNs. However, there is still a problem impeding the direct training of SNNs. To demonstrate this problem, we formulate the gradient at the layer \(l\) by the chain rule, given by

\[\frac{\partial L}{\partial\mathbf{W}^{l}}=\sum_{t}(\frac{\partial L}{\partial \mathbf{\boldsymbol{o}}^{(t),l}}\frac{\partial\mathbf{\boldsymbol{o}}^{(t),l}} {\partial\mathbf{\boldsymbol{u}}^{(t),l}}+\frac{\partial L}{\partial\mathbf{ \boldsymbol{u}}^{(t+1),l}}\frac{\partial\mathbf{\boldsymbol{u}}^{(t+1),l}}{ \partial\mathbf{\boldsymbol{u}}^{(t),l}})\frac{\partial\mathbf{\boldsymbol{u}}^ {(t),l}}{\partial\mathbf{\boldsymbol{W}}^{l}}, \tag{11}\]

where \(\frac{\partial\mathbf{\boldsymbol{o}}^{(t),l}}{\partial\mathbf{\boldsymbol{u}}^ {(t),l}}\) is the gradient of firing function at at \(t\)-th time step in \(l\)-th layer. Obviously, the non-differentiable firing activity of the spiking neuron will result in zero gradients everywhere, while infinity at \(V_{\mathrm{th}}\). Therefore, the gradient descent \((\mathbf{\boldsymbol{W}}^{l}\leftarrow\mathbf{\boldsymbol{W}}^{l}-\eta\frac{ \partial L}{\partial\mathbf{\boldsymbol{W}}^{l}})\) either freezes or updates to infinity in the backpropagation. To handle this problem, here, we also adopt the commonly used STE surrogate gradients as doing in other surrogate gradients (SG) methods [44, 20]. Mathematically, it is defined as:

\[\frac{d\mathbf{\boldsymbol{o}}}{d\mathbf{\boldsymbol{u}}}=\left\{\begin{array} []{ll}1,&\text{if }0\leq\mathbf{\boldsymbol{u}}\leq 1\\ 0,&\text{otherwise}\end{array}\right.. \tag{12}\]

Then, the SNN model can be trained end-to-end. The training and inference of our SNN are detailed in Algo. 1.

### Ablation Study

To verify the effectiveness of the MPBN, a lot of ablative studies using spiking ResNet20 architecture along with different time steps were conducted on the CIFAR-10 and CIFAR-100 datasets. The results of top-1 accuracy of these models are shown in Tab. 1. It's can be seen that the test accuracy of the SNNs with MPBN is always higher than these vanilla counterparts. For example, the accuracy of baseline SNN with 1 time step is 90.40%, while with MPBN, it will increase up to 92.22%, which is a huge improvement (more than 2.0%) in the SNN field. Moreover, we also show the test accuracy curves of ResNet20 using MPBN and its vanilla counterpart with 2 time steps on CIFAR-10/100 during training in Fig. 2. It can be observed obviously that the SNNs with MPBN also perform better on convergence speed. To sum up, the proposed MPBN can both improve accuracy and convergence speed, which are very important aspects in deep learning.

### Loss Landscape

We further inspect the 1D loss landscapes [36] of the SNNs with or without MPBN using spiking ResNet20 architecture in 2 time steps to show why the MPBN can improve accuracy and convergence speed in Fig. 3. It can be observed that the loss landscape of the SNN model with MPBN is flatter than that of the SNN without MPBN. This indicates that the MPBN makes the landscape of the corresponding optimization problem smoother [36], thus making the gradients more predictive and network faster convergence. The results here provide convincing evidence for ablation studies in Section 4.4.

## 5 Experiments

In this section, abundant experiments were conducted to verify the effectiveness of the MPBN using widely-used spiking ResNet20 [44, 46], VGG16 [44], ResNet18 [11],

\begin{table}
\begin{tabular}{c c c c} \hline \hline Dataset & Method & Time step & Accuracy \\ \hline \multirow{5}{*}{CIFAR-10} & baseline & 1 & 90.40\% \\  & w/ MPBN & 1 & 92.22\% \\ \cline{2-4}  & baseline & 2 & 92.80\% \\  & w/ MPBN & 2 & 93.54\% \\ \cline{2-4}  & baseline & 4 & 93.85\% \\  & w/ MPBN & 4 & 94.28\% \\ \hline \multirow{5}{*}{CIFAR-100} & baseline & 1 & 67.94\% \\  & w/ MPBN & 1 & 68.36\% \\ \cline{1-1} \cline{2-4}  & baseline & 2 & 70.18\% \\ \cline{1-1} \cline{2-4}  & w/ MPBN & 2 & 70.79\% \\ \cline{1-1} \cline{2-4}  & baseline & 4 & 71.77\% \\ \cline{1-1} \cline{2-4}  & w/ MPBN & 4 & 72.30\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Ablation experiments for MPBN.

Figure 3: The 1D loss landscape of spiking ResNet20 with and without MPBN.

Figure 2: The accuracy curves of spiking ResNet20 with or without MPBN using 2 time steps on CIFAR-10 (left) and CIFAR-100 (right). The MPBN based SNNs obviously enjoy higher accuracy and easier convergence.
ResNet19 [62], and ResNet34 [11] on both static datasets including CIFAR-10 [32], CIFAR-100 [32], and ImageNet [6], and one neuromorphic dataset, CIFAR10-DVS [35]. The specific introduction for these datasets has be detailed in many recent works [62, 44, 21, 38]. Here, we mainly introduce these hyper-parameters and data preprocessing in detail. We used the widely adopted LIF neuron in our SNN models as other works about direct training methods [44, 46]. These hyper-parameters for LIF neuron about the initial firing threshold \(V_{\mathrm{th}}\) and the membrane potential decaying constant \(\tau_{\mathrm{decay}}\) are \(0.5\) and \(0.25\) respectively. For static image datasets, since encoding the 8-bit RGB images into 1-bit spikes will lose too much information, we use an ANN-like convolutional layer and a LIF layer to encode the images to spikes for all the rest of the layers, as in recent works [62, 44, 21, 38].

### Comparison with SoTA Methods

**CIFAR-10.** On CIFAR-10, we trained our SNN model using the SGD optimizer with 0.9 momentum. The initial learning rate is 0.1 and decays to 0 in cosine form. The total training time is 400 epochs. To fairly compare with these recent SoTA methods [38, 20, 16], we also adopt data normalization, random horizontal flipping, cropping, and cutout [8] for data augmentation. We run three times for each experiment to report the "mean \(\pm\) std" in Tab. 2. It can be seen that our models can outperform other methods over all these chosen widely adopted architectures with

\begin{table}
\begin{tabular}{l l l l c c} \hline \hline Dataset & Method & Type & Architecture & Timestep & Accuracy \\ \hline \multirow{8}{*}{Diet-SNN [44]} & SpikeNorm [46] & ANN2SNN & VGG16 & 2500 & 91.55\% \\  & Hybrid-Train [45] & Hybrid training & VGG16 & 200 & 92.02\% \\  & Spike-basedBP [34] & SNN training & ResNet11 & 100 & 90.95\% \\  & Joint A-SNN [19] & SNN training & ResNet18 & 4 & 95.45\% \\  & GLIF [60] & SNN training & ResNet19 & 2 & 94.44\% \\  & PLIF [12] & SNN training & PLIFNet & 8 & 93.50\% \\ \hline \multirow{4}{*}{Diet-SNN [44]} & \multirow{2}{*}{SNN training} & VGG16 & 5 & 92.70\% \\  & & & 10 & 93.44\% \\ \cline{3-5}  & & ResNet20 & 5 & 91.78\% \\  & & & 10 & 92.54\% \\ \hline \multirow{4}{*}{RecDis-SNN [20]} & \multirow{2}{*}{SNN training} & \multirow{2}{*}{ResNet19} & 2 & 93.64\% \\  & & & 4 & 95.53\% \\  & & & 6 & 95.55\% \\ \hline \multirow{4}{*}{CIFAR-10} & \multirow{2}{*}{SNN training} & \multirow{2}{*}{ResNet20} & 2 & 93.13\% \\  & & & 4 & 93.66\% \\  & & & 6 & 94.25\% \\ \hline \multirow{4}{*}{CIFAR-10} & \multirow{2}{*}{STBP-tdBN [62]} & \multirow{2}{*}{SNN training} & 2 & 92.34\% \\  & & & 4 & 92.92\% \\  & & & 6 & 93.16\% \\ \hline \multirow{4}{*}{TET [7]} & \multirow{2}{*}{SNN training} & \multirow{2}{*}{ResNet19} & 2 & 94.16\% \\  & & & 4 & 94.44\% \\  & & & 6 & 94.50\% \\ \hline \multirow{4}{*}{Real Spike [21]} & \multirow{2}{*}{SNN training} & \multirow{2}{*}{ResNet19} & 2 & 95.31\% \\  & & & 4 & 95.51\% \\  & & & 6 & 96.10\% \\ \hline \multirow{4}{*}{InfLoR-SNN [16]} & \multirow{2}{*}{SNN training} & ResNet20 & 5 & 93.01\% \\  & & & 10 & 93.65\% \\ \hline \multirow{4}{*}{**MPBN**} & \multirow{4}{*}{SNN training} & ResNet19 & 1 & **96.06\%\(\pm 0.10\)** \\  & & & 2 & **96.47\%\(\pm 0.08\)** \\ \cline{1-1} \cline{3-5}  & & & 1 & **92.22\%\(\pm 0.11\)** \\ \cline{1-1} \cline{3-5}  & & & 2 & **93.54\%\(\pm 0.09\)** \\ \cline{1-1} \cline{3-5}  & & & 4 & **94.28\%\(\pm 0.07\)** \\ \cline{1-1} \cline{3-5}  & & VGG16 & 2 & **93.96\%\(\pm 0.09\)** \\ \cline{1-1} \cline{3-5}  & & & 4 & **94.44\%\(\pm 0.08\)** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison with SoTA methods on CIFAR-10.
fewer time steps. For example, The accuracy of spiking ResNet19 trained with MPBN with only 1 time step can be up to 96.06%, while the Real Spike [21] needs 6 time steps to reach a comparable result and the RecDis-SNN [61] even still underperforms 0.51% with 6 time steps. this superiority can also be observed in the results regarding the spiking ResNet20 and VGG16.

**CIFAR-100.** For CIFAR-100, we adopted the same settings as in CIFAR-10. The proposed MPBN also performs well on CIFAR-100. It can be seen that our method gets the best accuracy over all these networks even with fewer time steps. For instants, the ResNet19 trained with MPBN

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Dataset & Method & Type & Architecture & Timestep & Accuracy \\ \hline \multirow{8}{*}{ImageNet} & STBP-tdBN [62] & SNN training & ResNet34 & 6 & 63.72\% \\  & TET [7] & SNN training & ResNet34 & 6 & 64.79\% \\  & MS-ResNet [27] & SNN training & ResNet18 & 6 & 63.10\% \\  & OTTT [54] & SNN training & ResNet34 & 6 & 63.10\% \\  & Real Spike [21] & SNN training & ResNet18 & 4 & 63.68\% \\ \cline{2-6}  & SEW ResNet [11] & SNN training & ResNet18 & 4 & 63.18\% \\  & & ResNet34 & 4 & 67.04\% \\ \cline{2-6}  & **MPBN** & SNN training & ResNet18 & 4 & **63.14\%\(\pm 0.08\)** \\  & & ResNet34 & 4 & **64.71\%\(\pm 0.09\)** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison with SoTA methods on ImageNet.

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Dataset & Method & Type & Architecture & Timestep & Accuracy \\ \hline \multirow{8}{*}{ImageNet} & SpikeNorm [46] & ANN2SNN & ResNet20 & 2500 & 64.09\% \\  & RMP [23] & ANN2SNN & ResNet20 & 2048 & 67.82\% \\  & Hybrid-Train [45] & Hybrid training & VGG11 & 125 & 67.90\% \\  & IM-Loss [15] & SNN training & VGG16 & 5 & 70.18\% \\ \cline{2-6}  & Joint A-SNN [19] & SNN training & ResNet18 & 4 & 77.39\% \\  & & ResNet34 & 4 & 79.76\% \\ \hline \multirow{8}{*}{Dspike [38]} & \multirow{2}{*}{SNN training} & \multirow{2}{*}{ResNet20} & 2 & 71.68\% \\  & & & & 4 & 73.35\% \\  & & & & 6 & 74.24\% \\ \hline \multirow{8}{*}{CIFAR-100} & \multirow{8}{*}{TET [7]} & \multirow{8}{*}{SNN training} & \multirow{2}{*}{ResNet19} & 2 & 72.87\% \\  & & & & 4 & 74.47\% \\  & & & & 6 & 74.72\% \\ \cline{1-1} \cline{2-6}  & RecDis-SNN [20] & SNN training & ResNet19 & 4 & 74.10\% \\  & & VGG16 & 5 & 69.88\% \\ \cline{1-1} \cline{2-6}  & InfLoR-SNN [16] & SNN training & ResNet20 & 5 & 71.19\% \\  & & VGG16 & 5 & 71.56\% \\ \cline{1-1} \cline{2-6}  & Real Spike [21] & SNN training & ResNet20 & 5 & 66.60\% \\  & & VGG16 & 5 & 70.62\% \\ \cline{1-1} \cline{2-6}  & GLIF [60] & SNN training & ResNet19 & 2 & 75.48\% \\  & & & & 4 & 77.05\% \\ \hline \multirow{3}{*}{TEBN [10]} & \multirow{3}{*}{SNN training} & \multirow{3}{*}{ResNet19} & 2 & 75.86\% \\  & & & & 4 & 76.13\% \\ \cline{1-1}  & & & & 6 & 76.41\% \\ \hline \multirow{3}{*}{**MPBN**} & \multirow{3}{*}{SNN training} & \multirow{3}{*}{ResNet19} & \multirow{3}{*}{1} & **74.74\%\(\pm 0.11\)** \\  & & & & **78.71\%\(\pm 0.10\)** \\  & & & & 2 & **79.51\%\(\pm 0.07\)** \\ \cline{1-1} \cline{2-6}  & & ResNet20 & 2 & **70.79\%\(\pm 0.08\)** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison with SoTA methods on CIFAR-100.
can achieve 78.71% top-1 accuracy with only 1 time step, which outperforms other SoTA methods such as TET, GLIF, TEBN, and RecDis-SNN even with 4 or 6 time steps about 1.66%-3.99%.

**ImageNet.** On ImageNet, we used standard data normalization, random horizontal flipping, and cropping for data augmentation and trained the networks for 320 epochs as in [11]. The optimizer setting also keeps the same with CIFAR datasets. The results for ImageNet are presented in Tab. 4. It can be seen that the accuracy of our method is better than that of these recent SoTA methods, only relatively smaller compared with SEW ResNet [11] for spiking ResNet34. However, SEW ResNet is not a typical SNN model. It adopts the activation before addition form-based ResNet and its blocks will fire positive integer spikes. In this way, the event-driven and multiplication-addition transform advantages of SNNs will be lost. While we adopt the original ResNet, which fires standard binary spikes.

**CIFAR10-DVS.** We also adopted the neuromorphic dataset, CIFAR10-DVS in the paper to verify the effectiveness of the MPBN. We also split the dataset into 9K training images and 1K test images, and resize them to \(48\times 48\) for data augmentation as in [51, 21]. The learning rate is 0.01 and other settings are the same as CIFAR-10. It can be seen that the MPBN also shows superiority in this dataset.

### Extension of the MPBN

In CNNs, the most widely used BN is channel-wised. This is because element-wised BNs are very time-consuming and can not be folded into weights, otherwise, the channel-wised weight-sharing mechanism will be destroyed. However, the MPBN adopts the firing threshold-folded manner and the firing threshold need not keep the same along the channels, therefore, MPBN can use the element-wised form freely. In this way, \(V_{\mathrm{th}}\) will be transformed to element-wised ones as follows,

\[(\mathbf{\tilde{V}}_{\mathrm{th}})_{i,j,k}=\frac{(V_{\mathrm{th}}-\mathbf{\beta}_{i,j, k})\sqrt{\mathbf{\sigma}_{i,j,k}^{2}}}{\mathbf{\lambda}_{i,j,k}}+\mathbf{\mu}_{i,j,k}, \tag{13}\]

where\((\mathbf{\tilde{V}}_{\mathrm{th}})_{i,j,k}\) is the transformed firing threshold of the neuron comes from \(i\)-th channel in the spatial position \((j,k)\). To investigate the performance of the element-wised MPBN, here we also provide a comparison of the vanilla MPBN and its extension. The results of top-1 accuracy of the spiking ResNet20 with 4 time steps on CIFAR datasets are shown in Tab. 6. Though the two versions all perform well, the element-wised MPBN is relatively better than the channel-wised MPBN. This may be because element-wised MPBN can learn more firing threshold values, which means a richer representation ability for SNNs.

## 6 Conclusion

In the paper, we advocated adding the MPBN before the firing function to regulate the disturbed data flow again. We also provided a training-inference-decoupled re-parameterization technique to fold the trained MPBN into the firing threshold to eliminate the extra time burden induced by MPBN in the inference time. Furthermore, the channel-wised and element-wised MPBN in different granularities were explored. Extensive experiments verified that the proposed MPBN can consistently achieve good performance.

## Acknowledgment

This work is supported by grants from the National Natural Science Foundation of China under contracts No.12202412 and No.12202413.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Dataset & Method & Time step & Accuracy \\ \hline \multirow{2}{*}{CIFAR-10} & channel-wised & 4 & 94.28\% \\  & element-wised & 4 & 94.42\% \\ \hline \multirow{2}{*}{CIFAR-100} & channel-wised & 4 & 72.30\% \\  & element-wised & 4 & 72.49\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison with learnable firing threshold methods.

\begin{table}
\begin{tabular}{l l l l c c} \hline \hline Dataset & Method & Type & Architecture & Timestep & Accuracy \\ \hline \multirow{8}{*}{CIFAR10-DVS} & Rollout [33] & Rollout & DenseNet & 10 & 66.80\% \\  & LIAF-Net [53] & Conv3D & LIAF-Net & 10 & 71.70\% \\  & LIAF-Net [53] & LIAF & LIAF-Net & 10 & 70.40\% \\  & STBP-tdBN [62] & SNN training & ResNet19 & 10 & 67.80\% \\  & RecDis-SNN [20] & SNN training & ResNet19 & 10 & 72.42\% \\ \cline{2-6}  & Real Spike [21] & SNN training & ResNet19 & 10 & 72.85\% \\  & & ResNet20 & 10 & 78.00\% \\ \cline{2-6}  & **MPBN** & SNN training & ResNet19 & 10 & **74.40\%\(\pm 0.20\)** \\  & & ResNet20 & 10 & **78.70\%\(\pm 0.10\)** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Comparison with SoTA methods on CIFAR10-DVS.