Spiking Synaptic Penalty: Appropriate Penalty Term for Energy-Efficient
  Spiking Neural Networks
Spiking neural networks (SNNs) are energy-efficient neural networks because
of their spiking nature. However, as the spike firing rate of SNNs increases,
the energy consumption does as well, and thus, the advantage of SNNs
diminishes. Here, we tackle this problem by introducing a novel penalty term
for the spiking activity into the objective function in the training phase. Our
method is designed so as to optimize the energy consumption metric directly
without modifying the network architecture. Therefore, the proposed method can
reduce the energy consumption more than other methods while maintaining the
accuracy. We conducted experiments for image classification tasks, and the
results indicate the effectiveness of the proposed method, which mitigates the
dilemma of the energy--accuracy trade-off.
Kazuma Suetake, Takuya Ushimaru, Ryuji Saiin, Yoshihide Sawada
# Spiking Synaptic Penalty: Appropriate Penalty Term

###### Abstract

Spiking neural networks (SNNs) are energy-efficient neural networks because of their spiking nature. However, as the spike firing rate of SNNs increases, the energy consumption does as well, and thus, the advantage of SNNs diminishes. Here, we tackle this problem by introducing a novel penalty term for the spiking activity into the objective function in the training phase. Our method is designed so as to optimize the energy consumption metric directly without modifying the network architecture. Therefore, the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy. We conducted experiments for image classification tasks, and the results indicate the effectiveness of the proposed method, which mitigates the dilemma of the energy-accuracy trade-off.

Machine Learning, Neural Networks, Neural Networks

## 1 Introduction

With the rapid growth and spread of neural networks, realizing energy-efficient neural networks is an urgent mission for sustainable development. One such model is the spiking neural network (SNN), which is also known to be more biologically plausible than ordinary artificial neural networks (ANNs). SNNs are energy-efficiently driven on neuromorphic chips (Akopyan et al., 2015; Davies et al., 2018) or certain field-programmable gate arrays (FPGAs) (Maguire et al., 2007; Misra & Saha, 2010) by asynchronously processing spike signals. However, as the spike firing rate of an SNN increases, the energy consumption does as well, and thus, the advantage of the SNN diminishes. Therefore, in addition to the shift from ANNs to SNNs, it is advantageous to adopt training methods that reduce energy consumption in the inference phase. At the same time, such a training method should be independent of the network architecture to avoid limitations in the application. That is, our goal is to develop a training method that realizes energy-efficient SNNs without any constraint on the network architecture.

There are various approaches toward energy-efficient SNNs, such as pruning, quantization, and knowledge distillation (Kundu et al., 2021; Chowdhury et al., 2021; Lee et al., 2021), which are widely-used approaches also in ANNs. Further, there are SNN-specific approaches sparsifying the spiking activity related to the energy consumption (Lee et al., 2020; Kim & Panda, 2021; Naya et al., 2021), to which our method belongs. In particular, the methods that penalize the spike firing rate in the training phase are close to our aforementioned goal (Esser et al., 2016; Sorbaro et al., 2020; Pellegrini et al., 2021). However, they indirectly reduce the energy consumption by arbitrarily reducing the spike firing rate, where there is no strict positive correlation between them. Hence, reducing energy consumption while maintaining accuracy is difficult.

Principle and IdeaOur principle is that--we should optimize the metric as it is in the training phase. In this spirit, we propose to introduce a proper penalty term for the spiking activity--a _spiking synaptic penalty_--into the objective function. It is derived so that its expected value is precisely proportional to the energy consumption metric for the SNN (see Fig. 1). Although the difference between the proposed and existing methods is only at the

Figure 1: Comparison among layer-wise penalty terms. The \(x\)-axis represents the layer number, and the \(y\)-axis represents the ratio of some layer-wise metric or penalty term to some total metric or penalty term. The network architecture is CNN7 (App. A.1), and the computation details are described in Sec. 3. Our penalty term (blue) is precisely proportional to the ground truth (gray).
spike, we demonstrate that this minor correction causes significant improvement.

#### Main Contributions

* We derived a novel penalty term that can directly optimize the metric for the total energy consumption of an SNN without modifying the network architecture.
* We demonstrated that the proposed method is compatible with the weight decay, which imposes implicit sparsity on the network (Yaguchi et al., 2018), and that the proposed method creates a higher sparsification effect than the weight decay.
* We demonstrated that the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy for image classification tasks, which mitigates the dilemma of the energy-accuracy trade-off.

## 2 Related Work

### Spike Sparsification in Direct SNN Training

The most relevant approaches to our proposal introduce the penalty term for the spike firing rate and directly train SNNs by the surrogate gradient method (Esser et al., 2016; Pellegrini et al., 2021). It is a straightforward idea to penalize the spike firing rate to obtain energy-efficient SNNs because it appears in the SNN energy consumption metric (Lee et al., 2020; Kim and Panda, 2021). We refer to the reduction in spike firing rate as spike sparsification. Although the spike firing rate cannot be optimized by the ordinal backpropagation method owing to non-differentiability, it can be optimized by the surrogate gradient method, which is the same technique as training spiking neurons of SNNs (Zenke and Ganguli, 2018; Shrestha and Orchard, 2018). However, neither of these penalty terms (Esser et al. (2016); Pellegrini et al. (2021)) precisely matches the energy consumption metric. As opposed to them, our spiking synaptic penalty resolves this limitation.

### Spike Sparsification via Conversion from ANN

Other approaches introduce the penalty term for corresponding ReLU networks (ANNs with ReLU activations) and convert them to SNNs (Sorbaro et al., 2020; Narduzzi et al., 2022). Although there is no guarantee that the penalty terms for ReLU networks contribute to the reduction of the energy consumption for converted SNNs, ReLU networks can be optimized by the ordinal backpropagation method. Note that the same synaptic scaling factor for the penalty term as ours is proposed to reduce the energy consumption for SNNs in Sorbaro et al. (2020). However, they failed to provide evidence to support their claim, as they mentioned. As opposed to them, we provide theoretical and experimental proof in the setting of the direct SNN training by the surrogate gradient.

### Neuron Sparsification

Neuron sparsification means increasing the number of permanently zero-valued activations--dead neurons. It is a stronger condition than spike sparsification, which does not force neurons to be permanently inactive. In ReLU networks, the training with the Adam optimizer and weight decay regularization implicitly induce neuron sparsification (Yaguchi et al., 2018) because ReLU activations have an inactive state. However, this claim has yet to be demonstrated in the context of SNNs, where spiking neurons also have an inactive state, even though weight decay is usually adopted in SNNs. Therefore, to detect the effect of our method correctly, we shall also focus on the weight decay.

## 3 Method

In this section, we propose the spiking synaptic penalty. First, we describe the spiking neuron model with surrogate gradient mechanism. Next, we describe the metric for energy consumption, which can be represented by the spiking activity. Note that we need to optimize both the accuracy and energy efficiency in the training phase. Finally, we state that the spiking synaptic penalty is the proper penalty term to optimize the energy consumption metric.

### Neuron Model and Surrogate Gradient

In this study, we uses SNNs constructed by single-step spiking neurons, which are superior to the multi-time step SNNs in terms of training and inference costs for static tasks (Suetake et al., 2023). Note that the single-step spiking neurons are the same setup as that in a previous study of the penalty term (Esser et al., 2016).

Let us denote \(l\in\{l\in\mathbf{Z}\mid 1\leq l\leq L\}\) as the layer, \(d_{l}\) as the number of neurons in the \(l\)-th layer, \(\mathbf{s}_{0}=x\in X\subset\mathbf{R}^{d_{0}}\) as the input data, and the subscript \(i\) of any vector as its \(i\)-th component. Then, the single-step spiking neuron is defined as follows (Suetake et al., 2023):

**Definition 3.1**.: The forward mode of a single-step spiking neuron consists of two ingredients, the membrane potential \(\mathbf{u}_{l}\in\mathbf{R}^{d_{l}}\) and spikes emitted by neurons \(\mathbf{s}_{l}\in\{0,1\}^{d_{l}}\). They are defined using the Heaviside step function \(H\) as follows:

\[\mathbf{u}_{l} :=\mathbf{W}_{l}\mathbf{s}_{l-1}, \tag{1}\] \[s_{l,i}\left(u_{l,i}\right) :=H\left(u_{l,i}-u_{\mathrm{th}}\right)=\left\{\begin{array}{l l}1&\left(u_{l,i}\geq u_{\mathrm{th}}\right),\\ 0&\left(u_{l,i}<u_{\mathrm{th}}\right),\end{array}\right. \tag{2}\]

where \(\mathbf{W}_{l}\in\mathbf{R}^{d_{l}\times d_{l-1}}\) is the strength of the synapse con
nections, also called the weight matrix, and \(u_{\mathrm{th}}\in\mathbf{R}\) is the spike firing threshold (Eq. 1 for \(l=1\) corresponds to the direct encoding (Rueckauer et al., 2017)).

A backward mode of the single-step neuron as it is does not work in the standard backpropagation algorithm because the derivative of Eq. 2 vanishes almost everywhere. Therefore, we adopt the technique called surrogate gradient, _i.e.,_ we formally replace the derivative function with some reasonable function, for example, the following one (Suetake et al., 2023):

\[\frac{\partial s_{l,i}}{\partial u_{l,i}}\left(u_{l,i}\right)\coloneqq\left\{ \begin{array}{ll}\frac{1}{\tau}\frac{1}{u_{l,i}}&\left(u_{l,i}\geq u_{ \mathrm{th}}\right),\\ \frac{\partial\sigma}{\partial u_{l,i}}\left(u_{l,i}\right)&\left(u_{l,i}<u_ {\mathrm{th}}\right),\end{array}\right. \tag{3}\]

where \(\tau\) and \(\alpha\) are hyperparameters and \(\sigma_{\alpha}\) is the scaled sigmoid function expressed as follows:

\[\sigma_{\alpha}\left(u_{l,i}\right) :=\frac{1}{1+\exp\left(\left(-u_{l,i}+u_{\mathrm{th}}\right)/ \alpha\right)}, \tag{4}\] \[\frac{\partial\sigma_{\alpha}}{\partial u_{l,i}}\left(u_{l,i} \right) =\frac{1}{\alpha}\sigma_{\alpha}\left(u_{l,i}\right)\left(1-\sigma_{ \alpha}\left(u_{l,i}\right)\right). \tag{5}\]

Note that the choice of a function for the surrogate function is irrelevant to our proposal.

### Metric for Energy Consumption

We prepare the symbol \(\psi_{l,i}\) for the number of synapses outgoing from the \(i\)-th neuron in the \(l\)-th layer, _i.e._, the number of matrix elements in \((\mathbf{W}_{l})_{\ast,i}\in\mathbf{R}^{d_{l}}\) that is not forced to vanish in terms of network architecture. Let us denote \(W_{l},H_{l}\) as the width and height of the feature map, respectively, \(C_{l}\) as the channel size, and \(k_{l}\) as the kernel size in the \(l\)-th layer. We restrict both the kernel width and height to be identical to \(k_{l}\) for the sake of simplicity. Then, explicit forms of \(\psi_{l,i}\) are as follows, _e.g._, the standard fully connected (fc) and two-dimensional convolutional (conv) layers,

\[\psi_{l,i}=\psi_{l}=\left\{\begin{array}{ll}C_{l+1}&\left(\mathrm{fc}\right),\\ \frac{W_{l+1}}{W_{l}H_{l}}C_{l+1}k_{l}^{2}&\left(\mathrm{conv}\right),\end{array}\right. \tag{6}\]

where the convolutional layer assumes appropriate padding. Note that some non-trivial padding, stride, and dilation can induce \(i\)-dependency of \(\psi_{l,i}\).

Using \(\psi_{l,i}\), we can express the number of floating point operations (FLOPs), which is often used as a metric to measure the computational complexity in ANNs, as follows:

\[\text{FLOPs}(l):=\sum_{i=1}^{d_{l}}\psi_{l,i}, \tag{7}\]

and the layer-wise and total spike firing rates, which are also important metrics to measure the sparsity of spiking activity in SNNs, as follows:

\[R(l) :=\mathop{\mathbb{E}}_{x\in X}\left[\frac{\sum_{i=1}^{d_{l}}s_{l,i }}{d_{l}}\right], \tag{8}\] \[R :=\sum_{l=1}^{L}R(l), \tag{9}\]

where the operation \(\mathop{\mathbb{E}}_{x\in X}\) means taking the empirical expectation in the dataset \(X\). Then, the energy consumption metric that we should optimize is defined as follows.

**Definition 3.2**.: Let us denote \(T\) as the size of time steps and \(E_{\text{AC}}\) as the energy consumption per accumulate operation. Then, the layer-wise and total energy consumption metrics for the SNN are defined as follows:

\[E_{\text{SNN}}(l) :=TE_{\text{AC}}\mathop{\mathbb{E}}_{x\in X}\left[\sum_{i=1}^{d_{l }}\psi_{l,i}s_{l,i}\right], \tag{10}\] \[E_{\text{SNN}} :=\sum_{l=1}^{L}E_{\text{SNN}}(l). \tag{11}\]

Note that \(T\) is equal to one for the single-step neuron model, and we use \(E_{\text{AC}}=0.9\) [J] (Horowitz, 2014).

If \(\psi_{l,i}\) is independent of \(i\) (\(\exists\psi_{l},\forall i,\psi_{l,i}=\psi_{l}\)), by combining Eqs. 7 and 8, Eq. 10 is rewritten as follows:

\[E_{\text{SNN}}(l) =TE_{\text{AC}}\sum_{i=1}^{d_{l}}\psi_{l}\mathop{\mathbb{E}}_{x \in X}\left[\frac{\sum_{i=1}^{d_{l}}s_{l,i}}{d_{l}}\right]\] \[=TE_{\text{AC}}\text{FLOPs}(l)R(l), \tag{12}\]

which is the same metric as that used in Kim and Panda (2021). For the sake of simplicity, we consider the case where \(\psi_{l,i}\) is independent of \(i\) in Sec. 4.

### Spiking Synaptic Penalty

To optimize the energy consumption \(E_{\text{SNN}}\) (Eq. 11), we propose the following penalty terms.

**Definition 3.3**.: The layer-wise and total spiking synaptic penalty terms are defined as follows:

\[\Omega_{\text{syn}}(l) =\Omega_{\text{syn}}(l,\mathbf{s}_{l}):=\frac{1}{p}\sum_{i=1}^{d_{l}} \psi_{l,i}s_{l,i}^{p}, \tag{13}\] \[\Omega_{\text{syn}} =\Omega_{\text{syn}}(\mathbf{s}):=\sum_{l=1}^{L}\Omega_{\text{syn}}(l, \mathbf{s}_{l}), \tag{14}\]

where \(\mathbf{s}:=\{\mathbf{s}_{l}\}_{l=1}^{L}\) and \(p\geq 1\).

The equivalency between the total energy consumption metric and total spiking synaptic penalty immediately follows from their definitions and the equation \(s_{l,i}^{p}=s_{l,i}\), which is derived from Eq. 2.
**Theorem 3.4**.: _The expected value of the layer-wise and total spiking synaptic penalties are precisely proportional to the layer-wise and total energy consumption metrics of SNNs:_

\[pE_{\text{AC}}\mathop{\mathbb{E}}_{x\in X}\left[\Omega_{\text{syn }}(l)\right] =E_{\text{SNN}}(l), \tag{15}\] \[pE_{\text{AC}}\mathop{\mathbb{E}}_{x\in X}\left[\Omega_{\text{syn }}\right] =E_{\text{SNN}}, \tag{16}\]

_for arbitrary \(p\geq 1\)._

This fact means that optimizing Eq. 14 leads to optimizing Eq. 11. Hence, we strongly propose to use Eq. 14 as the penalty term to optimize the energy consumption metric. In the following, we indicate the total spiking synaptic penalty when simply referred to as the _spiking synaptic penalty_ (including Eqs. 9 and 11).

_Remark 3.5_.: The proposed penalty term can be optimized in the manner of the surrogate gradient as in Sec. 3.1, and using \(p\neq 1\) options controls the backward signal when the spike does not fire as follows:

\[\frac{1}{p}\frac{\partial s_{l,i}^{p}}{\partial u_{l,i}}\left(u_{ l,i}\right) =s_{l,i}^{p-1}\frac{\partial s_{l,i}}{\partial u_{l,i}}\] \[\simeq\left\{\begin{array}{ll}\frac{\partial s_{l,i}}{ \partial u_{l,i}}&\left(u_{l,i}\geq u_{\text{th}}\right),\\ 0&\left(u_{l,i}<u_{\text{th}}\right),\end{array}\right. \tag{17}\]

where we used Eq. 2. From Eqs. 13 and 17, there is no intrinsic difference when \(p>1\); hence, we do not consider \(p>1\) options except for \(p=2\), which is commonly used in several studies (Esser et al., 2016; Pellegrini et al., 2021). Note that the derivation of Eq. 17 is clearly independent of the choice of the surrogate gradient. However, the open problem still remains, _i.e.,_ it cannot be theoretically decided which choice is better, \(p=1\) or \(p>1\). We experimentally examined it for \(p=1,2\) as described in Sec. 4.

#### 3.3.1 Differences from Other Penalty Terms

The other candidates for the penalty term are as follows:

\[\Omega_{\text{total}} =\frac{1}{p}\sum_{l=1}^{L}\sum_{i=1}^{d_{l}}s_{l,i}^{p}, \tag{18}\] \[\Omega_{\text{balance}} =\frac{1}{p}\sum_{l=1}^{L}\sum_{i=1}^{d_{l}}\frac{1}{d_{l}}s_{l, i}^{p}, \tag{19}\]

where, for \(p=2\), \(\Omega_{\text{total}}\) and \(\Omega_{\text{balance}}\) are the same as those in Esser et al. (2016) and Pellegrini et al. (2021), respectively. However, we claim that neither can directly optimize the energy consumption because they do not have the proportional nature (Eq. 16), although they sparsify the spiking activity to some extent.

Figs. 1 and 2 and Table 1 show the discrepancy between the energy consumption metric and penalty terms of the model used in the following experiment. In these figures and table, we assumed that all the spiking neurons fired, _i.e._, \(s_{l,i}=1(\forall l,i)\), for the sake of simplicity. In this assumption, the ground truth is proportional to FLOPs without loss of generality. These figures and table indicate that the spiking synaptic penalty is precisely proportional to the energy consumption metric, but other penalties are not. In the next section, we will experimentally verify how this claim affects performance.

#### 3.3.2 Normalization of Penalty Terms

Penalty terms are included in the objective function with their intensity parameter \(\lambda\) as the coupling \(\lambda\Omega_{*}\), where the symbol \(*\) denotes "syn", "total", or "balance". For tractable treatment of the intensity parameter between various penalty terms or among models of various scales, we recommend normalizing the penalty terms by \(\Omega_{*}(\mathbf{1})\), where \(\mathbf{1}\) indicates \(\mathbf{s}=\mathbf{1}\), _i.e._, \(s_{l,i}=1\,\forall l,i\). Note that replacing \(\Omega_{*}\) with

\begin{table}
\begin{tabular}{l r r r r} \hline \hline Model & \(E_{\text{SNN}}/E_{\text{AC}}\) & \(\Omega_{\text{syn}}\) & \(\Omega_{\text{total}}\) & \(\Omega_{\text{balance}}\) \\ \hline CNN7 & 98895888 & 98895888 & 59688 & 6 \\ VGG11 & 2526060544 & 2526060544 & 249856 & 10 \\ ResNet18 & 553730048 & 553730048 & 671744 & 20 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison among total penalty terms. Our penalty term \(\Omega_{\text{syn}}\) is precisely equal to the ground truth \(E_{\text{SNN}}/E_{\text{AC}}\).

Figure 2: Comparison among layer-wise penalty terms. The network architectures are (A) VGG11 and (B) ResNet18 (App. A.1). As in Fig. 1, our penalty term (blue) is precisely proportional to the ground truth (gray).
\(\Omega_{*}/\Omega_{*}(\mathbf{1})\) is equivalent to replacing \(\lambda\) with

\[\lambda^{\prime}=\lambda/\Omega_{*}(\mathbf{1}). \tag{20}\]

Hence, we sometimes adopt the normalized notation \(\lambda^{\prime}\) instead of \(\lambda\).

## 4 Experiment

In this section, we evaluate the effectiveness of the proposed spiking synaptic penalty. First, we describe the setup for experiments. Next, we show that the proposed method can decrease energy consumption. Finally, we show that the proposed method can reduce the energy consumption more than other methods while maintaining the accuracy. In particular, we show that the proposed method is independent of the choice of a function for the surrogate gradient and better than the conversion approach (Sorbaro et al., 2020). Overall, the main objective is to analyze the behavior of our method rather than to achieve state-of-the-art performances.

### Experimental Setup

As the single-step spiking neuron is developed for static tasks (Suetake et al., 2023), we experimented for the Fashion-MNIST (Xiao et al., 2017), CIFAR-10, and CIFAR-100 (Krizhevsky, 2009) datasets widely used in SNN experiments (Esser et al., 2016; Zhang and Li, 2020; Chowdhury et al., 2021) with some network architectures, CNN7, VGG11, and ResNet18 (refer to App. A.1 for details). For these experiments, we implemented the program by the PyTorch framework and used one GPU, an NVIDIA GeForce RTX 3090, with 24 GB (refer to Table B.1 in the appendix for the difference in the training time).

In these experiments, we used the following objective function:

\[L= \frac{1}{n}\sum_{n=1}^{N}\left(CE(f(x_{n}),t_{n})+\lambda\Omega_{ \mathrm{syn}}(\mathbf{s}(x_{n}))\right)\] \[+\lambda_{\mathrm{WD}}\left(\|\mathbf{W}\|_{L_{2}}^{2}+B_{\mathrm{BN }}\|\mathbf{W}_{\mathrm{BN}}\|_{L_{2}}^{2}\right), \tag{21}\]

where \((x_{n},t_{n})\) denotes the pair of input data and its label, \(f\) denotes some spiking neural network, \(CE\) denotes the cross-entropy function, \(\|\mathbf{W}\|_{L_{2}}^{2}\) denotes the \(L_{2}\) penalty for the weights, _i.e._, weight decay, \(\|\mathbf{W}_{\mathrm{BN}}\|_{L_{2}}^{2}\) denotes the \(L_{2}\) penalty for the trainable parameters of batch normalization layers, \(\lambda\) and \(\lambda_{\mathrm{WD}}\) denote the intensity of penalties, and \(B_{\mathrm{BN}}\in\{0,1\}\). Note that we included the weight decay into the objective function to verify its sparsifying effect (Yaguchi et al., 2018) in the context of SNNs. In addition, we explicitly specified the weight decay for batch normalization layers because it was included in the default setting of the PyTorch framework (Paszke et al., 2019) but not in Yaguchi et al. (2018). The training of \(f\) was done by the backpropagation algorithm with the surrogate gradient of Eq. 3 unless otherwise stated. The optimizer was selected from the momentum SGD (mSGD) or Adam to confirm the claims in Yaguchi et al. (2018).

Refer to App. A for further details of the experimental setup such as hyperparameters.

### Energy Reduction by Spiking Synaptic Penalty

We investigated whether optimizing the spiking synaptic penalty (Eq. 14) led to optimizing the energy consumption metric (Eq. 11) and whether there was any conflict with other terms in the objective function (Eq. 21). The setting was as follows. The baseline model was trained for Eq. 21 with \(\lambda=\lambda_{\mathrm{WD}}=0\). The other models were trained from scratch with some combinations of the weight decay (\(\lambda_{\mathrm{WD}}>0\)), \(L_{2}\) penalty for batch normalization layers (\(B_{\mathrm{BN}}=1\)), and \(p=1\) spiking synaptic penalty (\(\lambda>0\)). The results are presented in Tables 2 and 3. Note that the values in this table were taken for \(\lambda\) and \(\lambda_{\mathrm{WD}}\) from a point where the accuracy was very low (approximately \(20\%\)) until the accuracy reached the upper bound and stopped changing.

From the result in Table 2, we can observe the following. First, as the intensity of the penalty term increases, the energy consumption metric decreases; the inference accuracy also decreases. Therefore, the intensity parameter \(\lambda\) controls the trade-off between them. Second, the combination of the spiking synaptic penalty and weight decay further reduces the energy consumption metric. Therefore, we propose to adopt both of them simultaneously. In addition, we found that the combination of the weight decay and Adam optimizer induces neuron sparsification even without the spiking synaptic penalty, though its contribution to the energy reduction is less than the spiking synaptic penalty. Furthermore, the neuron sparsification proceeds more strongly for the Adam optimizer than the mSGD optimizer, which is consistent with the claim in Yaguchi et al. (2018) (see also Table 3). Note that we cannot observe a remarkable difference between \(B_{\mathrm{BN}}=0\) and \(1\). Therefore, we adopt the weight decay with \(B_{\mathrm{BN}}=0\) in further experiments to simplify our objective function. Finally, all the above results hold for not only VGG11 but also CNN7 and ResNet18 (see Tables 8-11 in the appendix).

### Trade-off between Accuracy and Energy Efficiency

#### 4.3.1 Comparison between Penalties

To see how the difference between various penalty terms affected the training result, we conducted a comparative experiment. The setting was as follows. For fair comparison, we used the \(\lambda^{\prime}\) notation for the intensity parameter of penalties rather than the raw \(\lambda\) (see Eq. 20). The baseline model was trained for Eq. 21 with \(\lambda^{\prime}=B_{\mathrm{BN}}=0\), and we tuned
\(\lambda_{\rm WD}>0\) to obtain the highest accuracy. Then, the others were trained by varying \(\lambda^{\prime}>0\) and by replacing \(\Omega_{\rm syn}\) in Eq. 21 with \(\Omega_{\rm total}\) or \(\Omega_{\rm balance}\) from scratch. The results are shown in Fig. 3 (A) as \(\lambda^{\prime}\)-parameterized curves of the energy-accuracy trade-off, where the energy consumption rate was produced as the energy consumption of each model normalized by that of the baseline model. Note that it is better for data to be located at the upper left corner in the figure. Refer to App. A.3 for the sampling of \(\lambda^{\prime}\). In addition, the quantitative analysis is presented in Table 4, where higher scores are better for all the metrics: area under the curve (AUC), Spearman's rank correlation coefficient (Spearman), and the mutual information (MI). Note that the argument of each metric represents a cutoff parameter, where data with lower accuracy than it are omitted. We introduced the cutoff parameter because training tended to break as the intensity parameter was increased for all the methods. Refer to App. A.4 for details of quantitative metrics.

From the result in Fig. 3 (A) and Table 4, we can observe the following. First, for each \(\Omega_{*}\), the \(p=1\) option is apparently better than the \(p=2\) option. Therefore, we propose to adopt the \(p=1\) option. Note that this difference arises from the backward control as Eq. 17. However, the mechanism by which it affects the training result is still unclear; it remains our future work. Second, for \(p=1\), the trade-off curve of \(\Omega_{\rm syn}\) is the best, followed in order by \(\Omega_{\rm total}\) and \(\Omega_{\rm balance}\). Therefore, we experimentally clarified the advantage of the coefficient \(\psi_{l,i}\) for Eq. 13, which had remained an issue in the method proposed by Sorbaro et al. (2020). Finally, all the above results hold for not only CNN7 but also VGG11 and ResNet18 (see Fig. 5 and Tables 12-17 in the appendix).

#### 4.3.2 Independence of Surrogate Gradients

To see how the choice of a function for the surrogate gradient affected the training result, we conducted the same experiment as that in Sec. 4.3.1 except for the choice of a function. Instead of Eq. 3, we adopted the piece-wise linear function (Esser et al., 2016) and scaled sigmoid (Pellegrini et al., 2021) function for the surrogate gradient as follows:

\[\frac{\partial s}{\partial u} \simeq\max\left(1-|u-u_{\rm th}|,0\right), \tag{22}\] \[\frac{\partial s}{\partial u} \simeq\frac{\partial\sigma_{\alpha}}{\partial u}, \tag{23}\]

where \(\sigma_{\alpha}\) is the same as Eq. 5. The results are presented in Figs. 3 (B) and (C), and Tables 18 and 19 in the appendix.

From the result in Figs. 3 (B) and (C), the same observations as those in Sec. 4.3.1 hold. That is, the \(p=1\) option is apparently better than the \(p=2\) option; the trade-off curve of \(\Omega_{\rm syn}\) is the best, followed in order by \(\Omega_{\rm total}\) and \(\Omega_{\rm balance}\). Therefore, the spiking synaptic penalty works independent of the choice of a function for the surrogate gradient.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \(\lambda\) & \(\lambda_{\rm WD}\) & \(B_{\rm BN}\) & Accuracy [\%] & \(E_{\rm SNN}/E_{\rm baseline}\) [\%] & Dead rate [\%] & \(R\) [\%] \\ \hline

[MISSING_PAGE_POST]

 \hline \hline \end{tabular}
\end{table}
Table 2: Performance with respect to varying \(\lambda\), \(\lambda_{\rm WD}\), and \(B_{\rm BN}\). The network architecture is VGG11, the dataset is CIFAR-10, and the optimizer is Adam. \(E_{\rm SNN}\) denotes the energy consumption metric for the SNN (Eq. 11), \(E_{\rm baseline}\) denotes the \(E_{\rm SNN}\) for the model with \(\lambda=\lambda_{\rm WD}=B_{\rm BN}=0\), dead rate denotes the ratio of the number of dead neurons to the number of total neurons, and \(R\) denotes the spike firing rate (Eq. 9).
#### 4.3.3 Superiority to Conversion Approach

To see how the difference between training methods affected the training result, we also produced the trade-off curve for the conversion approach (Sorbaro et al., 2020). We trained a single QReLU network (an ANN with quantized ReLU activations) increasing the intensity of the penalty and evaluated the converted SNNs for each intensity in different time steps: \(T=1,5\), and \(10\) (refer to the original paper (Sorbaro et al., 2020) for details). The results are presented in Fig. 3 (D) and Table 20 in the appendix, where the energy consumption was normalized by that of the baseline for \(\Omega_{\mathrm{syn}}\).

From the result in Fig. 3 (D), we can observe that both the energy consumption and accuracy for the conversion approach are worse than those for the surrogate gradient approach. This is because the conversion process degrades the accuracy, and the penalty term for the QReLU network cannot directly optimize the energy consumption metric for the SNN. Hence, we should directly train SNNs by the surrogate gradient and spiking synaptic penalty to avoid such degradation.

## 5 Conclusion

We studied the training method to obtain energy-efficient SNNs in terms of the surrogate gradient. Based on our principle that we should optimize the metric as it is, we derived the spiking synaptic penalty to optimize the energy consumption metric. Then, we experimentally showed that the spiking synaptic penalty (especially for \(p=1\)) is superior to the existing penalties and conversion approach. Furthermore, its effectiveness is indifferent to the type of network

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline Method & AUC(\(70\))[\(\%\)] & AUC(\(50\))[\(\%\)] & Spearman(\(70\)) & Spearman(\(50\)) & MI(\(70\)) & MI(\(50\)) \\ \hline \hline \(\Omega_{\mathrm{syn}}\left(p=1\right)\) (Ours) & **68.02** & **79.60** & **0.9861** & **0.9865** & **3.465** & **3.610** \\ \(\Omega_{\mathrm{syn}}\left(p=2\right)\) (Ours) & 61.62 & 72.69 & 0.9474 & 0.9709 & 3.233 & 3.476 \\ \(\Omega_{\mathrm{total}}\left(p=1\right)\) & 63.30 & 76.05 & 0.9766 & 0.9767 & 3.244 & 3.319 \\ \(\Omega_{\mathrm{total}}\left(p=2\right)\) & 55.16 & 64.63 & 0.9831 & 0.9701 & 3.218 & 3.295 \\ \(\Omega_{\mathrm{balance}}\left(p=1\right)\) & 54.23 & 67.16 & 0.9412 & 0.9412 & 2.978 & 2.978 \\ \(\Omega_{\mathrm{balance}}\left(p=2\right)\) & 31.47 & 42.94 & 0.8500 & 0.8946 & 2.708 & 2.833 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Quantitative comparison corresponding to Fig. 3 (A). Higher scores are better. The best and the second-best results are highlighted in bold and underlined, respectively. Refer to App. A.4 for details of the quantitative metrics.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \(\lambda\) & \(\lambda_{\mathrm{WD}}\) & \(B_{\mathrm{BN}}\) & Accuracy [\%] & \(E_{\mathrm{SNN}}/E_{\mathrm{baseline}}\) [\%] & Dead rate [\%] & \(R\) [\%] \\ \hline

[MISSING_PAGE_POST]

 \hline \hline \end{tabular}
\end{table}
Table 3: Performance with respect to varying \(\lambda,\lambda_{\mathrm{WD}}\), and \(B_{\mathrm{BN}}\). The optimizer is
architectures and surrogate gradients. We conclude that our principle has worked well.

An apparent limitation is that the definition of the spiking synaptic penalty depends on that of the energy consumption metric. However, if the target metric becomes deformed, the penalty should be accordingly deformed in accordance with our principle--even though it is a metric irrelevant to the energy consumption. Another limitation is that although the target metric is directly included in the objective function, it is just indirectly optimized by the surrogate gradient. To overcome this issue, we need an alternative training method such as equilibrium training (Xiao et al., 2021).

We further list some outstanding issues. First, it is unclear why there was a difference between the training result for \(p=1\) and \(2\) of the spiking synaptic penalty. Elucidating the mechanism of this difference could help us understand the surrogate gradient. Second, we did not focus on the synergy between the spike sparsification and pruning. A pruning-aware sparsification training will help us obtain more energy-efficient SNNs. Finally, the high availability of the spiking synaptic penalty should be verified, for example, in the case of real datasets, large networks, and other tasks. By solving these issues, we can contribute to the realization of genuinely eco-friendly SNNs.

To facilitate further research, we will release the code shortly.
