Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency
  Spiking Neural Networks
Spiking Neural Networks (SNNs) have gained great attraction due to their
distinctive properties of low power consumption and fast inference on
neuromorphic hardware. As the most effective method to get deep SNNs, ANN-SNN
conversion has achieved comparable performance as ANNs on large-scale datasets.
Despite this, it requires long time-steps to match the firing rates of SNNs to
the activation of ANNs. As a result, the converted SNN suffers severe
performance degradation problems with short time-steps, which hamper the
practical application of SNNs. In this paper, we theoretically analyze ANN-SNN
conversion error and derive the estimated activation function of SNNs. Then we
propose the quantization clip-floor-shift activation function to replace the
ReLU activation function in source ANNs, which can better approximate the
activation function of SNNs. We prove that the expected conversion error
between SNNs and ANNs is zero, enabling us to achieve high-accuracy and
ultra-low-latency SNNs. We evaluate our method on CIFAR-10/100 and ImageNet
datasets, and show that it outperforms the state-of-the-art ANN-SNN and
directly trained SNNs in both accuracy and time-steps. To the best of our
knowledge, this is the first time to explore high-performance ANN-SNN
conversion with ultra-low latency (4 time-steps). Code is available at
https://github.com/putshua/SNN\_conversion\_QCFS
Tong Bu, Wei Fang, Jianhao Ding, PengLin Dai, Zhaofei Yu, Tiejun Huang
# Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks

###### Abstract

Spiking Neural Networks (SNNs) have gained great attraction due to their distinctive properties of low power consumption and fast inference on neuromorphic hardware. As the most effective method to get deep SNNs, ANN-SNN conversion has achieved comparable performance as ANNs on large-scale datasets. Despite this, it requires long time-steps to match the firing rates of SNNs to the activation of ANNs. As a result, the converted SNN suffers severe performance degradation problems with short time-steps, which hamper the practical application of SNNs. In this paper, we theoretically analyze ANN-SNN conversion error and derive the estimated activation function of SNNs. Then we propose the quantization clip-floor-shift activation function to replace the ReLU activation function in source ANNs, which can better approximate the activation function of SNNs. We prove that the expected conversion error between SNNs and ANNs is zero, enabling us to achieve high-accuracy and ultra-low-latency SNNs. We evaluate our method on CIFAR-10/100 and ImageNet datasets, and show that it outperforms the state-of-the-art ANN-SNN and directly trained SNNs in both accuracy and time-steps. To the best of our knowledge, this is the first time to explore high-performance ANN-SNN conversion with ultra-low latency (4 time-steps). Code is available at [https://github.com/putshua/SNN_conversion_QCFS](https://github.com/putshua/SNN_conversion_QCFS)

## 1 Introduction

Spiking neural networks (SNNs) are biologically plausible neural networks based on the dynamic characteristic of biological neurons (McCulloch and Pitts, 1943; Izhikevich, 2003). As the third generation of artificial neural networks (Maass, 1997), SNNs have attracted great attention due to their distinctive properties over deep analog neural networks (ANNs) (Roy et al., 2019). Each neuron transmits discrete spikes to convey information when exceeding a threshold. For most SNNs, the spiking neurons will accumulate the current of the last layer as the output within \(T\) inference time steps. The binarized activation has rendered dedicated hardware of neuromorphic computing (Pei et al., 2019; DeBole et al., 2019; Davies et al., 2018). This kind of hardware has excellent advantages in temporal resolution and energy budget. Existing work has shown the potential of tremendous energy saving with considerably fast inference (Stockl and Maass, 2021).

In addition to efficiency advantages, the learning algorithm of SNNs has been improved by leaps and bounds in recent years. The performance of SNNs trained by backpropagation through time and ANN-SNN conversion techniques has gradually been comparable to ANNs on large-scale datasets (Fang et al., 2021; Rueckauer et al., 2017). Both techniques benefit from the setting of SNN inference time. Setting longer time-steps in backpropagation can make the gradient of surrogate functions more reliable (Wu et al., 2018; Neftci et al., 2019; Zenke and Vogels, 2021). However, the price is enormous resource consumption during training. Existing platforms such as TensorFlow and PyTorch based on CUDA have limited optimization for SNN training. In contrast, ANN-SNN conversion usually depends on a longer inference time to get comparable accuracy as the original ANN (Sengupta et al., 2019) because it is based on the equivalence of ReLU activation and integrate-and-fire model's firing rate (Cao et al., 2015). Although longer inference time can further reduce the conversion error, it also hampers the practical application of SNNs on neuromorphic chips.
The dilemma of ANN-SNN conversion is that there exists a remaining potential in the conversion theory, which is hard to be eliminated in a few time steps (Rueckauer et al., 2016). Although many methods have been proposed to improve the conversion accuracy, such as weight normalization (Diehl et al., 2015), threshold rescaling (Sengupta et al., 2019), soft-reset (Han and Roy, 2020) and threshold shift (Deng and Gu, 2020), tens to hundreds of time-steps in the baseline works are still unbearable. To obtain high-performance SNNs with ultra-low latency (e.g., 4 time-steps), we list the critical errors in ANN-SNN conversion and provide solutions for each error. Our main contributions are summarized as follows:

* We go deeper into the errors in the ANN-SNN conversion and ascribe them to clipping error, quantization error, and unevenness error. We find that unevenness error, which is caused by the changes in the timing of arrival spikes and has been neglected in previous works, can induce more spikes or fewer spikes as expected.
* We propose the quantization clip-floor-shift activation function to replace the ReLU activation function in source ANNs, which better approximates the activation function of SNNs. We prove that the expected conversion error between SNNs and ANNs is zero, indicating that we can achieve high-performance converted SNN at ultra-low time-steps.
* We evaluate our method on CIFAR-10, CIFAR-100, and ImageNet datasets. Compared with both ANN-SNN conversion and backpropagation training methods, the proposed method exceeds state-of-the-art accuracy with fewer time-steps. For example, we reach top-1 accuracy 91.18% on CIFAR-10 with unprecedented 2 time-steps.

## 2 Preliminaries

In this section, we first briefly review the neuron models for SNNs and ANNs. Then we introduce the basic framework for ANN-SNN conversion.

**Neuron model for ANNs.** For ANNs, the computations of analog neurons can be simplified as the combination of a linear transformation and a non-linear mapping:

\[\mathbf{a}^{l}=h(\mathbf{W}^{l}\mathbf{a}^{l-1}),\ \ \ l=1,2,...,M \tag{1}\]

where the vector \(\mathbf{a}^{l}\) denotes the output of all neurons in \(l\)-th layer, \(\mathbf{W}^{l}\) denotes the weight matrix between layer \(l\) and layer \(l-1\), and \(h(\cdot)\) is the ReLU activation function.

**Neuron model for SNNs.** Similar to the previous works (Cao et al., 2015; Diehl et al., 2015; Han et al., 2020), we consider the Integrate-and-Fire (IF) model for SNNs. If the IF neurons in \(l\)-th layer receive the input \(\mathbf{x}^{l-1}(t)\) from last layer, the temporal potential of the IF neurons can be defined as:

\[\mathbf{m}^{l}(t)=\mathbf{v}^{l}(t-1)+\mathbf{W}^{l}\mathbf{x}^{l-1}(t), \tag{2}\]

where \(\mathbf{m}^{l}(t)\) and \(\mathbf{v}^{l}(t)\) represent the membrane potential before and after the trigger of a spike at time-step \(t\). \(\mathbf{W}^{l}\) denote the weight in \(l\)-th layer. As soon as any element \(m^{l}_{i}(t)\) of \(\mathbf{m}^{l}(t)\) exceeds the firing threshold \(\theta^{l}\), the neuron will elicit a spike and update the membrane potential \(v^{l}_{i}(t)\). To avoid information loss, we use the "reset-by-subtraction" mechanism (Rueckauer et al., 2017; Han et al., 2020) instead of the "reset-to-zero" mechanism, which means the membrane potential \(v^{l}_{i}(t)\) is subtracted by the threshold value \(\theta^{l}\) if the neuron fires. Based on the threshold-triggered firing mechanism and the "reset-by-subtraction" of the membrane potential after firing discussed above, we can write the update rule of membrane potential as:

\[\mathbf{s}^{l}(t)=H(\mathbf{m}^{l}(t)-\mathbf{\theta}^{l}), \tag{3}\] \[\mathbf{v}^{l}(t)=\mathbf{m}^{l}(t)-\mathbf{s}^{l}(t)\mathbf{\theta}^{l}. \tag{4}\]

Here \(\mathbf{s}^{l}(t)\) refers to the output spikes of all neurons in layer \(l\) at time \(t\), the element of which equals 1 if there is a spike and 0 otherwise. \(H(\cdot)\) is the Heaviside step function. \(\mathbf{\theta}^{l}\) is the vector of the firing threshold \(\theta^{l}\). Similar to Deng and Gu (2020), we suppose that the postsynaptic neuron in \(l\)-th layer receives unweighted postsynaptic potential \(\theta^{l}\) if the presynaptic neuron in \(l-1\)-th layer fires a spike, that is:

\[\mathbf{x}^{l}(t)=\mathbf{s}^{l}(t)\theta^{l}. \tag{5}\]
ANN-SNN conversion.The key idea of ANN-SNN conversion is to map the activation value of an analog neuron in ANN to the firing rate (or average postsynaptic potential) of a spiking neuron in SNN. Specifically, we can get the potential update equation by combining Equation 2 - Equation 4:

\[\mathbf{v}^{l}(t)-\mathbf{v}^{l}(t-1)=\mathbf{W}^{l}\mathbf{x}^{l-1}(t)-\mathbf{s}^{l}(t)\theta^{l}. \tag{6}\]

Equation 6 describes the basic function of spiking neurons used in ANN-SNN conversion. By summing Equation 6 from time \(1\) to \(T\) and dividing \(T\) on both sides, we have:

\[\frac{\mathbf{v}^{l}(T)-\mathbf{v}^{l}(0)}{T}=\frac{\mathbf{W}^{l}\sum_{i=1}^{T}\mathbf{x}^{l- 1}(i)}{T}-\frac{\sum_{i=1}^{T}\mathbf{s}^{l}(i)\theta^{l}}{T}. \tag{7}\]

If we use \(\phi^{l-1}(T)=\frac{\sum_{i=1}^{T}\mathbf{x}^{l-1}(i)}{T}\) to denote the average postsynaptic potential during the period from 0 to \(T\) and substitute Equation 5 into Equation 7, then we get:

\[\mathbf{\phi}^{l}(T)=\mathbf{W}^{l}\phi^{l-1}(T)-\frac{\mathbf{v}^{l}(T)-\mathbf{v}^{l}(0)}{T}. \tag{8}\]

Equation 8 describes the relationship of the average postsynaptic potential of neurons in adjacent layers. Note that \(\mathbf{\phi}^{l}(T)\geqslant 0\). If we set the initial potential \(\mathbf{v}^{l}(0)\) to zero and neglect the remaining term \(\frac{\mathbf{v}^{l}(T)}{T}\) when the simulation time-steps \(T\) is long enough, the converted SNN has nearly the same activation function as source ANN (Equation 1). However, high \(T\) would cause long inference latency that hampers the practical application of SNNs. Therefore, this paper aims to implement high-performance ANN-SNN conversion with extremely low latency.

## 3 Conversion error analysis

In this section, we will analyze the conversion error between the source ANN and the converted SNN in each layer in detail. In the following, we assume that both ANN and SNN receive the same input from the layer \(l-1\), that is, \(\mathbf{a}^{l-1}=\mathbf{\phi}^{l-1}(T)\), and then analyze the error in layer \(l\). For simplicity, we use \(\mathbf{z}^{l}=\mathbf{W}^{l}\mathbf{\phi}^{l-1}(T)=\mathbf{W}^{l}\mathbf{a}^{l-1}\) to substitute the weighted input from layer \(l-1\) for both ANN and SNN. The absolute conversion error is exactly the outputs from converted SNN subtract the outputs from ANN:

\[\mathbf{E}\mathbf{r}\mathbf{r}^{l}=\mathbf{\phi}^{l}(T)-\mathbf{a}^{l}=\mathbf{z}^{l}-\frac{\mathbf{v}^{l} (T)-\mathbf{v}^{l}(0)}{T}-h(\mathbf{z}^{l}), \tag{9}\]

where \(h(\mathbf{z}^{l})=\text{ReLU}(\mathbf{z}^{l})\). It can be found from Equation 9 that the conversion error is nonzero if \(\mathbf{v}^{l}(T)-\mathbf{v}^{l}(0)\neq 0\) and \(\mathbf{z}^{l}>0\). In fact, the conversion error is caused by three factors.

**Clipping error.** The output \(\mathbf{\phi}^{l}(T)\) of SNNs is in the range of \([0,\theta^{l}]\) as \(\mathbf{\phi}^{l}(T)=\frac{\sum_{i=1}^{T}\mathbf{x}^{l}(i)}{T}=\frac{\sum_{i=1}^{T}\bm {s}^{l}(i)}{T}\theta^{l}\) (see Equation 5). However, the output \(\mathbf{a}^{l}\) of ANNs is in a much layer range of \([0,a_{max}^{l}]\), where \(a_{max}^{l}\) denotes the maximum value of \(\mathbf{a}^{l}\). As illustrated in Figure 0(a), \(\mathbf{a}^{l}\) can be mapped to \(\mathbf{\phi}^{l}(T)\) by the following equation:

\[\mathbf{\phi}^{l}(T)=\text{clip}\left(\frac{\theta^{l}}{T}\left\lfloor\frac{a^{l} T}{\lambda^{l}}\right\rfloor,0,\theta^{l}\right). \tag{10}\]

\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Symbol** & **Definition** & **Symbol** & **Definition** \\ \hline \(l\) & Layer index & \(\mathbf{x}^{l}(t)\) & Unweighted PSP\({}^{l}\) \\ \(i\) & Neuron index & \(\mathbf{s}^{l}(t)\) & Output spikes \\ \(\mathbf{W}^{l}\) & Weight & \(\mathbf{\phi}^{l}(T)\) & Average unweighted PSP before time \(T\) \\ \(\mathbf{a}^{l}\) & ANN activation values & \(\mathbf{z}^{l}\) & Weighted input from \(l-1\) layer \\ \(t\) & Time-steps & \(h(\cdot)\) & ReLU function \\ \(T\) & Total time-step & \(H(\cdot)\) & Heaviside step function \\ \(\mathbf{\theta}^{l}\) & Threshold & \(L\) & Quantization step for ANN \\ \(\lambda^{l}\) & Trainable threshold in ANN & \(\mathbf{E}\mathbf{r}\mathbf{r}^{l}\) & Conversion Error \\ \(\mathbf{m}^{l}(t)\) & Potential before firing & \(\widehat{\mathbf{E}\mathbf{r}^{l}}\) & Estimated conversion Error \\ \(\mathbf{v}^{l}(t)\) & Potential after firing & \(\varphi\) & Shift of quantization clip-floor function \\ \hline \hline \end{tabular}

* Postsynaptic potential

\end{table}
Table 1: Summary of notations in this paper
Here the clip function sets the upper bound \(\theta^{l}\) and the lower bound \(0\). \(\lfloor\cdot\rfloor\) denotes the floor function. \(\lambda^{l}\) represents the actual maximum value of output \(\mathbf{a}^{l}\) mapped to the maximum value \(\theta^{l}\) of \(\mathbf{\phi}^{l}(T)\). Considering that nearly 99.9% activations of \(\mathbf{a}^{l}\) in ANN are in the range of \([0,\frac{a_{max}}{3}]\), Rueckauer et al. (2016) suggested to choose \(\lambda^{l}\) according to 99.9% activations. The activations between \(\lambda^{l}\) and \(a_{max}^{l}\) in ANN are mapped to the same value \(\theta^{l}\) in SNN, which will cause conversion error called clipping error.

**Quantization error (flooring error).** The output spikes \(\mathbf{s}^{l}(t)\) are discrete events, thus \(\mathbf{\phi}^{l}(T)\) are discrete with quantization resolution \(\frac{\theta^{l}}{T}\) (see Equation 10). When mapping \(\mathbf{a}^{l}\) to \(\mathbf{\phi}^{l}(T)\), there exists unavoidable quantization error. For example, as illustrated in Figure 0(a), the activations of ANN in the range of \([\frac{\lambda^{l}}{T},\frac{\Delta^{l}}{T^{l}})\) are mapped to the same value \(\frac{\theta^{l}}{T}\) of SNN.

**Unevenness error.** Unevenness error is caused by the unevenness of input spikes. If the timing of arrival spikes changes, the output firing rates may change, which causes conversion error. There are two situations: more spikes as expected or fewer spikes as expected. To see this, in source ANN, we suppose that two analog neurons in layer \(l-1\) are connected to an analog neuron in layer \(l\) with weights 2 and -2, and the output vector \(\mathbf{a}^{l-1}\) of neurons in layer \(l-1\) is \([0.6,0.4]\). Besides, in converted SNN, we suppose that the two spiking neurons in layer \(l-1\) fire 3 spikes and 2 spikes in 5 time-steps (T=5), respectively, and the threshold \(\theta^{l-1}=1\). Thus, \(\mathbf{\phi}^{l-1}(T)=\frac{\sum_{i=1}^{T}\mathbf{s}^{l-1}(i)}{T}\theta^{l-1}=[0.6,0.4]\). Even though \(\mathbf{\phi}^{l-1}(T)=\mathbf{a}^{l-1}\) and the weights are same for the ANN and SNN, \(\mathbf{\phi}^{l}(T)\) can be different from \(\mathbf{a}^{l}\) if the timing of arrival spikes changes. According to Equation 1, the ANN output \(\mathbf{a}^{l}=\mathbf{W}^{l}\mathbf{a}^{l-1}=[2,-2][0.6,0.4]^{T}=0.4\). As for SNN, supposing that the threshold \(\theta^{l}=1\), there are three possible output firing rates, which are illustrated in Figure 1 (b)-(d). If the two presynaptic neurons fires at \(t=1,3,5\) and \(t=2,4\) (red bars) respectively with weights 2 and -2, the postsynaptic neuron will fire two spikes at \(t=1,3\) (red bars), and \(\mathbf{\phi}^{l}(T)=\frac{\sum_{i=1}^{T}\mathbf{s}^{l}(i)}{\theta^{l}}=0.4=\mathbf{a}^{l}\). However, if the presynaptic neurons fires at \(t=1,2,3\) and \(t=4,5\), respectively, the postsynaptic neuron will fire four spikes at \(t=1,2,3,4\), and \(\mathbf{\phi}^{l}(T)=0.8>\mathbf{a}^{l}\). If the presynaptic neurons fires at \(t=3,4,5\) and \(t=1,2\), respectively, the postsynaptic neuron will fire only one spikes at \(t=5\), and \(\mathbf{\phi}^{l}(T)=0.2<\mathbf{a}^{l}\).

Note that the clipping error and quantization error have been proposed in Li et al. (2021). There exist interdependence between the above three kinds of errors. Specifically, the unevenness error will degenerate to the quantization error if \(\mathbf{v}^{l}(T)\) is in the range of \([0,\theta^{l}]\). Assuming that the potential \(\mathbf{v}^{l}(T)\) falls into \([0,\theta^{l}]\) will enable us to estimate the activation function of SNNs ignoring the effect of unevenness error. Therefore, an estimation of the output value \(\mathbf{\phi}^{l}(T)\) in a converted SNN can be formulated with the combination of clip function and floor function, that is:

\[\mathbf{\phi}^{l}(T)\approx\theta^{l}\operatorname{clip}\left(\frac{1}{T}\left[ \frac{\mathbf{z}^{l}T+\mathbf{v}^{l}(0)}{\theta^{l}}\right],0,1\right). \tag{11}\]

The detailed derivation is in the Appendix. With the help of this estimation for the SNN output, the estimated conversion error \(\widehat{\mathbf{Err}}^{l}\) can be derived from Equation 9:

\[\widehat{\mathbf{Err}}^{l}=\theta^{l}\operatorname{clip}\left(\frac{1}{T}\left[ \frac{\mathbf{z}^{l}T+\mathbf{v}^{l}(0)}{\theta^{l}}\right],0,1\right)-h(\mathbf{z}^{l}) \approx\mathbf{Err}^{l}. \tag{12}\]

Figure 1: Conversion error between source ANN and converted SNN. \(s_{1}^{l-1}\) and \(s_{2}^{l-1}\) denote the output spikes of two neurons in layer \(l-1\), and \(s_{1}^{l}\) denotes the output spikes of a neuron in layer \(l\).
## 4 Optimal ANN-SNN conversion

### quantization clip-floor activation function

According to the conversion error of Equation 12, it is natural to think that if the commonly used ReLU activation function \(h(\mathbf{z}^{l})\) is substituted by a clip-floor function with a given quantization steps \(L\) (similar to Equation 11), the conversion error at time-steps \(T=L\) will be eliminated. Thus the performance degradation problem at low latency will be solved. As shown in Equation 13, we proposed the quantization clip-floor activation function to train ANNs.

\[\mathbf{a}^{l}=\tilde{h}(\mathbf{z}^{l})=\lambda^{l}\operatorname{clip}\left(\frac{1} {L}\left[\frac{\mathbf{z}^{l}L}{\lambda^{l}}\right],0,1\right), \tag{13}\]

where the hyperparameter \(L\) denotes quantization steps of ANNs, the trainable \(\lambda^{l}\) decides the maximum value of \(\mathbf{a}^{l}\) in ANNs mapped to the maximum of \(\mathbf{\phi}^{l}(T)\) in SNNs. Note that \(\mathbf{z}^{l}=\mathbf{W}^{l}\mathbf{\phi}^{l-1}(T)=\mathbf{W}^{l}\mathbf{a}^{l-1}\). With this new activation function, we can prove that the estimated conversion error between SNNs and ANNs is zero, and we have the following Theorem.

**Theorem 1**.: _An ANN with activation function (13) is converted to an SNN with the same weights. If \(T=L\), \(\theta^{l}=\lambda^{l}\), and \(\mathbf{v}^{l}(0)=\mathbf{0}\), then:_

\[\widetilde{\mathbf{Err}}^{l}=\mathbf{\phi}^{l}(T)-\mathbf{a}^{l}=\mathbf{0}. \tag{14}\]

Proof.: According to Equation 12, and the conditions \(T=L\), \(\theta^{l}=\lambda^{l}\), \(\mathbf{v}^{l}(0)=\mathbf{0}\), we have \(\widetilde{\mathbf{Err}}^{l}=\mathbf{\phi}^{l}(T)-\mathbf{a}^{l}=\theta^{l}\operatorname{ clip}\left(\frac{1}{T}\left[\frac{\mathbf{z}^{l}T+\mathbf{z}^{l}(0)}{\theta^{l}}\right],0,1 \right)-\lambda^{l}\operatorname{clip}\left(\frac{1}{L}\left[\frac{\mathbf{z}^{ l}L}{\lambda^{l}}\right],0,1\right)=0\). 

Theorem 1 implies that if the time-steps \(T\) of the converted SNN is the same as the quantization steps \(L\) of the source ANN, the conversion error will be zero. An example is illustrated in Figure 1(a), where \(T=L=4\), \(\theta^{l}=\lambda^{l}\). The red curve presents the estimated output \(\mathbf{\phi}^{l}(T)\) of the converted SNNs with respective to different input \(\mathbf{z}^{l}\), while the green curve represents the out \(\mathbf{a}^{l}\) of the source ANN with respective to different input \(\mathbf{z}^{l}\). As the two curve are the same, the estimated conversion error \(\widetilde{\mathbf{Err}}^{l}\) is zero. Nevertheless, in practical application, we focus on the performance of SNNs at different time-steps. There is no guarantee that the conversion error is zero when \(T\) is not equal to \(L\). As illustrated in Figure 1(b), where \(L=4\) and \(L=8\), we can find the conversion error is greater than zero for some \(\mathbf{z}^{l}\). This error will transmit layer-by-layer and eventually degrading the accuracy of the converted SNN. One way to solve this problem is to train multiple source ANNs with different quantization steps, then convert them to SNNs with different time-steps, but it comes at a considerable cost. In the next section, we propose the quantization clip-floor activation function with a shift term to solve this problem. Such an approach can achieve high accuracy for different time-steps, without extra computation cost.

### quantization clip-floor-shift activation function

We propose the quantization clip-floor-shift activation function to train ANNs.

\[\mathbf{a}^{l}=\widehat{h}(\mathbf{z}^{l})=\lambda^{l}\operatorname{clip}\left(\frac {1}{L}\left[\frac{\mathbf{z}^{l}L}{\lambda^{l}}+\mathbf{\varphi}\right],0,1\right). \tag{15}\]

Figure 2: Comparison of SNN output \(\mathbf{\phi}^{l}(T)\) and ANN output \(\mathbf{a}^{l}\) with same input \(\mathbf{z}^{l}\)
Compared with Equation 13, there exists a hyperparameter vector \(\mathbf{\varphi}\) that controls the shift of the activation function. When \(L\neq T\), we cannot guarantee the conversion error is 0. However, we can estimate the expectation of conversion error. Similar to (Deng & Gu, 2020), we assume that \(z_{i}^{l}\) is uniformly distributed within intervals \([(t-1)\lambda^{l}/T,(t)\lambda^{l}/T]\) and \([(l-1)\lambda^{l}/L,(l)\lambda^{l}/L]\) for \(t=1,2,...,T\) and \(L=1,2,...,L\), we have the following Theorem.

**Theorem 2**.: _An ANN with activation function (15) is converted to an SNN with the same weights. If \(\theta^{l}=\lambda^{l}\), \(\mathbf{v}^{l}(0)=\theta^{l}\mathbf{\varphi}\), then for arbitrary \(T\) and \(L\), the expectation of conversion error reaches \(\mathbf{0}\) when the shift term \(\mathbf{\varphi}\) in source ANN is \(\frac{1}{2}\)._

\[\forall\ T,L\quad\mathbb{E}_{z}\left(\widetilde{\mathbf{Err}}^{l}\right)\Big{|}_ {\mathbf{\varphi}=\frac{1}{2}}=\mathbf{0}. \tag{16}\]

The proof is in the Appendix. Theorem 2 indicates that the shift term \(\frac{1}{2}\) is able to optimize the expectation of conversion error. By comparing Figure 1(b) and Figure 1(c), we can find that when the shift term \(\mathbf{\varphi}=\mathbf{0.5}\) is added, the mean conversion error reaches zero, even though \(L\neq T\). These results indicate we can achieve high-performance converted SNN at ultra-low time-steps.

\(L\) is the only undetermined hyperparameter of the quantization clip-floor-shift activation. When \(T=L\), the conversion error reaches zero. So we naturally think that the parameter \(L\) should be set as small as possible to get better performance at low time-steps. However, a too low quantization of the activation function will decrease the model capacity and further lead to accuracy loss when the time-steps is relatively large. Choosing the proper \(L\) is a trade-off between the accuracy at low latency and the best accuracy of SNNs. We will further analyze the effects of quantization steps \(L\) in the experiment section.

### algorithm for training quantization clip-floor-shift activation function

Training an ANN with quantization clip-floor-shift activation instead of ReLU is also a tough problem. To direct train the ANN, we use the straight-through estimator (Bengio et al., 2013) for the derivative of the floor function, that is \(\frac{\mathrm{d}\left|x\right|}{\mathrm{d}x}=1\). The overall derivation rule is given in Equation 17.

\[\frac{\partial\widehat{h}_{i}(\mathbf{z}^{l})}{\partial z_{i}^{l}}=\begin{cases}1,&-\frac{\lambda^{l}}{2L}<z_{i}^{l}<\lambda^{l}-\frac{\lambda^{l}}{2L}\,\ \frac{\partial\widehat{h}_{i}(\mathbf{z}^{l})}{ \partial\lambda^{l}}\ \ =\begin{cases}\frac{\widehat{h}_{i}(\mathbf{z}^{l})-z_{i}^{l}}{ \lambda^{l}},&-\frac{\lambda^{l}}{2L}\leqslant z_{i}^{l}<\lambda^{l}-\frac{ \lambda^{l}}{2L}\\ 0,&z_{i}^{l}<-\frac{\lambda^{l}}{2L}\\ 1,&z_{i}^{l}\geqslant\lambda^{l}-\frac{\lambda^{l}}{2L}\end{cases} \tag{17}\]

Here \(z_{i}^{l}\) is the i-th element of \(\mathbf{z}^{l}\). Then we can train the ANN with quantization clip-floor-shift activation using Stochastic Gradient Descent algorithm (Bottou, 2012).

## 5 Related Work

The study of ANN-SNN conversion is first launched by Cao et al. (2015). Then Diehl et al. (2015) converted a three-layer CNN to an SNN using data-based and model-based normalization. To obtain high-performance SNNs for complex datasets and deeper networks, Rueckauer et al. (2016) and Sengupta et al. (2019) proposed more accurate scaling methods to normalize weights and scale thresholds respectively, which were later proved to be equivalent (Ding et al., 2021). Nevertheless, the converted deep SNN requires hundreds of time steps to get accurate results due to the conversion error analyzed in Sec. 3. To address the potential information loss, Rueckauer et al. (2016) and Han et al. (2020) suggested using "reset-by-subtraction" neurons rather than "reset-to-zero" neurons. Recently, many methods have been proposed to eliminate the conversion error. Rueckauer et al. (2016) recommended 99.9% percentile of activations as scale factors, and Ho & Chang (2020) added the trainable clipping layer. Besides, Han et al. (2020) rescaled the SNN thresholds to avoid the improper activation of spiking neurons. Massa et al. (2020) and Singh et al. (2021) evaluated the performance of converted SNNs on the Loihi Neuromorphic Processor. Our work share similarity with Deng & Gu (2020); Li et al. (2021), which also shed light on the conversion error. Deng & Gu (2020) minimized the layer-wise error by introducing extra bias in addition to the converted SNN biases. Li et al. (2021) further proposed calibration for weights and biases using quantized fine-tuning. They got good results with 16 and 32 time-steps
without trails for more extreme time-steps. In comparison, our work aims to fit ANN into SNN with techniques eliminating the mentioned conversion error. The end-to-end training of quantization layers is implemented to get better overall performance. Our shift correction can lead to a single SNN which performs well at both ultra-low and large time-steps. Maintaining SNN performance within extremely few time-steps is difficult even for supervised learning methods like backpropagation through time (BPTT). BPTT usually requires fewer time-steps because of thorough training, yet at the cost of heavy GPU computation (Wu et al., 2018, 2019; Lee et al., 2019; Neftci et al., 2019; Lee et al., 2020; Zenke & Vogels, 2021). The timing-based backpropagation methods (Bohte et al., 2002; Tavanaei et al., 2019; Kim et al., 2020) could train SNNs over a very short temporal window, e.g. over 5-10 time-steps. However, they are usually limited to simple datasets like MNIST (Kheradpisheh & Masquelier, 2020) and CIFAR10 (Zhang & Li, 2020). Rathi et al. (2019) shortened simulation steps by initializing SNN with conversion method and then tuning SNN with STDP. In this paper, the proposed method achieves high-performance SNNs with ultra-low latency (4 time-steps).

## 6 Experiments

In this section, we validate the effectiveness of our method and compare our method with other state-of-the-art approaches for image classification tasks on CIFAR-10 (LeCun et al., 1998), CIFAR-100 (Krizhevsky et al., 2009), and ImageNet datasets (Deng et al., 2009). Similar to previous works, we utilize VGG-16 (Simonyan & Zisserman, 2014), ResNet-18 (He et al., 2016), and ResNet-20 network structures for source ANNs. We compare our method with the state-of-the-art ANN-SNN conversion methods, including Hybrid-Conversion (HC) from Rathi et al. (2019), RMP from Han et al. (2020), TSC from Han & Roy (2020), RNL from Ding et al. (2021), ReLUThreshold-Shift (RTS) from Deng & Gu (2020), and SNN Conversion with Advanced Pipeline (SNNC-AP) from Li et al. (2021). Comparison with different SNN training methods is also included to manifest the superiority of low latency inference, including HybridConversion-STDB (HC-STDB) from Rathi et al. (2019), STBP from Wu et al. (2018), DirectTraining (DT) from Wu et al. (2019), and TSSL from Zhang & Li (2020). The details of the proposed ANN-SNN algorithm and training configurations are provided in the Appendix.

### Test accuracy of ANN with quantization clip-floor-shift activation

We first compare the performance of ANNs with quantization clip-floor activation (green curve), ANNs with quantization clip-floor-shift activation (blue curve), and original ANNs with ReLU activation (black dotted line). Figure 3(a)-(d) report the results about VGG-16 on CIFAR-10, ResNet-20 on CIFAR-10, VGG-16 on CIFAR-100 and ResNet-20 on CIFAR-100. The performance of ANNs with quantization clip-floor-shift activation is better than ANNs with quantization clip-floor activation. These two ANNs can achieve the same performance as original ANNs with ReLU activation when \(L>4\). These results demonstrate that our quantization clip-floor-shift activation function hardly affects the performance of ANN.

### Comparison with the state-of-the-art

Table 2 compares our method with the state-of-the-art ANN-SNN conversion methods on CIFAR-10. As for low latency inference (T \(\leq 64\)), our model outperforms all the other methods with the same time-step setting. For T \(=32\), the accuracy of our method is slightly better than that of ANN (95.54% vs. 95.52%), whereas RMP, RTS, RNL, and SNNC-AP methods have accuracy loss of 33.3%, 19.48%, 7.42%, and 2.01%. Moreover, we achieve an accuracy of 93.96% using only 4 time-steps, which is 8 times faster than SNNC-AP that takes 32 time-steps. For ResNet-20, we achieve an accuracy of 83.75% with 4 time-steps. Notably, our ultra-low latency performance is

Figure 3: Compare ANNs accuracy.
comparable with other state-of-the-art supervised training methods, which is shown in Table S3 of the Appendix.

We further test the performance of our method on the large-scale dataset. Table 3 reports the results on ImageNet, our method also outperforms the others both in terms of high accuracy and ultra-low latency. For ResNet-34, the accuracy of the proposed method is 4.83% higher than SNNC-AP and 69.28% higher than RTS when \(T=32\). When the time-steps is 16, we can still achieve an accuracy of 59.35%. For VGG-16, the accuracy of the proposed method is 4.83% higher than SNNC-AP and 68.356% higher than RTS when \(T=32\). When the time-steps is 16, we can still achieve an accuracy of 50.97%. These results demonstrate that our method outperforms the previous conversion methods. More experimental results on CIFAR-100 is in Table S4 of the Appendix.

### Comparison of quantization clip-floor and quantization clip-floor-shift

Here we further compare the performance of SNNs converted from ANNs with quantization clip-floor activation and ANN with quantization clip-floor-shift activation. In Sec. 4, we prove that the expectation of the conversion error reaches 0 with quantization clip-floor-shift activation, no matter whether \(T\) and \(L\) are the same or not. To verify these, we set \(L\) to 4 and train ANNs with quantization clip-floor activation and quantization clip-floor-shift activation, respectively. Figure 4 shows how the accuracy of converted SNNs changes with respect to the time-steps \(T\). The accuracy of the converted SNN (green curve) from ANN with quantization clip-floor activation (green dotted line) first increases and then decreases rapidly with the increase of time-steps, because we cannot guarantee that the conversion error is zero when \(T\) is not equal to \(L\). The best performance is still lower than source ANN (green dotted line). In contrast, the accuracy of the converted SNN from ANN with quantization clip-floor-shift activation (blue curve) increases with the increase of \(T\). It gets the same accuracy as source ANN (blue dotted line) when the time-steps is larger than 16.

### Effect of quantization steps L

In our method, the quantization steps \(L\) is a hyperparameter, which affects the accuracy of the converted SNN. To analyze the effect of \(L\) and better determine the optimal value, we train VGG-16/ResNet-20 networks with quantization clip-floor-shift activation using different quantization steps L, including 2,4,8,16 and 32, and then converted them to SNNs. The experimental results

\begin{table}
\begin{tabular}{l l l l l l l l l l} \hline \hline Architecture & Method & ANN & T=2 & T=4 & T=8 & T=16 & T=32 & T=64 & T\(\geq\)512 \\ \cline{3-10}  & RMP & 93.63\% & - & - & - & - & 60.30\% & 90.35\% & 93.63\% \\ \cline{2-10}  & TSC & 93.63\% & - & - & - & - & - & 92.79\% & 93.63\% \\ \cline{2-10}  & RTS & 95.72\% & - & - & - & - & 76.24\% & 90.64\% & 95.73\% \\ \cline{2-10}  & RNL & 92.82\% & - & - & - & 57.90\% & 85.40\% & 91.15\% & 92.95\% \\ \cline{2-10}  & SNNC-AP & 95.72\% & - & - & - & - & 93.71\% & 95.14\% & 95.79\% \\ \cline{2-10}  & **Ours** & 95.52\% & 91.18\% & 93.96\% & 94.95\% & 95.40\% & 95.54\% & 95.55\% & 95.59\% \\ \hline \multirow{3}{*}{ResNet-20} & RMP & 91.47\% & - & - & - & - & - & - & 91.36\% \\ \cline{2-10}  & TSC & 91.47\% & - & - & - & - & - & 69.38\% & 91.42\% \\ \cline{2-10}  & **Ours** & 91.77\% & 73.20\% & 83.75\% & 89.55\% & 91.62\% & 92.24\% & 92.35\% & 92.41\% \\ \hline \multirow{3}{*}{ResNet-18} & RTS1 & 95.46\% & - & - & - & - & 84.06\% & 92.48\% & 94.42\% \\ \cline{2-10}  & SNNC-AP1 & 95.46\% & - & - & - & - & 94.78\% & 95.30\% & 95.45\% \\ \cline{1-1} \cline{2-10}  & **Ours** & 96.04\% & 75.44\% & 90.43\% & 94.82\% & 95.92\% & 96.08\% & 96.06\% & 96.06\% \\ \hline \hline \end{tabular}

* RTS and SNNC-AP use altered ResNet-18, while ours use standard ResNet-18.

\end{table}
Table 2: Comparison between the proposed method and previous works on CIFAR-10 dataset.

Figure 4: Compare quantization clip-floor activation with/without shift term
on CIFAR-10/100 dataset are shown in Table S2 and Figure 5, where the black dotted line denotes the ANN accuracy and the colored curves represent the accuracy of the converted SNN. In order to balance the trade-off between low latency and high accuracy, we evaluate the performance of converted SNN mainly in two aspects. First, we focus on the SNN accuracy at ultra-low latency (within 4 time-steps). Second, we consider the best accuracy of SNN. It is obvious to find that the SNN accuracy at ultra-low latency decreases as \(L\) increases. However, a too small \(L\) will decrease the model capacity and further lead to accuracy loss. When \(L=2\), there exists a clear gap between the best accuracy of SNN and source ANN. The best accuracy of SNN approaches source ANN when \(L>4\). In conclusion, the setting of parameter \(L\) mainly depends on the aims for low latency or best accuracy. The recommend quantization step \(L\) is 4 or 8, which leads to high-performance converted SNN at both small time-steps and very large time-steps.

## 7 Discussion and conclusion

In this paper, we present ANN-SNN conversion method, enabling high-accuracy and ultra-low-latency deep SNNs. We propose the quantization clip-floor-shift activation to replace ReLU activation, which hardly affects the performance of ANNs and is closer to SNNs activation. Furthermore, we prove that the expected conversion error is zero, no matter whether the time-steps of SNNs and the quantization steps of ANNs is the same or not. We achieve state-of-the-art accuracy with fewer time-steps on CIFAR-10, CIFAR-100, and ImageNet datasets. Our results can benefit the implementations on neuromorphic hardware and pave the way for the large-scale application of SNNs.

Different from the work of Deng and Gu (2020), which adds the bias of the converted SNNs to shift the theoretical ANN-SNN curve to minimize the quantization error, we add the shift term in the quantization clip-floor activation function, and use this quantization clip-floor-shift function to train the source ANN. We show that the shift term can overcome the performance degradation problem when the time-steps and the quantization steps are not matched. Due to the unevenness error, there still exists a gap between ANN accuracy and SNN accuracy, even when \(L=T\). Moreover, it is hard to achieve high-performance ANN-SNN conversion when the time-steps \(T=1\). All these problems deserve further research. One advantage of conversion-based methods is that they can reduce the overall computing cost while maintaining comparable performance as source ANN. Combining the conversion-based methods and model compression may help significantly reduce the neuron activity

Figure 5: Influence of different quantization steps

\begin{table}
\begin{tabular}{l l l l l l l l l} \hline \hline Architecture & Method & ANN & T=16 & T=32 & T=64 & T=128 & T=256 & T\(\geq\)1024 \\ \hline \multirow{4}{*}{ResNet-34} & RMP & 70.64\% & - & - & - & - & - & 65.47\% \\ \cline{2-9}  & TSC & 70.64\% & - & - & - & - & 61.48\% & 65.10\% \\ \cline{2-9}  & RTS & 75.66\% & - & 0.09\% & 0.12\% & 3.19\% & 47.11\% & 75.08\% \\ \cline{2-9}  & SNNC-AP & 75.66\% & - & 64.54\% & 71.12\% & 73.45\% & 74.61\% & 75.45\% \\ \cline{2-9}  & **Ours** & 74.32\% & 59.35\% & 69.37\% & 72.35\% & 73.15\% & 73.37\% & 73.39\% \\ \hline \multirow{4}{*}{VGG-16} & RMP & 73.49\% & - & - & - & - & 48.32\% & 73.09\% \\ \cline{2-9}  & TSC & 73.49\% & - & - & - & - & 69.71\% & 73.46\% \\ \cline{1-1} \cline{2-9}  & RTS & 75.36\% & - & 0.114\% & 0.118\% & 0.122\% & 1.81\% & 73.88\% \\ \cline{1-1} \cline{2-9}  & SNNC-AP & 75.36\% & - & 63.64\% & 70.69\% & 73.32\% & 74.23\% & 75.32\% \\ \cline{1-1} \cline{2-9}  & **Ours** & 74.29\% & 50.97\% & 68.47\% & 72.85\% & 73.97\% & 74.22\% & 74.32\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison between the proposed method and previous works on ImageNet dataset.
and thus reduce energy consumptions without suffering from accuracy loss (Kundu et al., 2021; Rathi and Roy, 2021), which is a promising direction.
## Acknowledgement

This work was supported by the National Natural Science Foundation of China under contracts No.62176003 and No.62088102.
