Spiking Neural Network for Ultra-low-latency and High-accurate Object
  Detection
Spiking Neural Networks (SNNs) have garnered widespread interest for their
energy efficiency and brain-inspired event-driven properties. While recent
methods like Spiking-YOLO have expanded the SNNs to more challenging object
detection tasks, they often suffer from high latency and low detection
accuracy, making them difficult to deploy on latency sensitive mobile
platforms. Furthermore, the conversion method from Artificial Neural Networks
(ANNs) to SNNs is hard to maintain the complete structure of the ANNs,
resulting in poor feature representation and high conversion errors. To address
these challenges, we propose two methods: timesteps compression and
spike-time-dependent integrated (STDI) coding. The former reduces the timesteps
required in ANN-SNN conversion by compressing information, while the latter
sets a time-varying threshold to expand the information holding capacity. We
also present a SNN-based ultra-low latency and high accurate object detection
model (SUHD) that achieves state-of-the-art performance on nontrivial datasets
like PASCAL VOC and MS COCO, with about remarkable 750x fewer timesteps and 30%
mean average precision (mAP) improvement, compared to the Spiking-YOLO on MS
COCO datasets. To the best of our knowledge, SUHD is the deepest spike-based
object detection model to date that achieves ultra low timesteps to complete
the lossless conversion.
Jinye Qu, Zeyu Gao, Tielin Zhang, Yanfeng Lu, Huajin Tang, Hong Qiao
# Spiking Neural Network for Ultra-low-latency and High-accurate Object Detection

###### Abstract

Spiking Neural Networks (SNNs) have garnered widespread interest for their energy efficiency and brain-inspired event-driven properties. While recent methods like Spiking-YOLO have expanded the SNNs to more challenging object detection tasks, they often suffer from high latency and low detection accuracy, making them difficult to deploy on latency sensitive mobile platforms. Furthermore, the conversion method from Artificial Neural Networks (ANNs) to SNNs is hard to maintain the complete structure of the ANNs, resulting in poor feature representation and high conversion errors. To address these challenges, we propose two methods: timesteps compression and spike-time-dependent integrated (STDI) coding. The former reduces the timesteps required in ANN-SNN conversion by compressing information, while the latter sets a time-varying threshold to expand the information holding capacity. We also present a SNN-based ultra-low latency and high accurate object detection model (SUHD) that achieves state-of-the-art performance on nontrivial datasets like PASCAL VOC and MS COCO, with about remarkable 750x fewer timesteps and 30% mean average precision (mAP) improvement, compared to the Spiking-YOLO on MS COCO datasets. To the best of our knowledge, SUHD is the deepest spike-based object detection model to date that achieves ultra low timesteps to complete the lossless conversion.

 Spiking neural network, Object detection, Low latency, Timesteps compression

## I Introduction

With the development of high-performance computing devices, artificial neural networks (ANNs) have made achievements in many artificial intelligence tasks such as image classification [1], object detection [2], and sequential decision-making [3] in recent years. However, ANNs has a huge energy consumption, which makes it difficult to deploy on mobile devices. Spiking Neural Networks (SNNs) is the third-generation artificial neural network [4]. Inspired by biological neurons [5], it changes the complex multiplication operation in the ANNs into a simple accumulation operation and transmits information through spikes sequences [6][7]. Due to the sparsity of spike events and the characteristics of event-driven computing, SNNs have remarkable energy efficiency and are the neural network of choice for neuromorphic chips [8, 9, 10, 11]. It is generally believed that SNNs has greater development potential and bionic value.

Because of the non-differentiation of SNNs, the gradients cannot be computed directly during backward propagation, which makes it difficult to obtain SNNs by training directly. There are currently two main methods of obtaining SNNs, one is obtained by conversion [12, 13, 14], i.e. processing the trained ANNs weights to obtain available SNNs, and the other one is learning SNNs weights from scratch [15][16] by spike-time-dependent plasticity (STDP) [17, 18, 19] or spike-time-dependent backpropagation (STDB) [20, 21]. On that basis, a variety of correlation optimization algorithms are derived [22, 23, 24]. The approach to learning from scratch requires a significant amount of time and computer resources relative to the conversion approach. The conversion approach takes full advantage of the ease of training of ANNs and can promptly obtain usable weights from the trained ANNs [25]. Both methods are widely used in shallow SNNs with good results. Q. Yu et al. used a double threshold scheme and an augmented spike scheme to achieve lossless conversion in MNIST, FashionMNIST and CIFAR10 [14]. C. Hong et al. proposed a modified SpikeProp learning algorithm, which ensures better learning stability for SNNs [26]. N. Rathi et al. proposed using the first convolutional layer as the coding layer and using a gradient descent-based training method to make the SNN accuracy close to that of an ANN with the same structure [27]. At the same time, many works have been devoted to the use of SNNs on deeper networks. J. Ding et al. achieved an ANN to SNN conversion with a loss of 0.8% using a PreActResNet-34 network on the CIFAR-100 dataset [28]. Y. Li et al. achieved a 0.23% accuracy loss ANN to SNN conversion on the MS COCO dataset using resnet50 [29]. Y. Hu et al. attempted the conversion on a deep resnet network and achieved an accuracy loss of about 1.16% under 50 layers on ImageNet dataset [30]. These works pushed the development of SNNs to deeper networks.

In the past, SNNs were mainly applied to simple tasks such as image classification [13, 14, 31, 32]. In recent years, some works have used tried to promote SNNs to more challenging tasks, such as multi-sensory integration learning [33], object detection [12], reinforcement learning [34]. One of the most concerned directions is the SNN based object detection. Spiking-YOLO [12] is the first SNN based
object detection model that pushes the boundaries of the field by achieving a near lossless ANN to SNN conversion at timesteps = 8000 based on a YOLOv3-tiny backbone network. Nevertheless, the practicality and deployment of this network is hampered by its excessive timesteps requirement and the depth of 23 layers limits its detection performance. FSHNN [35] combines the STDP, STBP, and Monte Carlo Dropout methods and makes FSHNN exceed the accuracy of its heterogeneous ANN based RetinaNet. It has reduced timesteps from thousands to 300, significantly improving energy efficiency. However, 300 timesteps is still not sufficient for the model to run on mobile robot platforms and the model's object detection accuracy on the COCO dataset still does not match the performance of current mainstream ANN models, such as YOLOv5.

In summary, most of the previous works [12, 36, 37, 38, 39, 24, 36] require tremendous timesteps to reach lossless conversion, which makes it difficult to deploy SNNs on the latency-sensitive mobile terminal devices. The excessively slow processing speed is also hardly acceptable for real-time object detection. In addition, the current ANN-SNN conversion methods do not apply to all ANN structures, which may cause some structural damage of the ANN partly during the conversions, reducing the accuracy of the ANN, which equates to a reduction in the accuracy of the converted SNN. So to convert the complete structure of a deep neural network into a SNN with both ultra-low timesteps and high accuracy is a significant issue.

To overcome the challenges mentioned above, we introduce two novel methods to reduce time latency while maintaining comparable accuracy and low energy cost: timesteps compression and spike-time-dependent integrated coding. Further, we present an low-latency and high-accurate SNN based object detection model called SUHD. Our contributions can be summarized as follows:

* A timesteps compression method is proposed, which compresses multi-timesteps into one timestep, reducing timesteps requirements for ANN to SNN conversion and inference, providing the possibility of SNNs deployment for engineering applications. Compared to Spiking-YOLO, we are able to reduce the timesteps requirement by more than 750 times with comparable accuracy.
* We propose a spike-time-dependent integrated coding (STDI) method and implement it using a time-varying threshold neuron model. This approach further reduces the inference time of SNN by approximately 38%, which is mainly caused by the increased information capacity of individual spikes.
* Fast (SPPF) structures, achieving lossless conversion of the Maxpool layer with any stride and realizing a lossless conversion of SPPF.
* Based on the methods mentioned above, we propose a ultra-low-latency and high-accurate SNN based object detection model, called SUHD. SUHD has demonstrated excellent performance on two challenging datasets (PASCAL VOC and MS COCO), achieving state-of-the-art results with 4 timesteps.

## II Related Works

### _Neuronal Coding_

Frequency coding is a widely used coding method that relies on spike firing rates within a certain timesteps to convey information. However, due to the binary nature of the spikes, the spike firing ratio does not transmit information very efficiently. As a result, when encountering complex information, frequency coding must ensure accurate transmission of the information at the cost of large timesteps.

Temporal coding is an advanced coding method that embeds time information into the spike train. The combination of time information and frequency information allows the spike train to carry more information. It includes time-to-first-spike [39, 40], rank-order coding [41], and phase coding [42]. It has achieved remarkable results in deep SNNs.

### _Conversion Methods_

The ANN to SNN conversion has been one of the hottest issues in the last few years. Many methods have been proposed and some significant developments have been achieved based on the conversion methods. The subtraction reset [36] discards the fixed reset potential and retains the spike intensity information. The temporal-separation (TS) [31] are proposed in 2022, which eliminates errors caused by incorrect firing order of negative spikes. TS separated the accumulation and firing phases, achieving lossless conversion in simple models. T. Bu et al. proposed the theory of membrane potential initialization [13] and pointed out that the membrane potential can be initialized to half of the threshold value, which can achieve lossless conversion in sufficiently large timesteps. The conversion of deep ResNet [30] was proposed by Y. Hu et al. to solve the conversion problem of bottleneck structures.

### _Spiking-YOLO_

Spiking-YOLO [12] is the first attempt to convert ANN to SNN to achieve object detection, which could get relatively high accuracy based on very large timesteps. It uses channel-wise normalization and signed neurons featuring imbalanced threshold to reduce the conversion error and is based on YOLOv3-tiny for conversion. At timesteps = 3000 on MS COCO dataset, it achieves close to lossless performance with an mAP of 25%. However, we found during implementation that the large timesteps required significant computational resources, making it impossible to use on common PC or mobile robotics platform. Additionally, the actual computing speed is extremely slow.

## III Methods

In this section, we present our systematic approach to converting ANNs to SNNs with competitive accuracy and reduced timesteps. In subsection A, we conduct a detailed analysis of the mathematical processes involved in ANN to SNN conversion, identifying the main sources of errors. In subsections B and C, we introduce our novel methods of timesteps compression and spike-time-dependent integrated coding, which help eliminate these errors. Finally, in subsection
C, we implement the conversion of the SPPF structure and apply our approaches to construct a state-of-the-art SUHD object detection model.

### _ANN to SNN Conversion Algorithm and Error Analysis_

First of all, we implemented an ANN neuron model base for a single neuron as follows:

\[x^{l}=\sum\limits_{i=1}^{n}w_{i}^{l-1}x_{i}^{l-1}+b_{i}^{l}, \tag{1}\]

where \(x^{l}\) denotes the output value of the neuron in layer \(l\), \(w_{i}^{l-1}\) denotes \(i\)-th weight of layer \(l-1\) to layer \(l\), \(x_{i}^{l-1}\) denotes the output value of \(i\)-th neuron in layer \(l-1\), and \(b_{i}^{l}\) denotes the bias of the output neuron.

The LIF and IF models are two basic biological neuronal models. The former has a continuous leaky current in the absence of stimulation, allowing the membrane potential to gradually return to resting potential, which may lead to information loss. In order to accurately represent the recurrent relationships between neurons, we adopted the IF neuron model as the basic neuron model.

Meanwhile, we use the spiking ratio for the SNN neuron model to substitute the simulated value \(x\) in Eq. 1. To establish the corresponding equivalence, we begin with the input potential of the SNN neuron and analyze its dynamics.

\[z^{l}(t)=\sum\limits_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t)+b_{i}^{l}. \tag{2}\]

where \(z^{l}(t)\) represents input of that neuron in layer \(l\) at time \(t\), \(s_{i}^{l-1}(t)\) represents the spike of \(i\)-th neuron of layer \(l-1\), it can be formulated as follows:

\[s=\begin{cases}1,&V_{mem}\geq V_{thr},\\ 0,&V_{mem}<V_{thr},\end{cases} \tag{3}\]

the threshold \(V_{thr}\) was set to 1. Then we can denote the sum of the input voltage of that neuron as follows:

\[U_{in}^{l}=\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t) +\sum\limits_{t=1}^{T}b_{i}^{l}, \tag{4}\]

then we denote the spiking ratio as \(r^{l}\) and denote output potential of that neuron in layer \(l\) as \(U_{out}^{l}\), Thus its numerical relationship can be represent as:

\[r^{l}=\frac{\sum\limits_{t=1}^{T}s^{l}(t)}{T}=\frac{U_{out}^{l}}{T}, \tag{5}\]

to limit the spiking ratio to a reasonable [0, 1], we performed channel-wise weights and bias normalization operations [12] to scale the activation values:

\[w^{l}_{\ i}=\frac{w_{i}^{l}\times\max_{in}}{\max_{out}},b^{l}_{\ i}=\frac{b_{i} ^{l}\times\max_{in}}{\max_{out}}, \tag{6}\]

where \(\text{max}_{in}\) and \(\text{max}_{out}\) denote the maximum input and output activation values of the current channel. Ideally, the voltage input to the neuron and the voltage output from the neuron should be the same, i.e. \(U_{in}=U_{out}\)[31]. Thus combining the Eq. 4 and Eq. 5, we will get:

\[r^{l}=\sum\limits_{i=1}^{n}w^{\prime l-1}_{\ i}r_{i}^{l-1}+b^{l}_{\ i}, \tag{7}\]

it can be seen from Eq. 7 that in an ideal state, the SNN replaces the analog value in the ANN with spiking ratio for lossless conversion and information transfer. For convenience, we use \(w\) and \(b\) to denote the normalized weights and biases, as shown in Eq. 6. Thus, we can obtain the relationship between the membrane potential at time \(T\) and \(0\):

\[\begin{split} V_{mem,i}^{l}(T)=V_{mem,i}^{l}(0)+\sum\limits_{t= 1}^{T}\sum\limits_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t)+\\ \sum\limits_{t=1}^{T}b_{i}^{l}-\sum\limits_{t=1}^{T}s_{i}^{l}(t), \end{split} \tag{8}\]

where \(V_{mem,i}^{l}(t)\) represent the membrane potential of the \(i\)-th neuron of layer \(l\) in time \(t\).

#### Ii-A1 Tremendous timesteps demand

Combining Eq. 5 and Eq. 8, Then we can get:

\[r_{i}^{l}=\frac{(V_{mem,i}^{l}(0)-V_{mem,i}^{l}(T))}{T}+\sum\limits_{i=1}^{n}w _{i}r_{i}^{l-1}+b_{i}^{l}, \tag{9}\]

ideally, in order to satisfy \(U_{in}=U_{out}\) and the equivalence of Eq. 7, \(V_{mem,i}^{l}(0)-V_{mem,i}^{l}(T)\) should be 0. But, from Eq. 8, we can see that \(\sum\limits_{t=1}^{T}s_{i}^{l}(t)\) is a step function. For \(V_{mem,i}^{l}(0)-V_{mem,i}^{l}(T)=0\) to be true, the \(\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t)+\sum\limits_ {t=1}^{T}b_{i}^{l}\) has to be the same step function. It certainly doesn't always satisfy this condition. Therefore:

\[V_{mem,i}^{l}(0)=V_{mem,i}^{l}(T)-\varepsilon,(\varepsilon<V_{thr}), \tag{10}\]

the \(\varepsilon\) is the residual membrane potential and the \(\frac{\varepsilon}{T}\) is called quantization error, as shown in Fig. 1. Referring to the Eq. 9 and Fig. 1(b), previous works have mainly focused on scaling down the quantization error by increasing T [12, 35, 43], as shown in Fig. 1(c). Since most of the current SNN-based

Fig. 1: Quantization errors. Assuming timesteps (\(T\)) is 5. In (a), the membrane potential does not reach the spiking threshold at the final \(t=T\) moment, leading to the potential remaining in the cell membrane without being transmitted. The equivalence to (b), only 6 values can be represented in this process: 0, 2, 0, 0, 0.6, 0.8, 1.0. Therefore, many values cannot be expressed precisely. In (c), we improve the density of expressed values by increasing \(T\), resulting in a more accurate expression of the spiking ratio.
works focus on image classification tasks, this approach is feasible. However, the object detection task requires more accurate position regression, which requires the spike sequence to have a very accurate numerical representation. Besides, most of such tasks are based on deep SNNs, which exacerbates quantization errors. Thus, we should often greatly improve the timesteps \(T\) to cope with the high accuracy of the numerical expression, which takes huge timesteps for conversion (256-8000) [12, 35, 43]. However, the timesteps are not infinitely long, because it brings a huge amount of energy consumption and a huge running time demand. This defeats the original purpose of real-time object detection. We believe that delivering more information in limited timesteps is an important direction for SNNs development.

#### Iii-A2 Truncation error

In Eq. 6, to reduce the spiking ratio to a reasonable range, we scale the weight based on the maximum activation value [12]. The maximum activation value is taken from the samples which is often based on a part of the conversion dataset. In most cases, this maximum activation value is applicable. However, in actual detection cases, the activation value may still be greater than the maximum value in the sample, as shown in Fig. 2. Therefore, for real applications in object detection tasks, \(r^{l}\) may be greater than the spiking ratio upper limit. At this time, the part higher than 1 is cut out by default. This error reduces SNN's accuracy.

### _Timesteps Compression_

The large timesteps can reduce the quantization error, unevenness error, and thus the conversion loss. However, larger timesteps also bring an increase in time latency. We propose the timesteps compression method to alleviate this problem. Timesteps compression means compressing information from multiple timesteps into one timestep and delivering the information using burst or binary spikes. Burst spikes [44, 45, 46] are a set of short inter-spike interval (ISI) spikes that can issue multiple spikes at one timestep.

As shown in Fig. 3, we represent the fully equivalent uncompressed and compressed processes. timesteps compression consists of i) input compression, ii) layer compression, and iii) output decompression. We assume that the timesteps have a compression scale of \(f_{c}\). Input compression compresses information from \(f_{c}\) timesteps into one timestep and reduces the timesteps to \(\frac{1}{f_{c}}\). This operation makes the input information the same as that with the uncompressed state. In layer compression, the compressed and uncompressed inputs within a single timestep are related as follows:

\[z_{c}=\sum_{t=1}^{f_{c}}z(t)=\sum_{i=1}^{n}\sum_{t=1}^{f_{c}}w_{i}^{l-1}s_{i}^{ l-1}(t)+f_{c}b, \tag{11}\]

where \(z_{c}\) represents the compressed input, \(s_{i}^{l-1}\) denotes the output spikes of \(i\)-th neuron in previous layer. Multiple inputs lead to changes in the spike firing as follows:

\[s_{c}=\begin{cases}min(k,f_{c}),&V_{mem}\geq k*V_{thr},\\ 0,&V_{mem}<V_{thr},\end{cases} \tag{12}\]

\(s_{c}\) represents the spike issued, and when the \(k>1\), the \(s_{c}\) is burst spike. Layer compression allows multiple spikes to be issued in one timestep, thereby increasing the density of information within a timestep. Ideal mathematical relationships can be presented as follows:

\[U_{in}^{l}=\sum_{i=1}^{n}\sum_{t=1}^{T_{c}}w_{i}^{l-1}s_{c,i}^{l-1}(t)+T_{c}f_ {c}b_{i}^{l}, \tag{13}\]

\[r_{c}^{l}=\frac{U_{in}^{l}}{T_{c}}=\sum_{i=1}^{n}w_{i}^{l-1}\frac{\sum\limits {{}_{l=1}^{T_{c}}s_{c,i}^{l-1}(t)}}{T_{c}}+f_{c}b_{i}^{l}, \tag{14}\]

input compression and layer compression are equivalent to compressing multiple timesteps into one timestep, which is the core of the timesteps compression. This relationship is illustrated in the layer compression stage and layer transfer stage in Fig. 3. By input compression and layer compression, the firing ratio is mapped from \(r\in[0,1]\) to \(r_{c}\in[0,f_{c}]\), and the oversized firing ratio does not match the firing ratio at \(f_{c}\) times uncompressed timesteps. Output decompression solves this problem by remapping the firing ratio from \(r_{c}\in[0,f_{c}]\) to \(r\in[0,1]\) as shown in Eq. 15.

\[r^{l}=\frac{r_{c}^{l}}{f_{c}}=\sum_{i=1}^{n}w_{i}^{l-1}\frac{\sum \limits_{t=1}^{T_{c}}s_{c,i}^{l-1}(t)}{T}+b_{i}^{l} \tag{15}\] \[=\sum_{i=1}^{n}w_{i}^{l-1}\frac{\sum\limits_{t=1}^{T}s_{i}^{l-1}(t )}{T}+b_{i}^{l},\]

as shown above, timesteps compression improved the information-carrying capacity of a single timestep and reduce the quantization error. The experiments in Sec. IV-C confirm this inference.

Fig. 2: Truncation errors. The image on the left illustrates the generation of truncation errors, where messages exceeding the upper limit of the firing ratio are truncated by default. In the image on the right, we randomly selected 100 images from the COCO dataset, measured their maximum activation value at each layer in the model, and represented that activation value on the horizontal axis. At the same time, we measured the maximum activation value at each layer using the ANN to SNN conversion sample dataset, and represent this activation value on the vertical axis. The blue slash indicates the ideal situation when the actual maximum activation is equal to the maximum activation of the sample data. The red dots are the relative positions of the activation values for the layer under different samples.
### _Spike-Time-Dependent Integrated Coding_

Recent object detection researches have increasingly emphasized real-time and energy efficiency. Although in the previous section, we used timesteps compression to reduce the requirement for timesteps in the model inference process, the use of frequency coding makes the model still inefficient. At the same time, it has been reported that neurons coordinate action potentials in different ways even when the spike firing rates are the same. These reports suggest that intercellular communication is the result of a combination of various coding methods.

In particular, we use TS [31] in our work to avoid the damage to accuracy caused by the negative spikes of misordered firing. The use of TS concentrates the spikes at the beginning of the timesteps, which gives us the opportunity to embed temporal information in the spike train. Therefore, we propose an encoding method called the spike-time-dependent integrated (STDI) coding method, which further improves the inference speed and energy efficiency of the model.

#### Iii-C1 Weighted Spikes

We first attach weights to the spikes based on their firing time. The weights are defined as follows:

\[\tau(t)=T-t+1, \tag{16}\]

where \(t\) denotes the time of spike firing over whole timesteps. After the weights are defined, the input value represented by a firing spike becomes \(s(t)*\tau\). Fig. 4(a) shows the correspondence between the weighted spikes and the input values. Fig. 4(a) also expresses that STDI can make the spike firing ratio within \(t\in[0,T]\) much greater than 1, which resolves the truncation error. When faced with excessive truncation errors, STDI can eliminate such errors by integrating multiple spikes (burst spikes) in one timestep, as shown in Fig. 4(a) when the input value is 12. Weighted spikes bring about a change in the way input information is encoded. In previous SNN object detection works, we needed to keep constant value information input throughout the timesteps and then encode it using the first layer of the SNN model. With spikes being weighted, the information only needs to be fed into the model at the first timestep. This reduces energy consumption to a certain extent.

#### Iii-C2 Model Adaption for STDI

Due to the weighted spikes, we need to make appropriate adjustments to the IF model. Firstly, the threshold \(V_{thr}\) was adjusted for weighted spikes as Eq. 17.

\[V_{thr}=\tau(t)*v_{thr}, \tag{17}\]

the \(v_{thr}\) denotes the initial threshold, often set to 1. Then we adjust the input as follows:

\[z^{l}=\sum_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t)*\tau(t)+b_{i}^{l}, \tag{18}\]

therefore Eq. 4 and Eq. 5 must be changed to:

\[U_{in}^{l}=\sum_{t=1}^{T}\sum_{i=1}^{n}w_{i}^{l-1}s_{i}^{l-1}(t)*\tau(t)+\sum _{t=1}^{T}b_{i}^{l}, \tag{19}\]

\[r^{l}=\sum_{i=1}^{n}w_{i}^{l-1}\frac{\sum\limits_{t=1}^{T}s_{i}^{l-1}(t)*\tau (t)}{T}+b_{i}^{l}, \tag{20}\]

the TS scheme was employed in the present study, leading to the observation of two distinct phases of STDI which are the accumulation phase and the firing phase. In the accumulation phase, the input information is accumulated to the membrane

Fig. 3: Process comparison between proposed timesteps compression and traditional methods. We first denote the compression scale as \(f_{c}\), which indicates that each compressed timestep contains information from \(f_{e}\) uncompressed timesteps. For the sake of an example, the \(f_{e}\) is set to 3. And then we denote compressed timesteps and firing ratio as \(T_{c}\) and \(r_{c}\).
potential according to the Eq. 19. During the firing phase, the threshold decrease according to the time. the accumulated membrane potential searches for the right firing time to release the spikes. Specifically, when the membrane potential is less than the spike threshold at the current moment, the spike is not fired and waits for the next moment. When the membrane potential is greater than the spike threshold at that moment, the spike fires and the membrane potential decreases according to the subtractive reset [36]. Cycle this process until the membrane potential is lower than \(V_{thr}\). The specific algorithm is shown in Algorithm 1. In the output layer, we need to decode the spike train to spiking ratio according to Eq. 20. Fig. 4(b) illustrates the whole process.

```
0:\(T\), \(x\)
0:\(s\)
0: Initialize \(V_{mem}\) to 0
0: Initialize \(s\) to 0
0: phase 1: fort = 1 to \(T\)do \(\tau(t)\gets T-t+1\) \(V_{mem}\gets V_{mem}\) + \(x\)[t-1]*\(\tau(t)\) endfor  phase 2: if\(V_{mem}\leq 0\)then \(V_{mem}=0\) endif fort = 1 to \(T\)do \(\tau(t)\gets T-t+1\) \(s[t-1]\gets s[t-1]+\lfloor V_{mem}/\tau(t)\rfloor\) \(V_{mem}\gets V_{mem}\) - \(s[t-1]^{*}\tau(t)\) endfor return\(s\)
```

**Algorithm 1** Algorithm for STDI

The advantage of applying STDI is that intercellular messaging can be accomplished with very few spikes. In combination with timesteps compression, we can express and convey information with fewer spikes and much lower timesteps. An example of this characteristic is given in Fig. 5. This example demonstrates that with the use of STDI and timesteps compression, information that would otherwise require six spikes can be expressed with only one binary or burst spike, with a correspondingly lower timesteps. Interestingly, STDI makes the spikes conform to the poisson distribution with or without the use of TS.

### _SPPF Structure Conversion_

After comparing the YOLO series models, we chose the YOLOv5s as the backbone of our ANN to SNN conversion. Compared to the YOLOv3-tiny used by Spiking-YOLO, YOLOv5s has a deeper network and an optimized feature extraction structure such as FPN+PAN (i.e. SPPF), making it more interesting for development.

We first use ReLU as the activation function and then replace the Upsampling layer with a Contranspose layer to form the

Fig. 4: Spike-time-dependent integrated (STDI) coding. (a) shows the correspondence between the input value and the moment of spike issuance. (b) illustrates the main process of information transfer after using STDI. Where the accumulation phase is in \(t\in[-T,0]\). In the firing phase, the red dashed line is the variable threshold over time.

Fig. 5: Timesteps compression and STDI. Here we give an example of the effect of applying the timesteps compression and STDI. Assuming timesteps \(T\) is 6, frequency coding, timesteps compression, and STDI are used to express \(spiking\ ratio=1\), respectively. Where the black vertical line is the binary spike and the red vertical line is the burst spike.
ANN version of SUHD. Compared to the previous ANN to SNN works, the SUHD conversion work has an additional SPPF structure's conversion. SPPF structure includes three Maxpool layers with \(stride=1\), as shown in SPPF in Fig. 6. According to the conversion method of YOLOv3-tiny in Spiking-YOLO [12], we first set the SPPF layer directly to the CBR layer (Conv. + BN + ReLU).

We get significant performance improvements after using this structure. The results are shown in Sec. IV-B. However, the SPPF structure of the ANN version of SUHD led to an increase of approximately 3% mAP, while the conversion method of the SPPF structure can be extended to the whole YOLO series and even more SNN based deep learning models, thus facilitating the conversion of more efficient and complex structures. Therefore, we perform the conversion of the SPPF structure. For achieving fast Maxpool operations with any stride in deep SNN models, we propose a Spike-Maxpooling mechanism based on membrane potentials.

#### Iv-B1 Spike-Maxpooling According to Membrane Potential

The existing Maxpool methods in SNNs mainly include two kinds: one is the "winner take all" [38, 47], that is, the pooling neurons accept all the previous spikes and add them. The pooling layer value obtained by this pooling method is often greater than the real value. The other one is to calculate the number of spikes of maximum firing rate neurons directly or indirectly. These methods are complicated, especially in the case of compressed timesteps and STDI. Therefore, we proposed a Spike-Maxpooling method based on membrane potential, which allows for the Maxpool of SNNs at any stride in a simple way.

As shown in Spike-Maxpooling Based on Membrane Potential in Fig. 6, the main process has five phases:

1. **Accumulation phase**: The input spiking train is decoded to obtain the input voltage \(U_{in}\) and accumulated to the membrane.
2. **Restore Normalization**: Restore the activation values by Eq. 21. \[V_{r}=max_{in}*U_{in},\] (21) where \(max_{in}\) denotes the maximum sample activation value of the input of this channel in ANN, and the \(V_{r}\) denotes the activation value of this channel after restoring normalization in SNN. The purpose of this phase is to eliminate errors due to integer and floating point calculations.
3. **Pooling phase**: We name the region where the neurons that are involved in \(V_{r}\) competition as a candidate region, as shown within the circle in the Fig. 6. Neurons in candidate regions undergo \(V_{r}\) competition. The winner's \(V_{r}\) is passed on.
4. **Re-normalization phase**: Normalize \(V_{r}\) value to output voltage as \(U_{out}\) by Eq. 22. \[U_{out}=\frac{V_{r}}{max_{out}},\] (22) where \(max_{out}\) denotes the sample maximum activation value of the output of this channel in ANN.
5. **Firing phase**: Encoding the \(U_{out}\) to spike train and release it.

#### Iv-B2 Spike-SPPF

Based on the above work, we completed the conversion of the Spike-SPPF structure, the exact structure is shown in Spike-SPPF in Fig. 6. The results, as shown in Sec. IV-B, demonstrate that the accuracy of the SNN model improves by about 3% after using Spike-SPPF.

Fig. 6: The comparison of SPPF and Spike-SPPF. The traditional SPPF (left figure) contains two CBR layers and three Maxpool layers with stride=1 and connects these layers through a concat operation. As a comparison, Spike-SPPF uses Spike-Conv. layer and Spike-Maxpooling based on membrane potential to achieve the same operation on the spike train. The extended diagram of the circle in the right figure shows the specific method of Spike-Maxpooling based on the membrane potential, which includes five stages.
## IV Experiments

### _Experiments Setup_

Different comparative experiments were set up to verify the effectiveness of different structures or methods. The model is the ANN/SNN version of the SUHD model. The initial conversion method (SNN base code) is frequency coding + TS. The membrane potential was initialized with a value of 0.5 with reference to the [13]. The whole experiment is based on Intel Core i7-8700K CPU or NVIDIA RTX2080Ti GPU with CUDA 10.1. The datasets used in this work are PASCAL VOC [48] and MS COCO [49], where the PASCAL VOC dataset consists of three parts, train (2007+2012), val (2007+2012), and test (2007), including 8218, 8333, and 4952 images respectively. Specifically, we used train (2007+2012) as the training set and test (2007) as the validation set. The COCO dataset consists of two parts, train2017 and val2017, with train2017 including 118,287 images and val2017 including 5,000 images. We used train2017 as the training set and val2017 as the evaluation set. Unless explicitly specified, the metric utilized to assess the accuracy of experiments is the mAP@0.5.

### _Ablation Study of Spike-SPPF_

In order to demonstrate the impact of Spike-SPPF on improving the accuracy upper bound, an ablation study was conducted with and without Spike-SPPF. To ensure impartiality and proper operation of the model, the SPPF component was replaced with a CBR (Conv. + BN + ReLU) layer in ANN of the model without Spike-SPPF. The results of the ablation study are shown in Tab. I.

The results demonstrate that the Spike-SPPF can improve accuracy by approximately 2.3%. In particular, the accuracy loss of the ANN to SNN conversion here does not exceed 0.2%, which verifies that our proposed Spike-Maxpooling has the ability to losslessly convert the Maxpool layer with \(stride=1\).

### _Optimization of Accuracy and Speed Ablation Experiments_

Our proposed methods aim to achieve a faster and more precise SNN-based object detection model within limited timesteps. In order to evaluate the effectiveness of these methods, we conducted ablation experiments to measure the impact on speed and accuracy improvements.

The results are presented in Tab. II and III. In these experiments, we used a base code with uncompressed timesteps and frequency encoding, then gradually increased the compression scale and applied STDI. The initial timesteps were set to 64, and the model achieved an mAP of 72.5% and 53.6% on the PASCAL VOC and MS COCO datasets, respectively, with a processing speed of over 6500 (CPU)/720 (GPU) ms per frame.

Increasing the compression scale to 16 and reducing the compressed timesteps to 4 significantly improved the processing speed without compromising accuracy, indicating the effectiveness of timesteps compression.

Changing the encoding method to STDI further improved the mAP by 2.8% and 1% on the PASCAL VOC and MS COCO datasets, respectively, and reduced the inference time by about 38% (CPU)/17% (GPU), confirming the effectiveness of STDI in reducing the conversion error and improving the speed of object detection.

Finally, we increased the compression scale to 64 with compressed timesteps of 1, without reducing detection accuracy. This allowed us to achieve the fastest detection speeds of 189.4ms/frame and 189.5ms/frame (CPU) on PASCAL VOC and MS COCO datasets, respectively. We also conducted the same experiment on the GPU and achieved a speed of 90.1ms/frame at 64x compression.

### _Compared with the-State-of-the-Arts_

We compared the performance of SUHD with that of other methods on the PASCAL VOC and MS COCO datasets respectively. The results are shown in Tab. IV, Tab. V and Fig. 7, where Burst refers to applying burst spikes to prevent harm caused by truncation errors [43]. The data in tables are taken from the corresponding papers [12, 43, 50]. In comparison with the current advanced spiking object detection, our methods have achieved the optimal result in terms of speed, precision, and timesteps. As shown in the Tab. IV, with the PASCAL VOC dataset, our proposed method achieves an accuracy improvement of about 23% using 2000x fewer timesteps compared to Spiking-YOLO. Compared to the Burst + MLIpooling + SpiCalib method, we achieve almost the same accuracy using 128x fewer timesteps. The results under the MS COCO dataset are shown in Tab. V. Compared to the Spiking-YOLO and Burst + MLIpooling + SpiCalib methods, our method achieves about 30% and 9% improvement in accuracy using 750x and 128x fewer timesteps, respectively. Compared to FSHNN, we achieve a 12% accuracy improvement using 75x fewer timesteps.

Fig. 7: Detection results using different methods. Our work achieved the best results in detection.
\begin{tabular}{c c c c} \hline \hline Methods & Timesteps & mAP@0.5 & 
\begin{tabular}{c} Speed(ms/frame) \\ CPU \\ \end{tabular} \\ \hline ANN & - & 75.3 & - & - \\ \hline Bascode & 64 & 72.5 & 6639.2 & 720 \\ \hline
16x Compression & 4 & 72.5 & 867.3 & 185.8 \\ \hline
16x Compression + STDI & **4** & **75.3** & 541 & **152.3** \\ \hline \hline \end{tabular}

TABLE IIIAccuracy and speed improvement by different methods on MS COCO

\begin{tabular}{c c c} \hline \hline Methods & Timesteps & mAP@0.5 & 
\begin{tabular}{c} Speed(ms/frame) \\ CPU \\ \end{tabular} \\ \hline ANN & - & 54.8 & - & - \\ \hline Bascode & 64 & 53.6 & 6562 & 770 \\ \hline
16x Compression & 4 & 53.6 & 882.6 & 185.9 \\ \hline
16x Compression + STDI & **4** & **54.6** & 552 & **155.9** \\ \hline \hline \end{tabular}

TABLE IVComparison with other works under the PASCAL VOC dataset
### _Model Robustness_

The noise immunity of the model is also one of the main indicators of model performance. To verify the stability of the model, we evaluate the performance of the model under adverse conditions using standard additive gaussian white noise with signal-to-noise ratios (SNR) of 15dB and 30dB respectively. The FSHNN [35] model was also added for comparison. The results are shown in Tab. VI.

When SNR = 30 dB, we can see that our model has almost no loss(\(\leq 2.2\%\)). When SNR = 15 dB, accuracy suffered a more serious decline. However, the model still has an advantage of approximately 6.9% over the FSHNN. At the same time, the noise immunity of the SUHD model is not inferior to that of the ANN version of SUHD at high noise levels(\(loss\leq 0.5\%\)), as shown in Tab. VI and Fig. 8. This suggests that our proposed methods do not have an excessive impact on the robustness of the model. Thus, our proposed model has satisfactory performance in terms of robustness.

### _Energy Efficiency_

Energy consumption is an important metric for measuring the cost of model inference. Referring to other works [12, 35, 51], the generic index \(FLOPs\) is used to measure the complexity of the model. The \(energy\) denotes the energy cost of models. As proposed in Horowitz et al. [12, 52], the energy cost of one operation is 4.6pJ(FLOAT3T2 MAC)/0.9pJ(FLOAT3T2 AC). To measure SNN's energy consumption more rationally, we use the average FLOPs based on the conversion sample dataset. Specifically, we define SNN FLOPs as:

\[FLOPs=\sum_{l=1}^{n}\sum_{t=1}^{T}s^{l}(t)\ +\ \sum p_{in}, \tag{23}\]

where \(n\) denotes the sum of layers of the model, and the \(\sum p_{in}\) represents the sum of pixels of the input figures. Considering that the model accepts analog inputs, we define the input layer operations as MAC operations. The other operations are caused by spikes as floating-point AC operations. In contrast to spiking-yolo, the SUHD model uses YOLOv5s as its backbone. Therefore, our evaluation is between YOLOv5s and SUHD. The energy efficiency profile of YOLOv5s is derived from official data. Based on this method, we compared the YOLOv5s and the SUHD's energy costs.

The results are shown in Tab. VII, which illustrate that the SUHD model is at least 200 times more energy efficient than the YOLOv5s model.

### _Algorithm Deployment and Evaluation in Robot Platform_

To evaluate the ability of the model to detect objects in dynamic scenarios and the performance of the model on mobile platforms, in this section we deploy the algorithm to a mobile robot platform and test it.

#### Iv-G1 Algorithm Deployment

We use the summit-xl mobile robot, as shown in Fig. 9. Summit-xl is equipped with an Intel Core i3-9100 CPU, 7.16GB RAM, four independent drive wheels, and an Axis gimbal camera. The deployment process consists of three main steps: We first built the python algorithm runtime environment based on ubuntu 16.04 on x86 architecture and deployed the algorithm to the robot. The video signal is then acquired using the gimbal camera. The video signal is in h265 format and is hard-decoded. Finally, the decoded video image stream is fed into the model in real time for object detection.

Fig. 8: Performance of the models at different SNR. The first three rows show the detection results using the SUHD model and the last row shows the detection results using the ANN model.

Fig. 9: Summit-xl mobile robot platform.
#### Iv-A2 Performance Evaluation in Dynamic Scenarios

We produced two video datasets. Both datasets contain one video each.

**Dataset A** was sampled from the robot gimbal camera at 6 fps. Three motion speeds, fast, medium, and slow were included throughout the sampling process. At the same time, both the robot and the object can move. This dataset is therefore comprehensive and can accurately assess the performance of the algorithm in dynamic scenes. The entire dataset contains 920 frames with a total of 4140 labels in 9 categories, with an average of 4.5 labels per frame.

**Dataset B** was sampled from a fixed viewpoint and contains 50 frames in 3 categories, with a total of 219 labels, averaging 4.38 labels per frame. The overall speed of object movement within the frames is slow.

First, we train the ANN model using the COCO dataset and then convert it into an SNN model. In the second step, we deploy the SNN models to the PC and mobile robot described at the beginning of this section, respectively. In the third step, we use datasets A and B to compute the mAP to obtain the object detection capability of the model in dynamic scenes. In the fourth step, we test the object recognition capability of the model deployed on the robot using a real-time robot video streaming. At the same time, we record video data in the same scene and then move the video to the PC to simulate object detection in a real-time dynamic scene. The results were shown in Tab. VIII, where the evaluation indicator of Dataset A and Dataset B is mAP@0.5 for detection accuracy, and the evaluation indicator of the real-time robot video streaming is recognition accuracy, which is defined as follows:

\[Acc=\frac{TP+TN}{P+N}, \tag{24}\]

where \(Acc\) is recognition accuracy, \(TP+TN\) is the sum of the number of objects correctly classified, and \(P+N\) is the sum of the number of all objects. The results of Dataset A and B show that the model deployed on the robot produced no additional losses. This result indicates that our algorithm has strong repeatability. Due to the use of moving observation points in dataset A, there is a more significant blurring of some of the frames, which results in an mAP of 69.8%. Dataset B uses a fixed viewpoint and therefore achieves an mAP of 92.5%. Fig. 10(a) (b) show the difference in performance between the model on the PC and the model deployed to the mobile robot. In addition, we examined the object detection of the algorithmic model on the robot side under real-time robot video streaming and compared its performance with that of the PC side model under the same dynamic scene video. The results are shown in Fig. 10(c) and the Real-time signal of Tab. VIII. The two results differ by only 2.7%. The reason for the error is that the robot has less computing ability than the PC, resulting in greater frame drops and blurring of the live video signal it receives, thus reducing recognition precision. This result suggests that the detection performance of the model is not overly compromised by deployment to the robot side.

Fig. 10: Performance evaluation in dynamic scenarios. (a) and (b) show detection results for Dataset A and Dataset B, respectively. The top row shows PC results and the bottom row shows robot results. (c) presents detection results for video streaming, with the top row showing the results for the simulated real-time video streaming on the PC and the bottom row showing the results for the real-time robot video streaming on the robot.
## V Conclusion

This paper presents three main methods for optimizing the conversion accuracy and run speed of SNNs. The first method is timesteps compression, which significantly reduces the required timesteps to the minimal one timestep during lossless conversion. The second method is STDI, which improves the efficiency of SNN inference by an average of 38% on the CPU and 17% on the GPU by increasing the information capacity of a single spike. The third approach is a Spike-Maxpooling mechanism based on membrane potential, which facilitates the lossless conversion of complex and efficient structures in ANNs.

By combining timesteps compression and STDI, we achieved results that are 34 times faster on the CPU than initial conversion method (frequency coding + TS). We have further built the spike-based high-performance object detection SUHD, which can run on mobile platforms with comprehensive performance up to the state-of-the-art. Especially, SUHD is currently the deepest object detection model that can achieve a lossless ANN to SNN conversion with an ultra low timesteps. Our methods have shown competitive results and have significant potential for improving the efficiency and accuracy of SNNs, while offering the possibility to solve the problem of deploying SNN models on the mobile terminal. We believe that our methods can contribute to the large-scale popularisation of SNNs application in the future.
