Enabling energy-Efficient object detection with surrogate gradient
  descent in spiking neural networks
Spiking Neural Networks (SNNs) are a biologically plausible neural network
model with significant advantages in both event-driven processing and
spatio-temporal information processing, rendering SNNs an appealing choice for
energyefficient object detection. However, the non-differentiability of the
biological neuronal dynamics model presents a challenge during the training of
SNNs. Furthermore, a suitable decoding strategy for object detection in SNNs is
currently lacking. In this study, we introduce the Current Mean Decoding (CMD)
method, which solves the regression problem to facilitate the training of deep
SNNs for object detection tasks. Based on the gradient surrogate and CMD, we
propose the SNN-YOLOv3 model for object detection. Our experiments demonstrate
that SNN-YOLOv3 achieves a remarkable performance with an mAP of 61.87% on the
PASCAL VOC dataset, requiring only 6 time steps. Compared to SpikingYOLO, we
have managed to increase mAP by nearly 10% while reducing energy consumption by
two orders of magnitude.
Jilong Luo, Shanlin Xiao, Yinsheng Chen, Zhiyi Yu
Enabling Energy-Efficient Object Detection with Surrogate Gradient Descent in Spiking Neural Networks

###### Abstract

Spiking Neural Networks (SNNs) are a biologically plausible neural network model with significant advantages in both event-driven processing and spatio-temporal information processing, rendering SNNs an appealing choice for energy-efficient object detection. However, the non-differentiability of the biological neuronal dynamics model presents a challenge during the training of SNNs. Furthermore, a suitable decoding strategy for object detection in SNNs is currently lacking. In this study, we introduce the Current Mean Decoding (CMD) method, which solves the regression problem to facilitate the training of deep SNNs for object detection tasks. Based on the gradient surrogate and CMD, we propose the SNN-YOLOv3 model for object detection. Our experiments demonstrate that SNN-YOLOv3 achieves a remarkable performance with an mAP of 61.87% on the PASCAL VOC dataset, requiring only 6 time steps. Compared to Spiking-YOLO, we have managed to increase mAP by nearly 10% while reducing energy consumption by two orders of magnitude1.

Footnote 1: The code is made available at [https://github.com/xiaolongren969/](https://github.com/xiaolongren969/) SNN-YOLOv3

Jilong Luo, Shanlin Xiao\({}^{*}\), Yinsheng Chen, Zhiyi Yu\({}^{*}\)+ Sun Yat-sen University, China Energy-Efficient, Object Detection, Spiking Neural Networks, Surrogate Gradient

## 1 Introduction

As a third-generation artificial neural network [1], Spiking Neural Networks (SNNs) are promising for implementing low-power artificial intelligence algorithms using event-driven neuromorphic hardware [2, 3, 4]. Based on biological plausibility [5], SNNs emulate information processing mechanisms observed in the biological neural system, where computation and information transfer between neurons occur through discrete binary events [6]. Despite the attractive energy efficiency of spiking neural networks, training SNNs remains a significant challenge. One of the primary reasons for this challenge is the complexity of the dynamics model and the non-differentiability of spiking neurons, typically modeled as IF or LIF neurons, which makes performing gradient descent-based backpropagation difficult [7, 8].

Several researchers have proposed training SNNs using ANN-to-SNN conversion method [9, 10], in which ANNs (Artificial Neural Networks) with ReLU activation function are initially trained via gradient descent and then converted into SNNs with integrate-and-fire neurons by applying appropriate threshold balancing techniques [11]. However, SNNs obtained through ANN-to-SNN methods generally occur 2000-3000 time steps to achieve acceptable accuracy. Here, a time step denotes the time unit for the forward propagation of a single layer, effectively represents network latency [12].

To reduce the latency, the gradient surrogate-based backpropagation algorithm [7, 13] has been introduced for end-to-end gradient descent learning on spiking train. Within these algorithms, the non-differentiable neuron model completes the backpropagation process by specifying a surrogate gradient as a continuous approximation of the actual gradient [14]. Training SNNs with gradient surrogate substantially decreases the inference latency by nearly 100x (e.g., only requiring fewer than 30 time steps). Despite the appealing property of SNNs, previous research has mainly focused on less complex tasks (image classification) and small-scale datasets (MNIST and CIFAR10), with relatively shallow network structures (\(<\)30 layers) [15, 16].

In this study, we investigate more complex machine learning problems (object detection) in deep SNNs, using the gradient surrogate approach. Object detection is considered a demanding and challenging task in computer vision, aiming to recognize multiple objects and calculate the exact coordinates of the bounding box in images or videos. Unlike image classification tasks, when predicting the output values of a neural network, object detection requires predicting continuous numerical or real outputs, rather than just selecting the category with the highest probability (using the argmax function), as is typically done in image classification tasks. Our contributions can be summarized as follows:

* We present the first SNN model that implement object detection using surrogate gradient to achieve the state-of-the-art performance (61.87% mAP) on the non-trivial dataset of PASCAL VOC.
* We introduce the Current Mean Decoding (CMD) method, which solve the regression problem to facilitate the training of deep SNNs for object detection tasks.
## 2 Methods

### Surrogate gradient for spiking neuron models

Unlike ANNs, SNNs utilize spike trains for computation and information transmission among neurons. The dynamics of the classic IF neuron model [17, 18] can be described as follows:

\[V_{mem,j}^{l}\left[t\right]=V_{mem,j}^{l}[t-1]+I_{j}^{l}[t]-V_{th}s_{j}^{l}[t] \tag{1}\]

where \(s_{j}^{l}[t]\) represents the spike state of the j-th neuron in the l-th layer at time step t. \(x_{j}^{l}[t]\) represents the input membrane potential, and \(V_{mem,j}^{l}[t]\) represents the membrane potential of the j-th neuron in the l-th layer. The description of \(I_{j}^{l}[t]\) can be expressed as follows:

\[I_{j}^{l}\left[t\right]=\sum_{i}w_{i,j}^{l}s_{i}^{l-1}\left[t\right]+b_{j}^{l} \tag{2}\]

where \(w\) and \(b\) represent the synaptic weights and bias, respectively. When the membrane potential \(V_{mem,j}^{l}[t]\) of the j-th neuron in the l-th layer exceeds the threshold voltage \(V_{th}\), a spike \(s_{j}^{l}[t]\) is emitted. The mathematical formula is as follows:

\[s_{j}^{l}\left[t\right]=H\left(V_{mem,j}^{l}\left[t\right]-V_{th}\right) \tag{3}\]

the mathematical description of \(H(\cdot)\) is the Heaviside step function. This function produces a value of 1 when x is greater than or equal to 0, and 0 otherwise. Due to the non-differentiability of Heaviside unit step function, the surrogate gradient method is used to estimate gradient computations during backpropagation. The fundamental idea of surrogate gradient is to update the weights using gradient backpropagation via a surrogate gradient function rather than the unit step function. In this research, we have chosen the arctangent function as our surrogate gradient function. Its mathematical expression is illustrated below:

\[g\left(x\right)=\ \frac{1}{\pi}\arctan\left(\frac{\pi}{2}\alpha x\right)+\frac {1}{2} \tag{4}\]

Its derivative is represented as follows:

\[g^{{}^{\prime}}\left(x\right)=\frac{\alpha}{2\left(1+\left(\frac{\pi}{2} \alpha x\right)^{2}\right)} \tag{5}\]

### Current mean decoding (CMD)

Rate decoding is a commonly used approach to transfer information in spiking neural networks that decodes information intensity dependent on the rate of neuron spike emissions [19, 20]. Nonetheless, in specific tasks, particularly those concerned with regression problems like object detection, rate decoding will struggle and require more advanced decoding strategies.

Object detection generally involves two primary tasks: classification and regression. In the classification task, the classification result can be determined by the max magnitude of the output neuron spike firing rate (rate decoding) in SNNs. However, it is often necessary to predict the location, size and shape of the object in regression tasks, which requires the output value space of network to be real-valued. Nevertheless, the discrete spikes employed for rate decoding are not directly mapped to a continuous numerical space, resulting

Figure 1: Traditional rate decoding and proposed current mean decoding (CMD) in SNN-YOLOv3. The left part shows that the input of the image is coded by a direct encoding way and the activation layer in the network is replaced with IF neurons. The last layer is the decoding layer, and the right part shows the different decoding methods used in the decoding layer.
in the network output being discrete. Consequently, the discrete event of rate decoding might lead to a loss of accuracy when representing continuous outputs. While continuous values may be approximated through rate decoding, this approximation need a compromise between precision and time step.

For this reason, we introduce current mean decoding(hereafter abbreviated as CMD) in spiking neural networks, which is a more powerful decoding technique that exploits the dynamic properties of neurons. Fig.1 illustrates a schematic diagram of CMD, which collects the currents produced at synapses upon neuron spike event. Subsequently, it accomplishes information decoding by computing the mean value of input current, which is defined as follows.

\[Output=\frac{\sum\limits_{t=1}^{T}\sum\limits_{i=1}^{n}x_{i}\left[t\right] \times w_{i}}{Timestep} \tag{6}\]

where \(x_{i}[t]\) represents the spike firing spike state of the \(i\)-th neuron of the presynaptic neuron at time step \(t\), and \(w_{i}\) represents the weight associated with the corresponding neuron. These values are multiplied to yield the synaptic current.

On the comparison with rate decoding, this decoding method provides a better approximation of continuous values and offers better accuracy and flexibility for regression problems.

## 3 Experiments

### Experimental setup

In this experiment, we choose the classical version of the YOLOv3 [21] real-time object detection network for validating the effectiveness of the CMD method. SNN-YOLOv3 was tested on the PASCAL VOC datasets [22] with simulations based on the PyTorch platform and all experiments were done on NVIDIA Tesla V100 GPUs. The SNN-YOLOv3 obtains more efficiency and robust spike feature trains by using the first convolutional set as the encoding layer without additional encoding layers.

During the training process, we adopt a stochastic gradient descent optimiser with a momentum parameter of 0.9 and a cosine decay scheduler for fine-tuning the learning rate. we set the weight decay for the upper bound parameter \(\theta\) of SNN-YOLOv3 to \(5\times 10^{-3}\). Furthermore, We used normalization and horizontal flipping method for data augmentation.

### Experimental results

In order to verify and analyse the effectiveness of our proposed method, we evaluate the performance of our methods for object detection tasks on PASCAL VOC datasets.

Fig. 2 shows the object detection performance of SNN-YOLOv3 as the number of training epochs increases. In the figure, the green and red curves represent the performance trends for the CMD method with \(T=4\) and \(T=6\), respectively, while the blue curves illustrate the accuracy variation for the rate decoding method with \(T=6\). From the experimental results, we observe the following conclusions: (1) In general, CMD brings higher performance compared to rate decoding. (2) As the current mean decoded SNN is trained with a larger number of time steps, its performance will further increase. The results clearly indicate that when utilizing rate decoding, the SNN network struggles to learn, leading to consistently low accuracy. In contrast, the CMD method shows a remarkable ability to improve accuracy with increasing number of training iterations. This illustrate the excep

Figure 3: Object detection results on PASCAL VOC dataset.

Figure 2: Experimental results of SNN-YOLOv3 on PASCAL VOC dataset for various time step; maximum mAP is in parentheses.
tional effectiveness of the CMD method for object detection tasks in SNNs.

Moreover, the remarkable performance of SNN-YOLOv3 is also shown in the other examples in Fig. 3. The SNN-YOLOv3 precisely locates and classifies various object categories within images, including person, cars, and bicycles which proves its excellent object localisation capability.

### SNN-YOLOv3 energy efficiency

In order to assess the outstanding energy efficiency of SNN-YOLOv3, we compare the computational operations of SNN-YOLOv3 and YOLOv3 within the realm of digital signal processing. Within convolutional deep neural networks, the convolutional layer is the main computational region, where the multiply-accumulate (MAC) operation is the main executive operation. However, the operation performed in the spiking neural network is an accumulation (AC) operation because spiking events are binary events. The input current is integrated or accumulated into the membrane potential only when the neuron received a spike. For a fair comparison, we focus on the number of MACs and ACs consumed during single-image object detection. According to the literature [23], 32-bit floating-point MAC operations consume 4.6 pJ and 32-bit floating-point AC operations consume 0.9 pJ.

Based on these operation energy results, we calculated the energy consumption of YOLOv3 and SNN-YOLOv3 by multiplying the FLOPs (floating point operations) and the energy consumption per MAC or AC operation. if it is a SNN model, it needs to be further multiplied by time step. According our simulations, the FLOPs for ANN-YOLOv3 and SNN-YOLOv3 were \(66.19\) and \(0.425\) GFLOPs, respectively. Fig. 4 shows the results, where SNN-YOLOv3 is more than 158 times energy efficient than YOLOv3 in 32-bit FL operations both under the \(T=4\) and \(T=6\).

### Comparison with the State-of-the-Art

We compare our approach with other state-of-the-art ANN-to-SNN conversion methods on the PASCAL VOC dataset to achieve SNN object detection on non-trivial datasets. Our calculation result as shown in Table 1. We calculated the energy consumption of SNN-YOLOv3 running on a neuromorphic chip (TrueNorth) and compared it to Spiking YOLO. the GFLOPS/W of TrueNorth is 300 GFLOPs/W, and we defined a time step as 1 ms (1 kHz synchronization signal in TrueNorth) [2]. For SNN-YOLOv3, ours proposed method can achieve 61.87% mAP using only 6 time steps. Compare to Spiking-YOLO, we have nearly 10% increase in mAP while requiring nearly two orders of magnitude less energy consumption. Considering that the TrueNorth chip was initially introduced in 2014, we can expect increased energy and computational efficiency as neuromorphic chips advance and produce better results.

## 4 Conclusion

In this paper, we introduce the energy-efficient SNN-YOLOv3, which is the first SNN model that implement object detection using surrogate gradient. It achieves the state-of-the-art performance (61.87% mAP) on the PASCAL VOC dataset using only 6 time steps. Compared to previous work, we can accomplish object detection with less energy consumption. In addition, we proposed current mean decoding method for solving regression problem in SNN, which provides a different approach to work out more advanced machine learning problems with deep SNNs.
