Optimising Event-Driven Spiking Neural Network with Regularisation and
  Cutoff
Spiking neural network (SNN), next generation of artificial neural network
(ANN) that more closely mimic natural neural networks offers promising
improvements in computational efficiency. However, current SNN training
methodologies predominantly employ a fixed timestep approach, overlooking the
potential of dynamic inference in SNN. In this paper, we strengthen the
marriage between SNN and event-driven processing with a proposal to consider
cutoff in SNN, which can terminate SNN anytime during the inference to achieve
efficient inference. Two novel optimisation techniques are presented to achieve
inference efficient SNN: a Top-K cutoff and a regularisation. The Top-K cutoff
technique optimises the inference of SNN, and the regularisation are proposed
to affect the training and construct SNN with optimised performance for cutoff.
We conduct an extensive set of experiments on multiple benchmark frame-based
datsets, such as Cifar10/100, Tiny-ImageNet and event-based datasets, including
CIFAR10-DVS, N-Caltech101 and DVS128 Gesture. The experimental results
demonstrate the effectiveness of our techniques in both ANN-to-SNN conversion
and direct training, affirming their compatibility and potential benefits in
enhancing accuracy and reducing inference timestep when integrated with
existing methods. Code available:
https://github.com/Dengyu-Wu/SNN-Regularisation-Cutoff
Dengyu Wu, Gaojie Jin, Han Yu, Xinping Yi, Xiaowei Huang
# Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff

###### Abstract

Spiking neural networks (SNNs), next generation of artificial neural networks (ANNs) with the benefit of energy efficiency, have achieved the accuracy close to its ANN counterparts, on benchmark datasets such as CIFAR10/100 and ImageNet. However, comparing with frame-based input (e.g., images), event-based inputs from e.g., Dynamic Vision Sensor (DVS) can make a better use of SNNs thanks to the SNNs' asynchronous working mechanism. In this paper, we strengthen the marriage between SNNs and event-based inputs with a proposal to consider anytime optimal inference SNNs, or AOI-SNNs, which can terminate anytime during the inference to achieve optimal inference result. Two novel optimisation techniques are presented to achieve AOI-SNNs: a regularisation and a cutoff. The regularisation enables the training and construction of SNNs with optimised performance, and the cutoff technique optimises the inference of SNNs on event-driven inputs. We conduct an extensive set of experiments on multiple benchmark event-based datasets, including CIFAR10-DVS, N-Caltech101 and DVS128 Gesture. The experimental results demonstrate that our techniques are superior to the state-of-the-art with respect to the accuracy and latency.

## I Introduction

SNNs have recently attracted significant research and industrial interests thanks to its energy efficiency and low latency [1], and there are neuromorphic chips such as Loihi [2] and TrueNorth [3] on which SNNs can be deployed. Mechanistically, SNNs mimic biological neurons, and the neurons process and forward spikes independently. With such an asynchronous working mechanism, only a (small) subset of neurons will be activated during inference. That is, energy efficiency is inherent to SNNs.

The asynchronous mechanism also suggests that event-based input may make a better use of SNNs. Actually, neuromorphic sensors such as Dynamic Vision Sensor [4, 5, 6] and Dynamic Audio Sensor (DAS) [7] have been developed to generate binary "events", which are ideal inputs to SNN. For example, unlike conventional frame-based cameras which measure the "absolute" brightness at a constant rate, DVS cameras are bio-inspired sensors that _asynchronously_ measure per-pixel brightness changes (called "events"), and output a stream of events that encode the time, location and sign of the brightness changes [8]. DVS reveals the sparsity and asynchronicity in recognition systems for computational efficiency [9, 10, 11]. To deal with event-based input, we propose to consider anytime optimal inference SNNs, or AOI-SNNs, which allow the termination at any time during the inference on a spike train (i.e., an input) and return the best possible inference result. Such SNNs enable the cutoff during the inference without (significantly) compromising the performance, and thus can achieve the best in terms of accuracy and latency.

Regarding the training of SNNs, a mainstream approach is through ANN-to-SNN conversion, which adopts the mature training regime of ANNs to first train a high-accuracy ANN, and then convert it into SNN. Such conversions via ANNs have resulted in research to focus on achieving the near-zero conversion loss. However, existing conversion methods [12, 13, 14] mostly conduct empirical experiments on frame-based benchmark datasets such as ImageNet [15] and CIFAR10/100 [16]. In this paper, we will focus on event-based input, and therefore the AOI-SNNs, and explore effective training and inference methods to improve accuracy and latency together.

When considering ANN-to-SNN conversions to deal with DVS inputs, there are two possible ways. The first one aggregates the sparse events in the DVS stream into a frame-based input, on which the SNN processes as a whole. This resembles the ANN processing a static input (such as an image). As explained in Section III-A, the frame-based input will base on the average spike rate, neglecting the spike timing information. The second is to directly work with the event-based input, by considering e.g., AOI-SNNs. An obvious benefit is that SNNs can exploit sparse events in the DVS input, enabling energy-efficient operation and reduced latency. In addition, unlike frame-based input, the event-based input does not need an encoder at or before the first layer, which allows SNNs to operate asynchronously and achieve extra low-latency (further explained in Section III-A).

This paper makes two key technical contributions. Firstly, we propose a regularisation technique to influence the activation distribution during ANN training, which results in an SNN that can classify with less input information. As will be discussed in Related Work (Section II), with our proposed regulariser, we can train an ANN without clipping and do not need to apply any quantisation-aware technique. Experiments in Section V-D show that we can achieve better accuracy than the state-of-the-art methods on both direct training and ANN-to-SNN conversion. Clipping (and quantisation-aware) techniques have been the status quo in this area due to the recent progress [17, 13, 12, 18] and our result suggests that there is an alternative, and probably better, way to get an improved SNN. Instead of simulating non-differentiable SNN activations during ANN training, our regulariser enables the attainment of a better distribution of SNN current by actively regularising the activations of the possible misclassifications. The regulariser is based on a new theoretical result (Section IV-A) that a smaller ratio of threshold voltages to average accumulated current can result in an SNN that can achieve optimised performance at any time during the inference.
The second contribution is that, instead of setting the inference length to always be \(T_{total}\), we can explore an early cutoff mechanism that enables the SNN model to automatically achieve optimal latency and energy efficiency. As shown in Fig. 1, the SNN model will run a monitoring mechanism to determine when it is sufficiently confident to make a decision. Once such a decision is made at time \(t<T_{total}\), a cutoff action is triggered so that the SNN will not take future inputs until the time \(T_{total}\). Therefore, not only will this lead to lower latency (because decision is made at time \(t\) rather than \(T_{total}\)), but it will be also more energy-efficient (because no spike will be generated after time \(t\)).

## II Related work

The application of SNNs to a data source can be separated into two phases: training and inference. Broadly speaking, the training algorithms for SNNs can be categorised into direct training (DT) and ANN-to-SNN conversion. Recently, Spike-based Error Backpropagation [20, 21, 22, 23, 24] direct train a neural network to process the temporal information of input spikes. However, either direct training or conversion algorithm [25, 18] needs to collapse the input spikes into frames for the training. More specifically, the first layer in former SNN needs to wait for the full spike train within one frame to generate one spike, while the latter can respond very fast as long as the SNN receives spikes. Normally, the number of frames in direct training is kept small to reduce training complexity and determines the latency of SNN in inference. In contrast, ANN-to-SNN conversion can incorporate the maximum number of spikes during training to consider the SNN with optimal latency.

For the ANN-to-SNN conversion, early studies [19, 26] use the maximum value of activation to normalise the weights from ANN, and [27] proves that the normalisation can also be achieved by greedily searching for the optimal threshold using the input spike train. A unified conversion framework is studied in [18]. Besides, there are hybrid methods [28, 29] that combine conversion and direct training. Tandem Learning [30] leverages the gradient from ANN to update SNN during training. The first two columns of Table I present the technical ingredients of different conversion methods for the training phase. Recent work [12, 18] shows that, outlier elimination (OE) in ANN activations can be implemented by applying _clipping_ operation after the Rectified Linear Unit (ReLU). Based on this, [17, 13] further minimise the quantisation error by Quantisation-aware (QA) training. Different from the above methods, we develop a new regulariser to achieve the better performance _without_ clipping, and moreover, noticeably, we are _free from_ applying QA training.

For the inference phase, as indicated in the last two columns

Fig. 1: An illustrative diagram showing the regularisation for improving SNN latency and the cutoff mechanism for reducing latency on Cifar10-DVS dataset. Cutoff is triggered when \(S_{gap}\) is greater than \(\beta\), a value dynamically determined by a confidence rate as introduced in Section IV-C.
of Table I, the soft-reset mechanism [19] and the additive white noise to membrane potential [12, 18, 13] can significantly increase the conversion efficiency. To the best of our knowledge, there is no existing work on cutoff in the inference phase, and our confidence-based method is the first of its kind.

## III Preliminary

In this section, we discuss the event-based input in spiking neuron and introduce the ANN-to-SNN conversion. To facilitate the analysis, we use **bold symbol** to represent vector, \(l\) to denote the layer index, and \(i\) to denote the index of elements. For example, \(\mathbf{a}^{l}\) is a vector and \(a_{i}^{l}\) is the \(i\)-th element in \(\mathbf{a}^{l}\). Inference time \(t\) represents the time length of input. \(T_{total}\) denotes the maximum time length of input and it can be various depending on dataset. \(\mathbf{W}^{l}\) is weight matrix at the \(l\)-th layer.

### _Integrated-and-fire model_

Conversion-based SNN uses integrate-and-fire (IF) neuron as the basic computing unit to approximate ReLU in ANN [18]. Fig. 2 illustrates the inference process in IF neurons. The input spike train \(X_{i}(t)\) charges the membrane potential \(V_{i}(t)\) with weighted current. The weighted current and bias current are translated from the weight \(\mathbf{W}^{l}\) and bias \(\mathbf{b}^{l}\) in ANN. When \(V_{i}(t)\) reaches the threshold \(V_{thr}\), the neuron will generate a spike and then reset the \(V_{i}(t)\) by subtracting \(V_{thr}\). The _reset by subtraction_ mechanism was firstly suggested in [19] to reduce information loss during inference. The dynamics of IF neuron can be described as

\[\mathbf{V}^{l}(t)=\left\{\begin{array}{ll}\mathbf{V}^{l}(t-1)+\mathbf{Z}^{l}(t)-\mathbf{ \theta}^{l}(t)V_{thr}^{l}&l>1\\ \mathbf{Z}^{1}(t)&l=1\end{array}\right. \tag{1}\]

where \(\mathbf{\theta}^{l}(t)\) is a step function i.e., \(\theta_{i}^{l}(t)=1\) if \(V_{i}^{l}(t-1)+Z_{i}^{l}(t)\geq V_{thr}^{l}\) and \(\theta_{i}^{l}(t)=0\) otherwise. \(\mathbf{Z}^{l}(t)\) is the input current such that

\[\mathbf{Z}^{l}(t)=\mathbf{W}^{l}\mathbf{\theta}^{l-1}(t)+\mathbf{b}^{l}\ \ \ \ \ \text{when }l>1. \tag{2}\]

For the event-based inputs (e.g., from a DVS sensor), \(\mathbf{Z}^{l}(t)\) at the first layer, i.e., \(\mathbf{Z}^{1}(t)\), can be initialised as

\[\mathbf{Z}^{1}(t)=\mathbf{W}^{1}\mathbf{X}(t)+\mathbf{b}^{1} \tag{3}\]

where \(\mathbf{X}(t)\) is the time-dependent spike train, i.e., the input may change the charging current with time during the inference. To consider the temporal information, we split the spike train into \(F\) frames and duration of each frame is equal to \(T=T_{total}/F\), where \(F\in\mathbb{Z}^{+}\). We write \(\mathbf{\bar{X}_{f}}\) to represent the average spiking rate of \(f\)-th frame, i.e., \(\mathbf{\bar{X}_{f}}=1/N_{\max}\sum_{t=T:(f-1)}^{T:f}\mathbf{X}(t)\) such that \(N_{\max}\) is the maximum spikes of all training frames in dataset \(D\). The spiking resolution of \(X(t)\) can be roughly computed as \(S_{r}=N_{\max}/T\). For event-based input, SNN can manifest faster inference due to immediate response after receiving the first spike, and it completes the inference whenever the spike train ends, i.e., at \(T_{total}\). The event-based benchmarks are further introduced in Section V-B. This characteristic makes it possible that the inference time is dynamic for different inputs. In this paper, with the cutoff technique as in Section IV-C, we will show that the average latency of the inference in SNN can be further reduced (to some \(t\leq T_{total}\)).

### _Temporal training_

Regarding the direct training of SNN in [22, 23], the resulted SNN can make decision by averaging output spikes of consecutive frames. Assuming that the inference of each frames is independent, such process can be approximated in ANN training by letting the loss function be

\[L_{TT}=\frac{1}{F}\sum_{f=1}^{F}L_{CE}(\mathbf{Y}_{f},\mathbf{\hat{Y}}) \tag{4}\]

where \(\mathbf{Y}_{f}\) is output of \(\mathbf{\bar{X}_{f}}\) after softmax, \(\mathbf{\hat{Y}}\) is the ground truth and \(L_{CE}\) is cross-entropy loss. Temporal training loss (\(L_{TT}\)) was suggested in [23] that achieves better generalisation. To simplify the theoretical analysis, we let \(F=1\) in Section III-C & IV. The further explanation of temporal training is given in appendix V-C, including the impact of \(F\) on ANN training and extension of theories to \(F>1\). To ensure the independence between frames, the membrane potential of hidden layer is reset after each frame, while that of output layer is reset after the last frame, which is feasible in hardware implementation [31, 32].

### _ANN-to-SNN conversion_

The conversion method is mainly based on integrated-and-fire (IF) neuron, which generates spikes depending on positive accumulated current, corresponding to ReLU activation in ANN. An existing conversion method [18] uses current normalisation methods by letting

\[\frac{1}{T\cdot S_{r}}\sum_{t=0}^{T}\mathbf{Z}^{1}(t)=\mathbf{a}^{1} \tag{5}\]

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{2}{c|}{Training} & \multicolumn{4}{c}{Inference} \\ \cline{2-6}  & OE through & Apply QA & Soft-reset & Additive Noise & Cutoff \\ \hline
[19] & - & - & \(\surd\) & - & - \\
[12] & clipping (COE) & - & \(\surd\) & - & - \\
[18] & clipping (COE) & - & \(\surd\) & \(\surd\) & - \\
[17] & clipping (COE) & \(\surd\) & \(\surd\) & - & - \\
[13] & clipping (COE) & \(\surd\) & \(\surd\) & \(\surd\) & - \\ \hline Ours & regularisation (ROE) & - & \(\surd\) & \(\surd\) & \(\surd\) \\ \hline \hline \end{tabular}
\end{table} TABLE I: Technical ingredients of different conversion methods. OE and QA denote Outlier Elimination and Quantisation-aware technique respectively. COE: Clipping for Outlier Elimination; ROE: Regularisation for Outlier Elimination.
where \(\mathbf{a}^{1}\) is the output of ReLU activation at the first layer of ANN. The spiking rate of each SNN neuron at layer \(l\) is defined as \(\mathbf{r}^{l}(t)=\mathbf{N}^{l}(t)/t\), where \(\mathbf{N}^{l}(t)\) is the number of spikes received up to time \(t\) by neuron at layer \(l\). The relationship between spiking rate in SNN and activation in ANN has been theoretically proved in [18], which gives

\[\mathbf{r}^{l}(t)=\frac{1}{V_{thr}^{l}}\Big{(}\mathbf{W}^{l}\mathbf{r}^{l-1}(t)+\mathbf{b}^{l} \Big{)}-\mathbf{\Delta}^{l}(t) \tag{6}\]

where \(\mathbf{\Delta}^{l}(t)\triangleq\mathbf{V}^{l}(t)/(tS_{r}V_{thr}^{l})\) represents the residual spiking rate. The spiking rate at the first layer can be initialised as \(\mathbf{r}^{1}(t)=\mathbf{a}^{1}/V_{thr}^{1}-\mathbf{\Delta}^{1}(t)\). Note that, we use \(tS_{r}\) to represent the timestep in [18]. Then, the current normalisation can be achieved by

\[\tilde{\mathbf{W}}^{l}\leftarrow\mathbf{W}^{l},\ \tilde{\mathbf{b}}^{l}\leftarrow\frac{1}{ \lambda^{l-1}}\mathbf{b}^{l},\ V_{thr}^{l}\leftarrow\frac{\lambda^{l}}{\lambda^{l -1}} \tag{7}\]

where \(\lambda^{l}\) be the maximum value of the activation at layer \(l\). For temporal training, the temporal input frames share the same \(\lambda^{l}\).

## IV Methods

We introduce two novel techniques: one is for the training and the other for the inference. Section IV-A presents the theoretical underpinning of the regulariser, which in turn is introduced in Section IV-B. This is followed by the introduction of cutoff mechanism in Section IV-C for the inference.

### _Anytime optimal inference SNNs_

The regulariser is based on an investigation into the design of AOI-SNNs. An AOI-SNN is able to perform optimally under different settings on the inference time for processing an input. When \(t\) is large enough to make \(\Delta^{l}(t)\) negligible, we define the desired spiking rate as follows:

\[\mathbf{r}_{d}^{l}=\mathbf{a}^{l}/V_{thr}^{1} \tag{8}\]

We start from establishing a theoretical underpinning between spiking rate \(\mathbf{r}^{l}(t)\) and its desired value \(\mathbf{r}_{d}^{l}\). Let \(\phi^{l}\) denote the angle between \(\mathbf{r}_{d}^{l}\) and \(\mathbf{r}^{l}(t)\). Then, we follow [33] to use cosine similarity between \(\mathbf{r}_{d}^{l}\) and \(\mathbf{r}^{l}(t)\), i.e., \(\cos(\phi^{l})\), for the measurement of the performance of SNN by \(t\). Actually, [33] shows that the cosine similarity between full precision and quantised neural network has a high correlation with the final accuracy of the quantised neural network. Similarly, we expect that higher cosine similarity between \(r^{l}(t)\) and \(r_{d}^{l}\) can result in less accuracy drop by \(t\).

The following theorem states that, for any \(t\), the performance of the SNN is of negative correlation with threshold \(V_{thr}^{l}\), and positive correlation with \(L_{2}\) norm over \(\mathbf{a}^{l}\), as stated in the below theorem.

**Theorem IV.1**: _For any inference time, assuming that the residual spiking rate \(\mathbf{\Delta}^{l}(t)\) is independent from \(\mathbf{r}_{d}^{l}\), the cosine similarity between \(\mathbf{r}_{d}^{l}\) and \(\mathbf{r}^{l}(t)\) is inversely proportional to the ratio of threshold to average accumulated current,_

\[\cos(\phi^{l})\propto\Big{(}\sqrt{n^{l}}\frac{V_{thr}^{l}}{\|\mathbf{a}^{l}\|_{2}} \Big{)}^{-1}\]

_where \(n^{l}\) is the dimension of \(\mathbf{a}^{l}\) and \(\|\mathbf{a}^{l}\|_{2}/\sqrt{n^{l}}\) denotes the average accumulated current._

We give a proof sketch of the theorem. Because \(\Delta^{l}(t)\) is independent from \(\mathbf{r}_{d}^{l}\), the angle between these two vectors tends to be \(\pi/2\) at high dimension. Then, by \(\mathbf{r}^{l}(t)=\mathbf{r}_{d}^{l}-\mathbf{\Delta}^{l}(t)\), we get a right angle triangle with \(\mathbf{r}_{d}^{l}\) and \(\Delta^{l}(t)\) as the legs, and \(\mathbf{r}^{l}(t)\) as the hypotenuse, as illustrated in Fig. 3. Moreover, we have

\[\cos(\phi^{l})=\frac{\|\mathbf{r}_{d}^{l}\|_{2}}{\|\mathbf{r}^{l}(t)\|_{2}}\geq\frac{\| \mathbf{r}_{d}^{l}\|_{2}}{\|\mathbf{r}_{d}^{l}\|_{2}+\|\mathbf{\Delta}^{l}(t)\|_{2}} \tag{9}\]

We are interested in increasing the lower bound of Equation 9, so that we have greater \(\cos(\phi^{l})\) for different \(t\). Combining with Equations (6) and (8), we have

\[\begin{split}\cos(\phi^{l})\geq\frac{\|\mathbf{a}^{l}/V_{thr}^{l}\|_{ 2}}{\|\mathbf{a}^{l}/V_{thr}^{l}\|_{2}+\|\mathbf{V}^{l}(t)/(tS_{r}V_{thr}^{l})\|_{2}}\\ =\frac{\|\mathbf{a}^{l}\|_{2}}{\|\mathbf{a}^{l}\|_{2}+\|\mathbf{V}^{l}(t)/(tS_{ r})\|_{2}}\end{split} \tag{10}\]

Fig. 3: Graphic illustration of the desired spiking rate \(\mathbf{r}_{d}^{l}\) and spiking rate \(\mathbf{r}^{l}(t)\)

Fig. 2: Inference in integrate-and-fire (IF) neuron with _reset by subtraction_ mechanism.
Assuming that elements in \(\mathbf{V}^{l}(t)\) satisfy uniform distribution over the time \(t\) and they are in \([0,V_{thr}]\), we can derive \(\mathbb{E}(\|V(t)/(tS_{r})\|_{2})\leq\sqrt{n^{l}}V_{thr}/(\sqrt{3}tS_{r})\) (proof in Appendix A). Moreover, at high dimensions, the relative error made as considering \(\mathbb{E}(\|V(t)/(tS_{r})\|_{2})\) instead of the random variable \(\|V(t)/(tS_{r})\|_{2}\) becomes asymptotically negligible [34, 33]. Therefore, Equation 10 can be computed with the following lower bound

\[\begin{split}\cos(\phi^{l})&\geq\frac{\|\mathbf{a}^{l} \|_{2}}{\|\mathbf{a}^{l}\|_{2}+\sqrt{n^{l}}V_{thr}^{l}/(\sqrt{3}tS_{r})}\\ &\quad=\frac{\sqrt{3}tS_{r}}{\sqrt{3}tS_{r}+\sqrt{n^{l}}V_{thr}^{ l}/\|\mathbf{a}^{l}\|_{2}}\end{split} \tag{11}\]

which explicitly explains that (1) the increase of \(t\) to \(t\gg\sqrt{n^{l}}V_{thr}^{l}/\|\mathbf{a}^{l}\|_{2}\) can increase the lower bound and (2) it is possible to minimise term \(\sqrt{n^{l}}V_{thr}^{l}/\|\mathbf{a}^{l}\|_{2}\) for developing an SNN with optimised performance at any time during the inference. In other words, an AOI-SNN expects a good (small) ratio of threshold voltage \(V_{thr}^{l}\) to average accumulated current, i.e., \(\|\mathbf{a}^{l}\|_{2}/\sqrt{n^{l}}\), while not degrading SNN classification performance. The point (2) corresponds with the theorem.

### _Regulariser for outlier elimination (ROE)_

This section shows how to design a regulariser based on Theorem IV.1. Recall from Equation (7) that \(V_{thr}^{l}\) is determined by \(\lambda^{l}\) and \(\lambda^{l-1}\), where \(\lambda^{l}\) is the maximum value of activation in the \(l\)-th layer. To simplify the complexity of optimisation, the impact of \(1/\lambda^{l-1}\) is omitted and the ratio of threshold to expected current approximately becomes proportional to \(\lambda^{l}/\|\mathbf{a}^{l}\|_{2}\). Therefore, we design a regulariser to minimise term \(\lambda^{l}/\|\mathbf{a}^{l}\|_{2}\) to develop an AOI-SNN. Firstly, we use matrix \(\mathbf{A}^{l}\) to represent a batch of \(\mathbf{a}^{l}\) during training. Secondly, we simply use maximum value in \(\mathbf{A}^{l}\) to approximate \(\lambda^{l}\), i.e., \(\lambda^{l}\approx\|\mathbf{A}^{l}\|_{\max}\). Then, we write \(\|\mathbf{A}^{l}\|_{2,q}=(\sum_{j}(\sum_{i}A_{ij}^{2})^{q/2})^{1/q}\) to denotes the \(L_{2,p}\) over \(\mathbf{A}^{l}\), where \(A_{ij}^{l}\) presents \(j\)-th \(\mathbf{a}_{i}^{l}\) in the batch and \(q\in\mathbb{Z}\). Finally, we can let the penalty term be the ratio between \(\|\mathbf{A}^{l}\|_{\max}\) and \(\|\mathbf{A}^{l}\|_{2,q}\) with scale constant \(\sqrt{n^{l}}\), i.e.,

\[R(\mathbf{A}^{l})=\sqrt{n^{l}}\frac{\|\mathbf{A}^{l}\|_{\max}}{\|(\mathbf{A}^{l})\|_{2,q}} \tag{12}\]

We let \(q\) be \(-\infty\) so that the penalty term can focus on the inputs with relatively small accumulated current in the batch. The final training objective is

\[L_{TT}+\alpha\sum_{l}\ln{(R(\mathbf{A}^{l}))} \tag{13}\]

where \(\alpha\) is a hyper-parameter to balance two loss terms. Logarithm is applied to reduce the impact from extremely large value. The regularisation-based training is to train an ANN based on \(\mathbf{\bar{X}}_{f}\) resulting in an SNN, then SNN operates with the event-based input (Equation 3). A small \(R(\mathbf{A})^{l}\) implies that it is less possible for \(\lambda^{l}\) to be an outlier and \(\|\mathbf{a}^{l}\|_{2}\) is generally large.

### _Cutoff mechanism to reduce inference time_

Thanks to the asynchronous working mechanism, event-driven SNNs can predict when only part of the spike train is processed. Nevertheless, a naive cutoff on the length of spike train (or the sampling time of event sensor) can easily result in accuracy loss. In this section, we suggest a principled method to determine the inference time. Technically, a new metric, called confidence rate and denoted as \(C(\hat{t},D\{S_{gap}>\beta\})\), is defined based on the statistical characteristics of processing a set \(D\) of inputs with respect to the discrete inference time \(\hat{t}\) and \(S_{gap}\). \(S_{gap}>\beta\) operates as a condition to identify the samples in \(D\) that are suitable for cutoff. Actually, we are able to plot a curve of confidence rate \(C(\hat{t},D\{S_{gap}>\beta\})\) with respect to the time \(\hat{t}\) and \(\beta\), respectively. During the processing of an individual input \(\mathbf{X}\), we will monitor another variable \(S_{gap}\), and once it is able to ensure the confidence rate can reach certain degree, an early cutoff signal can be sent (see Fig. 1). The following provides the details.

We write \(\mathbf{X}[\hat{t}]=\sum_{t=0}^{\hat{t}}\mathbf{X}(t)\) to denote the accumulation of \(\mathbf{X}(t)\) from 0 up to \(\hat{t}\). Then, we let \(f(\mathbf{X}[\hat{t}])\) return the prediction of \(f\) based on the partial input \(\mathbf{X}[\hat{t}]\). Based on this, we define a function

\[g(\mathbf{X})=\arg\min_{\hat{t}}\{\forall\hat{t}_{1}>\hat{t}:\mathbf{1}(f(\mathbf{X}[\hat{t }_{1}])=\mathbf{y})\} \tag{14}\]

to express the earliest time from which the model \(f\) is able to confidently and correctly classify according to the partial input. \(\mathbf{1}(\cdot)\) is the indicator function, i.e., \(\mathbf{1}(x_{1}=x_{2})=1\) and \(\mathbf{1}(x_{1}\neq x_{2})=0\). \(\mathbf{1}(f(\mathbf{X}[\hat{t}_{1}])=\mathbf{y})\) suggests that \(f(\mathbf{X}[\hat{t}])\) is the same as the ground truth \(\mathbf{y}\). Then, recall that \(\mathbf{N}^{L}(t)\) is the number of spikes received by \(t\) by the output layer \(L\). We write \(Top_{k}(\mathbf{N}^{L}(t))\) as the top \(k\) spikes that occur in some neuron of layer \(L\). Then, we let

\[S_{gap}=Top_{1}(\mathbf{N}^{L}(t))-Top_{2}(\mathbf{N}^{L}(t)) \tag{15}\]

be a variable denoting the gap of top-1 and top-2 number of spikes. A large \(S_{gap}\) implies little possibility of switching the prediction results during inference. Then, we let \(D\{\cdot\}\) denote the inputs in subset of \(D\) that satisfy a certain condition. Now, we can define the confidence rate as follows:

_Confidence rate:_ \[C(\hat{t},D\{S_{gap}>\beta\})=\] \[\frac{1}{|D\{S_{gap}>\beta\}|}\sum_{\mathbf{X}\in D\{S_{gap}>\beta\}}(g( \mathbf{X})\leq\hat{t})\]

which intuitively computes the percentage of inputs in \(D\) that can achieve the prediction success on or before a prespecified time \(\hat{t}\), i.e., \(g(\mathbf{X})\leq\hat{t}\). \(|D\{S_{gap}>\beta\}|\) denotes the number of samples in \(D\) satisfying the condition. It is not hard to see that, when \(\hat{t}=0\), \(C(\hat{t},D\{S_{gap}>\beta\})\) is also \(0\), and with the increase of time \(\hat{t}\), \(C(\hat{t},D\{S_{gap}>\beta\})\) will also increase until reaching \(1\). Our algorithm searches for a minimum \(\beta\in\mathbb{Z}^{+}\) at a specific \(\hat{t}\), as expressed in the following optimisation objective:

\[\arg\min_{\beta}C(\hat{t},D\{S_{gap}>\beta\})\geq 1-\epsilon \tag{16}\]

where \(\epsilon\) is a pre-specified constant such that \(1-\epsilon\) represents an acceptable level of confidence for activating cut-off, and a set of \(\beta\) is extracted under different \(\hat{t}\) using training samples.
Equation IV-C is visualised in Fig. 4 that shows the impact of inference time and \(\beta\) on confidence. Time ratio denotes the normalised inference time. We characterise the confidence metric with training samples and eventually use testing samples for evaluation. Note that, on Cifar10-DVS, all model achieves 100% for training accuracy, however, they perform differently on confidence. With regularisation, SNN-ROE can further improve the confidence than SNN-QA, e.g., it is 0.01 higher at \(0.125T_{total}\) and 0.07 higher at \(0.25T_{total}\). Therefore, SNN-ROE can have a better performance at any time during the inference, as there are more inputs join the early cutoff. Fig. 3(b) presents that the input with large \(S_{gap}\) has more consistent prediction over time, which supports the use of \(S_{gap}>\beta\) as the cutoff condition.

## V Experiment

We implement the ROE and conduct an extensive set of experiments to validate it. We consider its comparison with the state-of-the-art CNN-to-SNN conversion methods. In this section, 'SNN-QA' denotes the method in [13], which includes both COE and QA during training, and outperforms the other methods on image input. In contrast, 'SNN-COE' denotes the SNN with only COE. Our proposed method is denoted by 'SNN-ROE'. To reduce the accuracy loss during inference, we followed [18, 13] to add extra current \(V_{thr}^{l}/2\) to each neuron.

Our method is validated against three event-based datasets, e.g., Cifar10-DVS [35], N-Caltech101 [36] and DVS128 Gesture [9]. We train the neural network using Tensorflow with Keras API and convert it into SNN by SpKeras [18]. Our work is publicly available 1. As [13] did not cover the event-based input, we replicate their work as our baseline for comparison and set the quantisation length of SNN-QA to 16 for all datasets, which yields optimal performance. Note that, we use original input from DVS camera without any pre-processing for the inference so that SNN can remain asynchronous to the input events.

Footnote 1: [https://github.com/Dengyu-Wu/SNN-Regularisation-Cutoff](https://github.com/Dengyu-Wu/SNN-Regularisation-Cutoff)

### _Experiment setup_

The network architectures for difference datasets are given in Table II, which are modified from VGG-11 [37] for Cifar10-DVS & N-Caltech101 and VGG-like structure [22] for DVS128 Gesture.

Batch Normalisation [38] is applied after each convolutional and fully-connected layer to accelerate the convergence of ANN training. For all experiment, the learning rate is set to 0.1 and decays to zero after 300 epochs based on cosine decay schedule [39]. Weight decay is set to 0.0005. We set \(\alpha\) to 0.003 for the regulariser proposed in Section 4.2 and use pixel shifting as the data augmentation for all models, i.e., both width and height are randomly shifted by the range [-20%,20%]. Dropout is applied after fully-connected layer for DVS128 Gesture to improve the training and the dropout rate is 0.2. We set the batch size to 128 for \(F=1\) and 32 for \(F>1\) to reduce memory consumption.

For SNN training, we inherit the conversion methods and most of the notation from [18, 13]. Particularly, the relationship between ReLU and IF is from [18] and the relationship between quantised ReLU and IF is from [13]. Moreover, ROE minimisation is based on [18] and operates as a penalty term during ANN training.

### _Event-based datasets_

The samples in the event-based datasets record the event addresses with on/off events over a period of time. For Cifar10-DVS, it consists of 10,000 samples extracted from Cifar10 [16]. Each sample has 128\(\times\)128 spatial resolution. The length of each spike train is less equal to \(1.3s\). For N-Caltech101, it has 8709 samples categorised into 101 classes. The number of samples in each class ranges from 31 to 800. The length of each spike train is about \(0.3s\). The width in x-direction does not exceed 240 pixels and in y-direction does

\begin{table}
\begin{tabular}{c c} \hline \hline Dataset & Network Architecture \\ \hline Cifar10-DVS & C64k84-C64-C128-C2562-C256-C51282 \\ N-Caltech101 & -C512-C512s2-C512-AP2-FC512-(10 or 101) \\ \hline DVS128 Gesture & C128k84-[C128-MP2]* -FC512-FC128-(11) \\ \hline \hline \end{tabular}
\end{table} TABLE II: Network architectures for difference datasets. C64k8s4 represents the convolutional layer with \(filters=64\), \(kernel\ size=4\) and \(strides=4\). The default values of Kernel size and strides are 3 and 1 respectively. AP2 is the average pooling layer, MP2 is the max pooling layer with \(kernel\ size=2\) and FC is the fully-connected layer.

Fig. 4: Evaluation of confidence on Cifar10-DVS
not exceed 180 pixels. For this two datasets, we use 90% samples in each class for training and 10% for testing. DVS128 Gesture consists of 1341 samples with 11 categories. Each sample is repetitive over \(6.0s\).

### _applying temporal training in ANN_

Similar to the direct training in [22], the temporal training in ANN, shown in Fig. 5, reshapes the temporal frames before forwarding them into the neural network and computes average loss after multiple outputs for optimisation. However, temporal training uses ReLU as the activation function and has no iterative operation during forward propagation. Although it ignores the correlation between the neighbouring frames in hidden layers, our experiment shows that SNN still can achieve good performance. Normally, iterative operation can be expensive when the number of iteration is large, i.e., large memory required [22]. Fig. 6 presents that the increase of F can improve the accuracy on DVS128 Gesture, while it has little effect on Cifar10-DVS. We did not examine \(F\) in N-Calch101 due to its large size.

Moreover, since temporal training treats consecutive frames as individual frames and generates most spike for the prediction, regulariser and cutoff can be directly deployed when \(F>1\),

### _Experimental results_

This section presents a comparison between SNN-ROE, SNN-QA and SNN-COE on accuracy w.r.t time ratio and performance improvement after cutoff. The inference time \(t=\text{Time Ratio}\times T_{total}\), where \(T_{total}\) is equal to 1.3\(s\), 0.3\(s\) and 1.2\(s\) for Cifar10-DVS, N-Caltech101 and DVS128 Gesture respectively.

It is not hard to see, with cutoff, the performance of all models is improved in Fig. 7(a, b, c). For example, the accuracy curve moves above its original curve, which means that same accuracy can have less inference time for same model. Thanks to the increase of confidence (recall the results in Fig. 4a), SNN-ROE can have general higher accuracy before the time point (red dash line) and it shows consistent results in different datasets. The confidence evaluation for N-Caltech101 and DVS128 Gesture is provided in Fig. A-1. It has been argued in [24] that the temporal information in Cifar10-DVS is not the dominant information, which is similar in N-Caltech101. Unlike N-Caltech101 and Cifar10-DVS, the correlation between the temporal events in DVS128 Gesture is high. This phenomenon can also be observed from the Fig. 5. Therefore, we set \(F=1\) on N-Caltech101 and Cifar10-DVS for efficient training and \(F=4\) on DVS128 Gesture to incorporate temporal information. Moreover, the result in Fig. 7c shows that cutoff also can improve SNN trained with \(F>1\). Fig. 7d presents that the time ratio becomes adaptive after applying cutoff and regulariser can generally increase the cutoff performance with more accurate predictions at early inference time.

We collect the results around the time point in Table III, which has similar inference time, to show the performance of each model on cutoff. SNN-ROE achieves the superior performance on accuracy and latency. Spiking resolution \(S_{r}\) is calculated to estimate maximum average spikes per second.

## VI Conclusions

This paper promotes anytime optimal inference SNNs (AOISNNs), which maintain the optimal performance throughout the inference stage, and therefore are suitable for event-driven inputs such as those from dynamic vision sensor or dynamic audio sensor. Two technical novelties are proposed to optimise the attainment of AOI-SNNs, one for the training stage and the other for the inference stage. Our experiments demonstrate the superior performance with respect to the accuracy and latency, comparing to the state-of-the-art.
