Adaptive Axonal Delays in feedforward spiking neural networks for
  accurate spoken word recognition
Spiking neural networks (SNN) are a promising research avenue for building
accurate and efficient automatic speech recognition systems. Recent advances in
audio-to-spike encoding and training algorithms enable SNN to be applied in
practical tasks. Biologically-inspired SNN communicates using sparse
asynchronous events. Therefore, spike-timing is critical to SNN performance. In
this aspect, most works focus on training synaptic weights and few have
considered delays in event transmission, namely axonal delay. In this work, we
consider a learnable axonal delay capped at a maximum value, which can be
adapted according to the axonal delay distribution in each network layer. We
show that our proposed method achieves the best classification results reported
on the SHD dataset (92.45%) and NTIDIGITS dataset (95.09%). Our work
illustrates the potential of training axonal delays for tasks with complex
temporal structures.
Pengfei Sun, Ehsan Eqlimi, Yansong Chua, Paul Devos, Dick Botteldooren
# Adaptive Axonal Delays in Feedforward Spiking Neural Networks for Accurate Spoken Word Recognition

###### Abstract

Spiking neural networks (SNN) are a promising research avenue for building accurate and efficient automatic speech recognition systems. Recent advances in audio-to-spike encoding and training algorithms enable SNN to be applied in practical tasks. Biologically-inspired SNN communicates using sparse asynchronous events. Therefore, spike-timing is critical to SNN performance. In this aspect, most works focus on training synaptic weights and few have considered delays in event transmission, namely axonal delay. In this work, we consider a learnable axonal delay capped at a maximum value, which can be adapted according to the axonal delay distribution in each network layer. We show that our proposed method achieves the best classification results reported on the SHD dataset (92.45%) and NTIDIGITS dataset (95.09%). Our work illustrates the potential of training axonal delays for tasks with complex temporal structures.

Pengfei Sun\({}^{1}\), Ehsan Eqlimi\({}^{1}\), Yansong Chua\({}^{2}\), Paul Devos\({}^{1}\), Dick Botteldooren\({}^{1}\)+\({}^{1}\)WAVES Research Group, Ghent University, Belgium

\({}^{2}\)China Nanhu Academy of Electronics and Information Technology, China

Footnote †: This research is supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO2003000).

+
Footnote †: This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO2003000).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO200300).

+
Footnote †: This research is also also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen”. This research is also supported by the National Key Research and Development Program of China (Grant No. 2021ZDO2003000).

+
Footnote †: This research is also supported by the Research Foundation - Flanders under grant number GOA0220N and the Flemish Government under the ”Onderzoeksprogramma Artificiele Intelligentie
where \(W_{ij}^{l-1}\) indicates the synaptic weight from neuron \(j\) to \(i\) at layer \(l-1\) and \(u_{i}^{l}\) refers to the membrane potential of neuron \(i\) in layer \(l\), while \(s_{j}^{l-1}\) is the incoming spike pattern from the preceding neuron \(j\). In this experiment, we use the response signal \(a(t)=(\epsilon*s^{J-1})(t)\) to describe the response of neurons by convolving input spikes \(s^{l-1}(t)\) with the response kernel \(\epsilon\), where \(\epsilon(t)=\frac{t}{\tau_{s}}\exp(1-\frac{t}{\tau_{s}})\Theta(t)\). Here, \(\Theta(t)\) represents the Heaviside step function. Likewise, the refractory signal can be described as \((\nu*s^{J})(t)\), where \(\nu(t)=-2\theta_{u}\,\frac{1}{\nu_{t}}\exp(1-\frac{t}{\tau_{s}})\Theta(t)\). Here, the parameter \(\tau_{s}\) and \(\tau_{s}\) are the time constant of the corresponding kernels. An output is generated whenever the \(u_{i}^{l}\) surpasses the pre-defined threshold \(\theta_{u}\). This spike-generation process can be formulated as

\[s_{i}^{l}(t)=\Theta(u_{i}^{l}(t)-\theta_{u}) \tag{2}\]

### Axonal delay module and adaptive delay caps

In Fig.1, the axonal delay module and an adaptive training scheduler for delay caps are shown. The axonal delay is part of spike transmission, and we formulate it such that it can be jointly learned with an optimization algorithm.

\[s_{d}^{l}(\hat{t})=\delta(t-d^{l})*s^{J}(t) \tag{3}\]

In layer \(l\), \(d^{l}\) represents the set of delays \(\{d_{1},d_{2},..,d_{n}\}\) subject to the constraint that \(\{d_{1}<d_{2},..,<d_{n}\leq\theta_{d}\}\). Meanwhile, \(s_{d}^{l}(\hat{t})\) denotes the spike trains output by the delay module at a shifted time \(\hat{t}\). From the optimization point of view, constraining the delay value can facilitate learning. Here, we compute the fraction of neurons against the total number of neurons within a sliding window [24] in the adaptive training scheduler so as to optimize training.

Consider the sliding window \(m\), when the fraction of delay neurons within this window exceeds the pre-defined cap fraction \(\alpha_{\theta}\), the sliding window right-shifts by 1, and the delay cap \(\theta_{d}\) also increases by 1. Pseudo-code of the proposed adaptive training scheduler is presented in Algorithm 1.

During training (in the second while loop), the delay will be clipped as follows

\[d=max(0,min(d,\theta_{d})) \tag{4}\]

where the \(\theta_{d}\) and delays \(d\) will be adaptively adjusted according to our scheduler.

## 3 Experimental Setup

### Datasets

We evaluate our proposed methods on the SHD [25] and NTIDIGITS [26] datasets. These datasets are the spike patterns processed by the artificial cochlear models[25, 26] from the original audio signals and have high temporal complexity [27]. The SHD dataset consists of 10,420 utterances of varying durations (0.24\(s\) to 1.17\(s\)) by 12 speakers. It contains a total of 20 digits classes from '0' to '9' in English and German language. We adopt the same data preprocessing method in [28], and the same train/validation/test split of the dataset as the official [25].

The NTIDIGITS is another event-based spoken recognition task. The original TIDIGITS is processd by the 64-channel CochleaAMS1b sensor and its output is recorded. In total, there are 11 spoken digits (the digits '0' to '9' in the

Figure 1: Illustration of how the adaptive delay caps are determined and the axonal delays adjusted. The generated spikes \(s^{1}(t)\) will be shifted in time by \(d_{i}\) and then output as spike trains \(s_{d}^{1}(\hat{t})\) in the axonal delay module. The adaptive scheduler will adjust the delay cap accordingly. The delay value may be the same across neurons, such as the top two neurons with the same delay value \(d_{1}\). The layer can be a traditional convolutional layer, dense layer, or recurrent layer.
English language and 'oh'). We follow the same train and test split as used in [26].

### Implementation details

In our experiment, we use the Adam optimizer to jointly update the synaptic weight and axonal delay with a constant learning rate of 0.1 and a minibatch size of 128. The initial caps for delay are set to 64 for SHD and 128 for NTIDIGITS, respectively. The pre-train epochs \(x\) are set as 40 and the simulation time step is 1 \(ms\) for both datasets. Table 1 lists the parameters we used in our experiments.

We train our SNN on the SLAYER-Pytorch framework [8]. This recently introduced GPU-accelerated software is publicly available and has proven to be effective for training SNN. For both tasks, we use an SNN with two hidden fully-connected layers with 128 hidden units for SHD and 256 neurons for NTIDIGITS. The total number of model parameters is approximately 0.11M for SHD dataset and 0.08M for NTIDIGITS dataset.

## 4 Results

### Ablation Study of Different Cap Fraction and Sliding Window Size

The window size and cap fraction are important parameters to get a reasonable delay cap. To evaluate their influence, we design an ablation study to compare the influence, which contains 5 different window sizes and 3 different fraction parameters. The impact of these parameters on SHD and NTIDIGITS datasets is shown in Fig 2. For the SHD dataset, we observe that a small fraction can always get good results, and the best results can be obtained by controlling the number of the largest two delayed neurons within 5% of the total number of neurons, which means the window size is set as 2 and the cap fraction is 5%. While for the NTIDIGITS dataset, the bigger window size and fraction are more helpful, and the accuracy keeps going up when the window size grows bigger besides the situation of window size 5 and cap fraction 10%.

### Shd

The result of performance provided by feed-forward SNN, recurrent spiking neural network (RSNN) with adaptive firing threshold, heterogeneous RSNN, RSNN with temporal attention, and our proposed method is given in Table 2. As can be seen from Table 2, our proposed method achieves an accuracy of 92.45%, which is the best performance reported for

\begin{table}
\begin{tabular}{c c c c c c} \hline
**Dataset** & \(\tau_{s}\) & \(\tau_{r}\) & **Initial**\(\theta_{d}\) & \(\theta_{u}\) & \(T_{steps}\) \\ \hline SHD & 1 & 1 & 64 & 10 & 150 \\ NTIDIGITS & 5 & 5 & 128 & 10 & 150 \\ \hline \end{tabular}
\end{table}
Table 1: Detailed parameters settings for different datasets

Figure 2: Ablation study of different sliding window sizes \(m\) and cap fractions \(\alpha_{\theta}\) based on (Top) SHD and (Bottom) N-TIDIGITS. The x-axis indicates the sliding window size, and the y-axis refers to the accuracy. There are 3 different cap fractions (5%, 10%, and 15%) and 5 different window sizes (1,2,3,4, and 5) we use in our experiment.
this task. More importantly, it can be observed that the RSNN outperforms the feed-forward SNN, which implies that the task is inherently dynamic. However, our results show that a feed-forward SNN, without recurrent connections, but with an adaptive axonal delay module, can achieve flexible short-term memory and better accuracy, using fewer parameters.

### Nitigits

When the ANN (RNN and Phased-LSTM) are applied to the digits recognition task, they achieve an accuracy of 90.90% and 91.25% (see Table 2), respectively. However, these networks cannot fully exploit the advantages of sparse event-based information and have to rely on the event synthesis algorithm, effectively losing the advantage gained when processing information embedded in spike-timing. Using our adaptive axonal delay module, we can achieve the best performance of 95.09%, and compared with Zhang et al.[30] which directly use the spike train level features, our model can improve 1.4% performance while only using 23% of parameters.

### Effect of the Delay Value

As shown in Table 3, the delay cap is important to the performance of the model. For both audio classification tasks, the performance is competitive without limiting the range of delay values, demonstrating the effectiveness of the axonal delay module. However, it is still necessary to limit the delay range, and our experiments show that an appropriate delay cap will improve the classification ability of the model. Taking the SHD dataset as an example, our adaptive training scheduler can determine the optimal delay distribution, and this distribution enables the model to achieve the best performance. For other combinations of delay caps, the obtained accuracy drops, may indicate that each network structure has an optimal delay cap, but it is difficult to find the delay caps manually. Our method provides an adaptive training scheduler to search for these optimal parameters. It is worth noting that the NTIDIGITS dataset conforms to this phenomenon.

## 5 Conclusions

In this paper, we integrate the learnable axonal delay module into the spiking neuron model and then introduce an adaptive training scheduler to adjust the caps of axonal delay in each network layer. Compared to previous work that adopts a static delay cap, our proposed method significantly improves the classification capability without extra parameters. Furthermore, our adaptive scheduler can be easily integrated into the existing delay module and determine the optimal delay distribution of the network adaptively. We achieve the best performance in SHD (92.45%) and NTIDIGITS (95.09%) datasets with the fewest parameters. These results suggest that a neuron axonal delay with an adaptive delay cap can be used to model a lightweight flexible short-term memory module so as to achieve an accurate and efficient spoken word recognition system. We conjecture that the axonal delay mechanism introduces a form of short-term memory without increasing the number of trainable parameters. For certain data-sets in ASR, whereby 1) information is organized in short sequences, without need for long-term memory, and 2), data is limited in size and hence prone to overfitting, the axonal delay mechanism may work best, in combination with a feed-forward, small SNN. Our experiments seem to agree with the above and further confirm the great potential of using spike timing as part of the solution to an ASR problem. Furthermore, the use of spike-based losses [31] can expedite decision-making, thereby reducing the impact of additional latency even more.

\begin{table}
\begin{tabular}{c c c c} \hline Datasets & **Method** & **Params** & **Accuracy** \\ \hline \multirow{6}{*}{SHD} & Feed-forward SNN [25] & 0.11 MB & \(48.6\pm 0.9\%\) \\  & RSNN [25] & 1.79 MB & \(83.2\pm 1.3\%\) \\  & RSNN with Adaption [28] & 0.14 MB & \(84.4\%\) \\  & Heterogeneous RSNN [20] & 0.11 MB & \(82.7\pm 0.8\%\) \\  & SNN with Time Attention [29] & 0.12 MB & \(91.08\%\) \\  & **This work (m=2, \(\alpha_{\theta}\) = 5\%)** & **0.11 MB** & \(\mathbf{92.45\%}\) \\ \hline \multirow{6}{*}{NTIDIGITS} & GRU-RNN [26]\({}^{\dagger}\) & 0.11 MB & \(90.90\%\) \\  & Phased-LSTM [26]\({}^{\dagger}\) & 0.61 MB & \(91.25\%\) \\ \cline{1-1}  & ST-RSBP [30] & 0.35 MB & \(93.63\pm 0.27\%\) \\ \cline{1-1}  & SNN with Axonal delay [23] & 0.08 MB & \(94.45\%\) \\ \cline{1-1}  & **This work (m=4, \(\alpha_{\theta}\) = 10\%)** & **0.08 MB** & \(\mathbf{95.09\%}\) \\ \hline \end{tabular} \({}^{\dagger}\) Non-SNN implementation.

\end{table}
Table 2: Comparison with the state-of-the-art in terms of network size and accuracy.

\begin{table}
\begin{tabular}{c c c c c} \hline
**Dataset** & **Method** & \((\theta_{d_{d}},\theta_{d_{d}})\) & **Params** & **Accuracy** \\ \hline \multirow{6}{*}{\(\mathbf{\Delta\theta}\)} & Manual & (0, 0) & 108,820 & 67.05\% \\  & Manual & (64, 64) & 109,076 & 86.84\% \\  & Manual & (128, 128) & 109,076 & 87.24\% \\  & Adaptive & (107, 175) & 109,076 & \(\mathbf{92.45\%}\) \\  & Manual & (\(+\infty,+\infty\)) & 109,076 & \(\mathbf{84.99\%}\) \\ \hline \multirow{6}{*}{\(\mathbf{\Delta\theta}\)} & Manual & (0, 0) & 85,259 & 78.86\% \\  & Manual & (128, 128) & 85,771 & 94.19\% \\ \cline{1-1}  & Adaptive & (215, 215) & 85,771 & \(\mathbf{95.09\%}\) \\ \cline{1-1}  & Manual & (\(+\infty,+\infty\)) & 85,771 & 93.83\% \\ \hline \end{tabular}
\end{table}
Table 3: Ablation studies for different delay cap methods and the effect of the delay cap \(\theta_{d}\) in the axonal delay module. \(\theta_{d_{d}}\) indicates the delay of \(\hat{r}^{th}\) layer. ’Manual’ refers to using a static delay cap, while ’Adaptive’ refers to using our proposed adaptive scheduler.