Learning Delays in Spiking Neural Networks using Dilated Convolutions
  with Learnable Spacings
Spiking Neural Networks (SNNs) are a promising research direction for
building power-efficient information processing systems, especially for
temporal tasks such as speech recognition. In SNNs, delays refer to the time
needed for one spike to travel from one neuron to another. These delays matter
because they influence the spike arrival times, and it is well-known that
spiking neurons respond more strongly to coincident input spikes. More
formally, it has been shown theoretically that plastic delays greatly increase
the expressivity in SNNs. Yet, efficient algorithms to learn these delays have
been lacking. Here, we propose a new discrete-time algorithm that addresses
this issue in deep feedforward SNNs using backpropagation, in an offline
manner. To simulate delays between consecutive layers, we use 1D convolutions
across time. The kernels contain only a few non-zero weights - one per synapse
- whose positions correspond to the delays. These positions are learned
together with the weights using the recently proposed Dilated Convolution with
Learnable Spacings (DCLS). We evaluated our method on three datasets: the
Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its
non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which
require detecting temporal patterns. We used feedforward SNNs with two or three
hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We
showed that fixed random delays help and that learning them helps even more.
Furthermore, our method outperformed the state-of-the-art in the three datasets
without using recurrent connections and with substantially fewer parameters.
Our work demonstrates the potential of delay learning in developing accurate
and precise models for temporal data processing. Our code is based on PyTorch /
SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays
Ilyass Hammouamri, Ismail Khalfaoui-Hassani, Timothée Masquelier
# Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings

###### Abstract

Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights - one per synapse - whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: [https://github.com/Thvnvtos/SNN-delays](https://github.com/Thvnvtos/SNN-delays)

## 1 Introduction

Spiking neurons are coincidence detectors [27; 33]: they respond more when receiving synchronous, rather than asynchronous, spikes. Importantly, it is the spike arrival times that should coincide, not the spike emitting times - these times are different because propagation is usually not instantaneous. There is a delay between spike emission and reception, called delay of connections, which can vary across connections. Thanks to these heterogeneous delays, neurons can detect complex spatiotemporal spike patterns, not just synchrony patterns [23] (see Figure 1).

In the brain, the delay of a connection corresponds to the sum of the axonal, synaptic, and dendritic delays. It can reach several tens of milliseconds, but it can also be much shorter (1 ms or less) [23]. For example, the axonal delay can be reduced with myelination, which is an adaptive process that is required to learn some tasks (see [4] for a review). In other words, learning in the brain can not be reduced to synaptic plasticity. Delay learning is also important.
A certain theoretical work has led to the same conclusion: Maass and Schmitt demonstrated, using simple spiking neuron models, that a SNN with k adjustable delays can compute a much richer class of functions than a threshold circuit with k adjustable weights [29].

Finally, on most neuromorphic chips, synapses have a programmable delay. This is the case for Intel Loihi [8], IBM TrueNorth [1], SpiNNaker [13] and SENeCA [45].

All these points have motivated us and others (see related works in the next section) to propose delay learning rules. Here we show for the first time that delays can be learned together with the weights, using backpropagation, in arbitrarily deep SNNs. The trick is to simulate delays using temporal convolutions and to learn them using the recently proposed Dilated Convolution with Learnable Spacings [24; 25]. In practice, the method is fully integrated with PyTorch and leverages its automatic-differentiation engine.

## 2 Related Work

### Deep Learning for Spiking Neural Networks

Recent advances in SNN training methods like the surrogate gradient method [30; 35] and the ANN2SNN conversion methods [5; 9; 19] made it possible to train increasingly deeper spiking neural networks. The surrogate gradient method defines a continuous relaxation of the non-smooth spiking nonlinearity: it replaces the gradient of the Heaviside function used in the spike generating process by a smooth surrogate gradient that is suitable for optimization. On the other hand, the ANN2SNN methods convert conventional artificial neural networks (ANNs) into SNNs by copying the weights from ANNs while trying to minimize the conversion error.

Other works have explored improving the spiking neurons using inspiration from biological mechanisms or techniques used in ANNs. The Parametric Leaky Integrate-and-Fire (PLIF) [12] incorporates learnable membrane time constants that could be trained jointly with synaptic weights.[18] proposes a method to dynamically adapt firing thresholds in order to improve continual learning in SNNs. Spike-Element-Wise ResNet [11] addresses the problem of vanishing/exploding gradient in the plain Spiking ResNet caused by sigmoid-like surrogate functions, and successfully trained the first deep SNN with more than 150 layers. Spikformer [48] adapts the softmax-based self-attention mechanism of Transformers [40] to a spike-based formulation.

These efforts have resulted in closing the gap between the performance of ANNs and SNNs on many widely used benchmarks.

Figure 1: Coincidence detection: we consider two neurons \(N1\) and \(N2\) with the same positive synaptic weight values. \(N2\) has a delayed synaptic connection denoted \(d_{21}\) of \(8\)ms, thus both spikes from spike train \(S1\) and \(S2\) will reach \(N2\) quasi-simultaneously. As a result, the membrane potential of \(N2\) will reach the threshold \(\vartheta\) and \(N2\) will emit a spike. On the other hand, \(N1\) will not react to these same input spike trains.
### Delays in SNNs

Few previous works considered learning delays in SNNs. [41] proposed a similar method to ours in which they convolve spike trains with an exponential kernel so that the gradient of the loss with respect to the delay can be calculated. However, their method is used only for a shallow SNN with no hidden layers.

Other methods like [16; 17; 47; 39] also proposed learning rules developed specifically for shallow SNNs with only one layer. [21] proposed to learn temporal delays with Spike Timing Dependent Plasticity (STDP) in weightless SNNs. [20] proposed a method for delay-weight supervised learning in optical spiking neural networks. [2] proposed a method for deep feedforward SNNs that uses a set of multiple fixed delayed synaptic connections for the same two neurons before pruning them depending on the magnitude of the learned weights.

To the best of our knowledge, [38; 37] are the only ones to learn delays and weights jointly in a deep SNN. They proposed an adaptive maximum delay value that depends on the distribution of delays on each network layer. However, they use finite difference approximation to numerically estimate the gradients of the spikes in respect to the delays, and we think that those gradients are not suitable as we achieve similar performance in our experiments with fixed random delays.

We propose a control test that was not considered by the previous works and that we deem necessary: the SNN with delay learning should outperform an equivalent SNN with fixed random and uniformly distributed delays, especially with sparse connectivity.

## 3 Methods

### Spiking Neuron Model

The spiking neuron, which is the fundamental building block of SNNs, can be simulated using various models. In this work, we use the Leaky Integrate-and-Fire model [14], which is the most widely used for its simplicity and efficiency. The membrane potential \(u_{i}^{(l)}\) of the \(i\)-th neuron in layer \(l\) follows the differential equation:

\[\tau\frac{du_{i}^{(l)}}{dt}=-(u_{i}^{(l)}(t)-u_{reset})+RI_{i}^{(l)}(t) \tag{1}\]

where \(\tau\) is the membrane time constant, \(u_{reset}\) the potential at rest, \(R\) the input resistance and \(I_{i}^{(l)}(t)\) the input current of the neuron at time \(t\). In addition to the sub-threshold dynamics, a neuron emits a unitary spike \(S_{i}^{(l)}\) when its membrane potential exceeds the threshold \(\vartheta\), after which it is instantaneously reset to \(u_{reset}\). Finally, the input current \(I_{i}^{(l)}(t)\) is stateless and represented as the sum of afferent weights \(W_{ij}^{(l)}\) multiplied by spikes \(S_{j}^{(l-1)}(t)\):

\[I_{i}^{(l)}(t)=\sum_{j}W_{ij}^{(l)}S_{j}^{(l-1)}(t) \tag{2}\]

We formulate the above equations in discrete time using Euler's method approximation, and using \(u_{reset}=0\) and \(R=\tau\).

\[u_{i}^{(l)}[t] =(1-\frac{1}{\tau})u_{i}^{(l)}[t-1]+I_{i}^{(l)}[t] \tag{3}\] \[I_{i}^{(l)}[t] =\sum_{j}W_{ij}^{(l)}S_{j}^{(l-1)}[t]\] (4) \[S_{i}^{(l)}[t] =\Theta(u_{i}^{l}[t]-\vartheta) \tag{5}\]

We use the surrogate gradient method [30] and define \(\Theta^{\prime}(x)\triangleq\sigma^{\prime}(x)\) during the backward step, where \(\sigma(x)\) is the surrogate arctangent function [12].
### Synaptic Delays as a Temporal Convolution

A feed-forward SNN model with delays is parameterized with \(W=(w_{ij}^{(l)})\in\mathbb{R}\) and \(D=(d_{ij}^{(l)})\in\mathbb{R}^{+}\), where the input of neuron \(i\) at layer \(l\) is

\[I_{i}^{(l)}[t]=\sum_{j}w_{ij}^{(l)}S_{j}^{(l-1)}[t-d_{ij}^{(l)}] \tag{6}\]

We model a synaptic connection from neuron \(j\) in layer \(l-1\) to neuron \(i\) in layer \(l\) which have a synaptic weight \(w_{ij}^{(l)}\) and delay \(d_{ij}^{(l)}\) as a one dimensional temporal convolution (see Figure 2) with kernel \(k_{ij}^{(l)}\) as follows:

\(\forall n\in\llbracket 0,...\ T_{d}-1\rrbracket\):

\[k_{ij}^{(l)}[n]=\begin{cases}w_{ij}^{(l)}&\text{if }n=T_{d}-d_{ij}^{(l)}-1\\ 0&\text{otherwise}\end{cases} \tag{7}\]

where \(T_{d}\) is the kernel size or maximum delay + 1. Thus we redefine the input \(I_{i}^{(l)}\) in Equation 6 as a sum of convolutions:

\[I_{i}^{(l)}=\sum_{j}k_{ij}^{(l)}*S_{j}^{(l-1)} \tag{8}\]

We used a zero left-padding with size \(T_{d}-1\) on the input spike trains \(S\) so that \(I[0]\) does correspond to \(t=0\).

Moreover, a zero right-padding could also be used, but it is optional; it could increase the expressivity of the learned delays with the drawback of increasing the processing time as the number of time-steps after the convolution will increase.

To learn the kernel elements positions (i.e., delays), we use the 1D version of DCLS [24] with a Gaussian kernel [25] centered at \(T_{d}-d_{ij}^{(l)}-1\), where \(d_{ij}^{(l)}\in\llbracket 0,\ T_{d}-1\rrbracket\), and of standard deviation \(\sigma_{ij}^{(l)}\in\mathbb{R}^{*}\), thus we have:

\(\forall n\in\llbracket 0,...\ T_{d}-1\rrbracket\):

\[k_{ij}^{(l)}[n]=\frac{w_{ij}^{(l)}}{c}\ \text{exp}\left(-\frac{1}{2}\left( \frac{n-T_{d}+d_{ij}^{(l)}+1}{\sigma_{ij}^{(l)}}\right)^{2}\right) \tag{9}\]

Figure 2: Example of one neuron with 2 afferent synaptic connections, convolving \(K1\) and \(K2\) with the zero left-padded \(S_{1}\) and \(S_{2}\) is equivalent to following Equation 6
With

\[c=\epsilon+\sum_{n=0}^{T_{d}-1}\text{exp}\left(-\frac{1}{2}\left(\frac{n-T_{d}+d_{ ij}^{(l)}+1}{\sigma_{ij}^{(l)}}\right)^{2}\right) \tag{10}\]

a normalization term and \(\epsilon=1e-7\) to avoid division by zero, assuming that the tensors are in float32 precision. During training, \(d_{ij}^{(l)}\) are clamped after every batch to ensure their value stays in \([0,...\,T_{d}-1]\).

The learnable parameters of the 1D DCLS layer with Gaussian interpolation are the weights \(w_{ij}\), the corresponding delays \(d_{ij}\), and the standard deviations \(\sigma_{ij}\). However, in our case, \(\sigma_{ij}\) are not learned, and all kernels in our model share the same decreasing standard deviation, which will be denoted as \(\sigma\). Throughout training, we exponentially decrease \(\sigma\) as our end goal is to have a sparse kernel where only the delay position is non-zero and corresponds to the weight.

The Gaussian kernel transforms the discrete positions of the delays into a smoother kernel (see Figure 3), which enables the calculation of the gradients \(\frac{\partial L}{\partial d_{ij}^{(l)}}\).

By adjusting the parameter \(\sigma\), we can regulate the temporal scale of the dependencies. A small value for \(\sigma\) enables the capturing of variations that occur within a brief time frame. In contrast, a larger value of \(\sigma\) facilitates the detection of temporal dependencies that extend over longer durations. Thus \(\sigma\) tuning is crucial to the trade-off between short-term precision and long-term dependencies.

We start with a high \(\sigma\) value and exponentially reduce it throughout the training process, after each epoch, until it reaches its minimum value of 0.5 (Fig. 4). This approach facilitates the learning of distant long-term dependencies at the initial time. Subsequently, when \(\sigma\) has a smaller value, it enables refining both weights and delays with more precision, making the Gaussian kernel more similar to the discrete kernel that is used at inference time. As we will see later in our ablation study (Section 4.3), this approach outperforms a constant \(\sigma\).

Indeed, the Gaussian kernel is only used to train the model; when evaluating on the validation or test set, it is converted to a discrete kernel as described in Equation 7 by rounding the delays. This permits to implement sparse kernels for inference which are very useful for uses on neuromorphic

Figure 3: Gaussian convolution kernels for \(N\) synaptic connections. The Gaussians are centered on the delay positions, and the area under their curves corresponds to the synaptic weights \(w_{i}\). On the right, we see the delayed spike trains after being convolved with the kernels. (the \(-1\) was omitted for figure clarity).
hardware, for example, as they correspond to only one synapse between pairs of neurons, with the corresponding weight and delay.

## 4 Experiments

### Experimental Setup

We chose to evaluate our method on the SHD and SSC/GSC datasets [6], as they require leveraging temporal patterns of spike times to achieve a good classification accuracy, unlike most computer vision spiking benchmarks. Both spiking datasets are constructed using artificial cochlear models to convert audio speech data to spikes; the original audio datasets are the Heidelberg Dataset (HD) and the Google Speech Commands v0.02 Dataset (SC) [42] for SHD and SSC, respectively.

The SHD dataset consists of 10k recordings of 20 different classes that consist of spoken digits ranging from zero to nine in both English and German languages. SSC and GSC are much larger datasets that consist of 100k different recordings. The task we consider on SSC and GSC is the top one classification on all 35 different classes (similar to [6; 3]) which is more challenging than the original key-word spotting task on 12 classes, proposed in [42].

For the two spiking datasets, we used spatio-temporal bins to reduce the input dimensions. Input neurons were reduced from 700 to 140 by binning every 5 neurons; as for the temporal dimension we used a discrete time-step \(\Delta t=10\) ms, and a zero right-padding to make sure all recordings in a batch have the same time duration. As for the non-spiking GSC, we used the Mel Spectrogram representation of the waveforms with 140 frequency bins and approximately 100 timesteps to remain consistent to the input sizes used in SSC.

Figure 4: This figure illustrates the evolution of the same delay kernels for an example of eight synaptic connections of one neuron throughout the training process. The x-axis corresponds to time, and each kernel is of size \(T_{d}=25\). And the y-axis is the synapse id. (a) corresponds to the initial phase where the standard deviation of the Gaussian \(\sigma\) is large (\(\frac{T_{d}}{2}\)), allowing to take into consideration long temporal dependencies. (b) corresponds to the intermediate phase, (c) is taken from the final phase where \(\sigma\) is at its minimum value (0.5) and weight tuning is more emphasized. Finally, (d) represents the kernel after converting to the discrete form with rounded positions.
We used a very simple architecture: a feedforward SNN with two or three hidden fully connected layers. Each feedforward layer is implemented using a DCLS module where each synaptic connection is modeled as a 1D temporal convolution with one Gaussian kernel element (as described in Section 3.2), followed by batch normalization, a LIF module (as described in Section 3.1) and dropout. Table 1 lists the values of some hyperparameters used for the three datasets (for more details, refer to the code repository).

The readout layer consists of \(n_{\text{classes}}\) LIF neurons with infinite threshold (where \(n_{\text{classes}}\) is 20 or 35 for SHD and SSC/GSC respectively). Similar to [3], the output \(\text{out}_{i}[t]\) for every neuron \(i\) at time \(t\) is

\[\text{out}_{i}[t]=\text{softmax}(u_{i}^{(r)}[t])=\frac{e^{u_{i}^{(r)}[t]}}{ \sum_{j=1}^{n_{\text{classes}}}e^{u_{j}^{(r)}[t]}} \tag{11}\]

where \(u_{i}^{(r)}[t]\) is the membrane potential of neuron \(i\) in the readout layer \(r\) at time \(t\).

And the final output of the model after \(T\) time-steps is defined as

\[\hat{y_{i}}=\sum_{t=1}^{T}\text{out}_{i}[t] \tag{12}\]

We denote the batch size by \(N\) and the ground truth by \(y\). We calculate the cross-entropy loss for one batch as

\[\mathcal{L}=\frac{1}{N}\sum_{n=1}^{N}-\log(\text{softmax}(\hat{y}_{y_{n}}[n])) \tag{13}\]

The Adam optimizer [26] is used for all models and groups of parameters with base learning rates \(lr_{w}=0.001\) for synaptic weights and \(lr_{d}=0.1\) for delays. We used a one-cycle learning rate scheduler [36] for the weights and cosine annealing [28] without restarts for the delays learning rates. Our work is implemented1 using the PyTorch-based SpikingJelly[10] framework.

Footnote 1: Our code is available at: [https://github.com/Thvnvtos/SNN-delays](https://github.com/Thvnvtos/SNN-delays)

### Results

We compare our method in Table 2 to previous works on the SHD, SSC and GSC-35 (35 denoting the 35 classes harder version) benchmark datasets in terms of accuracy, model size and whether recurrent connections or delays were used.

The reported accuracy of our method corresponds to the accuracy on the test set using the best performing model on the validation set. However, since there is no validation set provided for SHD we use the test set as the validation set (similar to [3]). The margins of error are calculated at a 95% confidence level using a t-distribution (we performed ten and five experiments using different random seeds for SHD and SSC/GSC respectively).

Our method outperforms the previous state-of-the-art accuracy on the three benchmarks (with a significant improvement on SSC and GSC) without using recurrent connections and with a substantially lower number of parameters, and using only vanilla LIF neurons. Other methods that use delays do have a slightly lower number of parameters than we do, yet we outperform them significantly on SHD; while they didn't report any results on the harder benchmarks SSC/GSC. Finally, by increasing the number of hidden layers, we found that the accuracy plateaued after two hidden layers for SHD, and three for SSC/GSC.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Dataset & \# Hidden Layers & \# Hidden size & \(\tau\)(ms) & Maximum Delay(ms) & Dropout rate \\ \hline SHD & 2 & 256 & \(10.05^{*}\) & 250 & 0.4 \\ SSC/GSC & 2 or 3 & 512 & 15 & 300 & 0.25 \\ \hline \hline \end{tabular}

*We found that a LIF with quasi-instaneous leak \(\tau=10.05\) (since \(\Delta t=10\)) is better than using a Heaviside function for SHD.

\end{table}
Table 1: Network parameters for different datasets
### Ablation study

In this section, we conduct control experiments aimed at assessing the effectiveness of our delay learning method. The model trained using our full method will be referred to as _Decreasing_\(\sigma\), while _Constant_\(\sigma\) will refer to a model where the standard deviation \(\sigma\) is constant and equal to the minimum value of \(0.5\) throughout the training. Additionally, _Fixed random delays_ will refer to a model where delays are initialized randomly and not learned, while only weights are learned. Meanwhile _Decreasing_\(\sigma\) - _Fixed weights_ will refer to a model where the weights are fixed and only delays are learned with a decreasing \(\sigma\). Finally, _No delays_ denotes a standard SNN without delays. To ensure equal parameter counts across all models (for fair comparison), we increased the number of hidden neurons in the _No delays_ case. Moreover, to make the comparison even fairer, all models have the same initialization for weights, and if required, the same initialization for delays.

We compared the five different models as shown in Figure 4(a). The models with delays (whether fixed or learned) significantly outperformed the _No delays_ model both on SHD (FC) and SSC (FC); for us, this was an expected outcome given the temporal nature of these benchmarks, as achieving a high accuracy necessitates learning long temporal dependencies. However, we didn't expect the Fixed random delays model to be almost on par with models where delays were trained, with Decreasing \(\sigma\) model only slightly outperforming it.

To explain this, we hypothesized that a random uniformly distributed set of delay positions will most likely cover the whole possible temporal range. This hypothesis is plausible given the fact that the number of synaptic connections vastly outnumbers the total possible discrete delay positions for each kernel. Therefore, as the number of synaptic connections within a layer grows, the necessity of moving delay positions away from their initial state diminishes. And only tuning the weights of this set of fixed delays is enough to achieve comparable performance to delay learning.

In order to validate this hypothesis, we conducted a comparison using the same models with a significantly reduced number of synaptic connections. We applied fixed binary masks to the network's synaptic weight parameters. Specifically, for each neuron in the network we reduced the number of its synaptic connections to ten for both datasets (except for the No delays model, which has more connections to ensure equal parameters counts). This corresponds to 96% sparsity for SHD and 98% sparsity for SSC. With the number of synaptic connections reduced, it is unlikely that the random

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline Dataset & Method & Recurrent & Delays & \# Params & Top1 Accuracy \\ \hline \multirow{8}{*}{**SHD**} & EventProp-GeNN [31] & ✓ & ✗ & N/a & 84.80 \(\pm\) 1.5\% \\  & Cuba-LIF [7] & ✗ & ✗ & N/a & 87.80 \(\pm\) 1.1\% \\  & Adaptive SRNN [44] & ✓ & ✗ & N/a & 90.40\% \\  & SNN with Delays [2] & ✗ & ✓ & 0.1M & 90.43\% \\  & TA-SNN [43] & ✗ & ✗ & N/a & 91.08\% \\  & STSC-SNN [46] & ✗ & ✗ & 2.1M & 92.36\% \\  & Adaptive Delays [37] & ✗ & ✓ & 0.1M & 92.45\% \\  & RadLIF [3] & ✓ & ✗ & 3.9M & 94.62\% \\  & **Our work (2 hidden layers)** & ✗ & ✓ & **0.2M** & **95.07 \(\pm\) 0.24\%** \\ \hline \multirow{4}{*}{**SSC**} & Recurrent SNN [6] & ✓ & ✗ & N/a & 50.90 \(\pm\) 1.1\% \\  & Heterogeneous RSNN [32] & ✓ & ✗ & N/a & 57.30\% \\  & SNN-CNN [34] & ✗ & ✓ & N/a & 72.03\% \\  & Adaptive SRNN [44] & ✓ & ✗ & N/a & 74.20\% \\  & SpikGRU [7] & ✓ & ✗ & N/a & 77.00 \(\pm\) 0.4\% \\  & RadLIF [3] & ✓ & ✗ & 3.9M & 77.40\% \\  & **Our work (2 hidden layers)** & ✗ & ✓ & **0.7M** & **79.77 \(\pm\) 0.09\%** \\  & **Our work (3 hidden layers)** & ✗ & ✓ & **1.2M** & **80.29 \(\pm\) 0.06\%** \\ \hline \multirow{4}{*}{**GSC-35**} & MSAT [22] & ✗ & ✗ & N/a & 87.33\% \\  & RadLIF [3] & ✓ & ✗ & 1.2M & 94.51\% \\ \cline{1-1}  & **Our work (2 hidden layers)** & ✗ & ✓ & **0.7M** & **94.91 \(\pm\) 0.09\%** \\ \cline{1-1}  & **Our work (3 hidden layers)** & ✗ & ✓ & **1.2M** & **95.29 \(\pm\) 0.11\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification accuracy on SHD, SSC and GSC-35 datasets
uniform initialization of delay positions will cover most of the temporal range. Thus, specific long term dependencies will need to be learned by moving the delays.

The test accuracies corresponding to this control test are shown in Figure 4(b). And it illustrates the difference in performance between the Fixed random delays model and the Decreasing/Constant \(\sigma\) models in the sparse case. This enforces our hypothesis and shows the need to perform this control test for delay learning methods. Furthermore, it also indicates the effectiveness of our method.

In addition, we also tested a model where only the delays are learned while the synaptic weights are fixed (Decreasing \(\sigma\) - Fixed weights). It can be seen that learning only the delays gives acceptable results in the fully connected case (in agreement with [15]) but not in the sparse case. To summarize, it is always preferable to learn both weights and delays (and decreasing \(\sigma\) helps). If one has to choose, then learning weights is preferable, especially with sparse connectivity.

## 5 Conclusion

In this paper, we propose a method for learning delays in feedforward spiking neural networks using dilated convolutions with learnable spacings (DCLS). Every synaptic connection is modelled as a 1D Gaussian kernel centered on the delay position, and DCLS is used to learn the kernel positions (i.e delays). The standard deviation of the Gaussians is decreased throughout training, such that at the end of training we obtain a SNN model with one discrete delay per synapse, which could potentially be compatible with neuromorphic implementations. We show that our method outperforms the state-of-the-art in the temporal spiking benchmarks SHD and SSC and the non-spiking benchmark GSC-35, while using fewer parameters than previous proposals. And we also perform a rigorous control test that demonstrates the effectiveness of our method. Future work could investigate the use of other kernel functions than the Gaussian, or applying our method to other network architectures like convolutional networks.

#### Limitations

The primary limitations of our work revolve around the compatibility and constraints of our delay learning method. Specifically, our method is limited to offline training conducted in discrete-time simulations, and it cannot handle recurrent connections. Additionally, a maximum delay limit, which corresponds to the size of the kernel, must be predetermined and fixed before the learning process.

Figure 5: Barplots of test accuracies on SHD and SSC datasets for different models. With (a): fully connected layers (FC) and (b): with sparse synaptic connections (S). Reducing the number of synaptic connections of each neuron to ten, for both SHD and SSC.
#### Computational resources

This project required about 500 GPU hours on a single Nvidia Tesla T4 GPU with two Intel(R) Xeon(R) CPUs @ 2.20 GHz threads. Given this hardware configuration, a single training session lasted approximately 1 hour for the SHD runs, while for the SSC/GSC runs, a single training session lasted around 7 hours. The available computing resources allowed us to perform the required calculations efficiently, leading to accurate and competitive outcomes within a reasonable time.

#### Acknowledgment

This research was supported in part by the Agence Nationale de la Recherche under Grant ANR-20-CE45-0005 BRAIN-Net. This work was granted access to the HPC resources of CALMIP supercomputing center under the allocation 2023-[P22021]. Support from the ANR-3IA Artificial and Natural Intelligence Toulouse Institute is gratefully acknowledged. We also want to thank Wei Fang for developing the SpikingJelly framework that we used in this work.
