Spiking-LEAF: A Learnable Auditory front-end for Spiking Neural Networks
Brain-inspired spiking neural networks (SNNs) have demonstrated great
potential for temporal signal processing. However, their performance in speech
processing remains limited due to the lack of an effective auditory front-end.
To address this limitation, we introduce Spiking-LEAF, a learnable auditory
front-end meticulously designed for SNN-based speech processing. Spiking-LEAF
combines a learnable filter bank with a novel two-compartment spiking neuron
model called IHC-LIF. The IHC-LIF neurons draw inspiration from the structure
of inner hair cells (IHC) and they leverage segregated dendritic and somatic
compartments to effectively capture multi-scale temporal dynamics of speech
signals. Additionally, the IHC-LIF neurons incorporate the lateral feedback
mechanism along with spike regularization loss to enhance spike encoding
efficiency. On keyword spotting and speaker identification tasks, the proposed
Spiking-LEAF outperforms both SOTA spiking auditory front-ends and conventional
real-valued acoustic features in terms of classification accuracy, noise
robustness, and encoding efficiency.
Zeyang Song, Jibin Wu, Malu Zhang, Mike Zheng Shou, Haizhou Li
# Spiking-Leaf: A Learnable Auditory Front-End for

###### Abstract

Brain-inspired spiking neural networks (SNNs) have demonstrated great potential for temporal signal processing. However, their performance in speech processing remains limited due to the lack of an effective auditory front-end. To address this limitation, we introduce Spiking-LEAF, a learnable auditory front-end meticulously designed for SNN-based speech processing. Spiking-LEAF combines a learnable filter bank with a novel two-compartment spiking neuron model called IHC-LIF. The IHC-LIF neurons draw inspiration from the structure of inner hair cells (IHC) and they leverage segregated dendritic and somatic compartments to effectively capture multi-scale temporal dynamics of speech signals. Additionally, the IHC-LIF neurons incorporate the lateral feedback mechanism along with spike regularization loss to enhance spike encoding efficiency. On keyword spotting and speaker identification tasks, the proposed Spiking-LEAF outperforms both SOTA spiking auditory front-ends and conventional real-valued acoustic features in terms of classification accuracy, noise robustness, and encoding efficiency.

Zeyang Song\({}^{1}\), Jibin Wu\({}^{2,*}\), Malu Zhang\({}^{3}\), Mike Zheng Shou\({}^{1}\), Haizhou Li\({}^{1,4}\)+\({}^{1}\)National University of Singapore, Singapore

\({}^{2}\)The Hong Kong Polytechnic University, Hong Kong SAR, China

\({}^{3}\)University of Electronic Science and Technology of China, China

\({}^{4}\)Shenzhen Research Institute of Big Data, School of Data Science,

The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China Spiking neural networks, speech recognition, learnable audio front-end, spike encoding

Footnote †: This work was supported in part by National University of Singapore under FD-GhrbICS: Joint Lab for FD-SOI Always-on Intelligent & Connected Systems (Award L2001E0053) and the Hong Kong Polytechnic University under Project P0043563, P0046094, and P0046810.

## 1 Introduction

Recently, the brain-inspired spiking neural networks (SNNs) have demonstrated superior performance in sequential modeling [1, 2]. However, their performance in speech processing tasks still lags behind that of state-of-the-art (SOTA) non-spiking artificial neural networks (ANNs) [3, 4, 5, 6, 7, 8, 9]. This is primarily due to the lack of an effective auditory front-end that can synergistically perform acoustic feature extraction and neural encoding with high efficacy and efficiency.

The existing SNN-based auditory front-ends first extract acoustic features from raw audio signals, followed by encoding these real-valued acoustic features into spike patterns that can be processed by the SNN. For feature extraction, many works directly adopt the frequently used acoustic features based on the Mel-scaled filter-bank [3, 4, 5] or the GammaTone filter-bank [10]. Despite the simplicity of this approach, these handcrafted filter-bank are found to be suboptimal in many tasks when compared to learnable filter-bank [11, 12, 13, 14]. In another vein of research, recent works have also looked into the neurophysiological process happening in the peripheral auditory system and developed more complex biophysical models to enhance the effectiveness of feature extraction [15, 16]. However, these methods not only require fine-tuning a large number of hyperparameters but are also computationally expensive for resource-constrained neuromorphic platforms.

For neural encoding, several methods have been proposed that follow the neurophysiological processes within the cochlea [15, 16]. For instance, Cramer et al. proposed a biologically inspired cochlear model with the model parameters directly taken from biological studies [15]. Additionally, other methods propose to encode the temporal variations of the speech signals that are critical for speech recognition. The Send on Delta (SOD) [17] and threshold coding methods [10, 18, 19], for instance, encode the positive and negative variations of signal amplitude into spike trains. However, these neural encoding methods lack many essential characteristics as seen in the human's peripheral auditory system that are known to be important for speech processing, such as feedback adaptation [20].

To address these limitations, we introduce a Spiking LEarable Audio front-end model, called Spiking-LEAF. The Spiking-LEAF leverages a learnable auditory filter-bank to extract discriminative acoustic features. Furthermore, inspired by the structure and dynamics of the inner hair cells (IHCs) within the cochlea, we further proposed a two-compartment neuron model for neural encoding, namely
IHC-LIF neuron. Its two neuronal compartments work synergistically to capture the multi-scale temporal dynamics of speech signals. Additionally, the lateral inhibition mechanism along with spike regularization loss is incorporated to enhance the encoding efficiency. The main contributions of this paper can be summarized as follows:

* We propose a learnable auditory front-end for SNNs, enabling the joint optimization of feature extraction and neural encoding processes to achieve optimal performance in the given task.
* We propose a two-compartment spiking neuron model for neural encoding, called IHC-LIF, which can effectively extract multi-scale temporal information with high efficiency and noise robustness.
* Our proposed Spiking-LEAF shows high classification accuracy, noise robustness, and encoding efficiency on both keyword spotting and speaker identification tasks.

## 2 Methods

As shown in Fig. 1, similar to other existing auditory front-ends, the proposed Spiking-LEAF model consists of two parts responsible for feature extraction and neural encoding, respectively. For feature extraction, we apply the Gabor 1d-convolution filter bank along with the Per-Channel Energy Normalization (PCEN) to perform frequency analysis. Subsequently, the extracted acoustic feature is processed by the IHC-LIF neurons for neural encoding. Given that both the feature extraction and neural encoding parts are parameterized, they can be optimized jointly with the backend SNN classifier.

### Parameterized acoustic feature extraction

In Spiking-LEAF, the feature extraction is performed with a 1d-convolution Gabor filter bank along with the PCEN that is tailored for dynamic range compression [21]. The Gabor 1d-convolution filters have been widely used in speech processing [22, 14], and its formulation can be expressed as per:

\[\phi_{n}(t)=e^{i2\pi\eta_{n}t}\frac{1}{\sqrt{2\pi}\sigma_{n}}e^{-\frac{\sigma^ {2}}{2\sigma_{n}^{2}}} \tag{1}\]

where \(\eta_{n}\) and \(\sigma_{n}\) denote learnable parameters that characterize the center frequency and bandwidth of filter n, respectively. In particular, for input audio with a sampling rate of 16 kHz, there are a total of 40 convolution filters, with a window length of 25ms ranging over \(t=-L/2,...,L/2\) (\(L=401\) samples), have been employed in Spiking-LEAF. These 1d-convolution filters are applied directly to the audio waveform \(x\) to get the time-frequency representation \(F\).

Following the neurophysiological process in the peripheral auditory system, the PCEN [14, 21] has been applied subsequently to further compress the dynamic range of the obtained acoustic features:

\[PCEN(F(t,n))=\left(\frac{F(t,n)}{(\varepsilon+M(t,n))^{\alpha_{n}}+\delta_{n}} \right)^{r_{n}}-\delta_{n}^{r_{n}} \tag{2}\]

\[M(t,n)=(1-s)M(t-1,n)+sF(t,n) \tag{3}\]

In Eqs. 2 and 3, \(F(t,n)\) represents the time-frequency representation for channel \(n\) at time step \(t\). \(r_{n}\) and \(\alpha_{n}\) are coefficients that control the compression rate. The term \(M(t,n)\) is the moving average of the time-frequency feature with a smoothing rate of \(s\). Meanwhile, \(\varepsilon\) and \(\delta_{n}\) stands for a positive offset introduced specifically to prevent the occurrence of imaginary numbers in PCEN.

### Two-compartment spiking neuron model

The Leaky Integrate-and-Fire (LIF) neuron model [23], with a single neuronal compartment, has been widely used in brain simulation and neuromorphic computing [3, 4, 5, 7, 8]. The internal operations of a LIF neuron, as illustrated in Fig. 2 (a), can be expressed by the following discrete-time formulation:

\[I[t]=\Sigma_{i}w_{i}S[t-1]+b \tag{4}\] \[U[t]=\beta*U[t-1]+I[t]-V_{th}S[t-1]\] (5) \[S[t]=\mathbb{H}(U[t]-V_{th}) \tag{6}\]

Figure 1: The overall architecture of the proposed SNN-based speech processing framework.

Figure 2: Computational graphs of LIF and IHC-LIF neurons.
where \(S[t-1]\) represents the input spike at time step \(t\). \(I[t]\) and \(U[t]\) denote the transduced synaptic current and membrane potential, respectively. \(\beta\) is the membrane decaying constant that governs the information decaying rate within the LIF neuron. As the Heaviside step function indicated in Eq. 6, once the membrane potential exceeds the firing threshold \(V_{th}\), an output spike will be emitted.

Despite its ubiquity and simplicity, the LIF model possesses inherent limitations when it comes to long-term information storage. These limitations arise from two main factors: the exponential leakage of its membrane potential and the resetting mechanism. These factors significantly affect the model's efficacy in sequential modeling. Motivated by the intricate structure of biological neurons, recent work has developed a two-compartment spiking neuron model, called TC-LIF, to address the limitations of the LIF neuron [24]. The neuronal dynamics of TC-LIF neurons are given as follows:

\[I[t] =\Sigma_{i}w_{i}S[t-1]+b \tag{7}\] \[U_{d}[t] =U_{d}[t-1]+\beta_{d}*U_{s}[t-1]+I[t]\] (8) \[\quad-\gamma*S[t-1]\] \[U_{s}[t] =U_{s}[t-1]+\beta_{s}*U_{d}[t-1]-V_{th}S[t-1]\] (9) \[S[t] =\mathbb{H}(U[t]-V_{th}) \tag{10}\]

where \(U_{d}[t]\) and \(U_{s}[t]\) represent the membrane potential of the dendritic and somatic compartments. The \(\beta_{d}\) and \(\beta_{s}\) are two learnable parameters that govern the interaction between dendritic and somatic compartments. Facilitated by the synergistic interaction between these two neuronal compartments, TC-LIF can retain both short-term and long-term information which is crucial for effective speech processing [24].

### IHC-LIF neurons with lateral feedback

Neuroscience studies reveal that lateral feedback connections are pervasive in the peripheral auditory system, and they play an essential role in adjusting frequency sensitivity of auditory neurons [25]. Inspired by this finding, as depicted in Figure 2 (b), we further incorporate lateral feedback components into the dendritic compartment and somatic compartment of the TC-LIF neuron, represented by \(I_{f}[t]\) and \(I_{LI}[t]\) respectively. Specifically, each output spike will modulate the neighboring frequency bands with learnable weight matrices \(ZeroDiag(W_{f})\) and \(ZeroDiag(W_{LI})\), whose diagonal entries are all zeros.

The lateral inhibition feedback of hair cells within the cochlea is found to detect sounds below the thermal noise level and in the presence of noise or masking sounds [26, 27]. Motivated by this finding, we further constrain the weight matrix \(W_{LI}\geq 0\) to enforce lateral inhibitory feedback at the somatic compartment, which is responsible for spike generation. This will suppress the activity of neighboring neurons after the spike generation, amplifying the signal of the most activated neuron while suppressing other neurons. This results in a sparse yet informative spike representation of input signals. The neuronal dynamics of the resulting IHC-LIF model can be described as follows:

\[I_{s}[t] =\Sigma_{i}w_{i}S[t-1]+b \tag{11}\] \[I_{f}[t] =ZeroDiag(W_{f})*S[t-1]\] (12) \[I_{LI}[t] =ZeroDiag(W_{LI})*S[t-1]\] (13) \[U_{d}[t] =U_{d}[t-1]+\beta_{d}*U_{s}[t-1]+I_{s}[t]\] (14) \[\quad-\gamma*S[t-1]+I_{f}[t]\] \[U_{s}[t] =U_{s}[t-1]+\beta_{s}*U_{d}[t-1]-V_{th}S[t-1])\] (15) \[\quad-I_{LI}[t]\] \[S[t] =\mathbb{H}(U_{s}[t]-V_{th}) \tag{16}\]

To further enhance the encoding efficiency, we incorporate a spike rate regularization term \(L_{SR}\) into the loss function \(L\). It has been applied alongside the classification loss \(L_{cls}\) : \(L=L_{cls}+\lambda L_{SR}\) where \(L_{SR}=ReLU(R-SR)\). Here, \(R\) represents the average spike rate per neuron per timestep and \(SR\) denotes the expected spike rate. Any spike rate higher than \(SR\) will incur a penalty, and \(\lambda\) is the penalty coefficient.

## 3 Experimental Results

In this section, we evaluate our model on keyword spotting (KWS) and speaker Dataset V2 [28], which contains 105,829 one-second utterances of 35 commands. For speaker identification (SI), we use the Voxceleb1 dataset [29] with 153,516 utterances from 1,251 speakers, resulting in a classification task with 1,251 classes. We focus our evaluations on the auditory front-end by keeping model architecture and hyper-parameters of the backend SNN classifier fixed. The source codes will be released to ensure reproducibility.

### Superior feature representation

Table 1 compares our proposed Spiking-LEAF model with other existing auditory front-ends on both KWS and SI tasks. Our results reveal that the Spiking-LEAF consistently outperforms the SOTA spike encoding methods as well as the fbank features [3], demonstrating a superior feature representation power. In the following section, we validate the effectiveness of key components of Spiking-LEAF: learnable acoustic feature extraction, two-compartment LIF (TC-LIF) neuron model, lateral feedback \(I_{f}\), lateral inhibition \(I_{LI}\), and firing rate regulation loss \(L_{SR}\).

### Ablation studies

**Learnable filter bank and two-compartment neuron.** As illustrated in row 1 and row 2 of Table 2, the proposed learnable filter bank achieves substantial enhancement in feature representation when compared to the widely adopted Fbank feature. Notably, further improvements in classification accuracy are observed (see row 3) when replacing LIF neurons with TC-LIF neurons that offer richer neuronal dynamics. However, it is important to acknowledge that this improvement comes at the expense of an elevated firing rate,
which has a detrimental effect on the encoding efficiency.

**Lateral feedback.** Row 4 and row 5 of Table 2 highlight the potential of lateral feedback mechanisms in enhancing classification accuracy, which can be explained by the enhanced frequency sensitivity facilitated by the lateral feedback. Furthermore, the incorporation of lateral feedback is also anticipated to enhance the neuron's robustness in noisy environments. To substantiate this claim, our model is trained on clean samples and subsequently tested on noisy test samples contaminated with noise from the NOISEX-92 [31] and CHiME-3 [32] datasets. Fig. 3 illustrates the results of this evaluation, demonstrating that both the learnable filter bank and lateral feedback mechanisms contribute to enhanced noise robustness. This observation aligns with prior studies that have elucidated the role of the PCEN in fostering noise robustness [14]. Simultaneously, Fig. 4 showcases how the lateral feedback aids in filtering out unwanted spikes.

**Lateral inhibition and spike rate regularization loss.** As seen in Fig. 4 (b), when the spike regularization loss and lateral inhibition are not applied, the output spike representation involves a substantial amount of noise during non-speech periods. Introducing lateral inhibition or spike regularization loss alone can not suppress the noise that appeared during such periods (Figs. (b) and (c)). Particularly, introducing the spike regularization loss alone results in a uniform reduction in the output spikes (Fig. 4 (d)). However, this comes along with a notable reduction in accuracy as highlighted in Table 2 row 6. Notably, the combination of lateral inhibition and spike rate regularization (Fig. 4 (e)) can effectively suppress the unwanted spike during non-speech periods, yielding a sparse and yet informative spike representation.

## 4 Conclusion

In this paper, we presented a fully learnable audio front-end for SNN-based speech processing, dubbed Spiking-LEAF. The Spiking-LEAF integrated a learnable filter bank with a novel IHC-LIF neuron model to achieve effective feature extraction and neural encoding. Our experimental evaluation on KWS and SI tasks demonstrated enhanced feature representation power, noise robustness, and encoding efficiency over SOTA auditory front-ends. It, therefore, opens up a myriad of opportunities for ultra-low-power speech processing at the edge with neuromorphic solutions.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{Tasks} & \multirow{2}{*}{Front-end} & Classifier & Classifier & Test \\  & & Structure & Type & Accuracy (\%) \\ \hline \multirow{7}{*}{KWS} & Fbank [3] & 512-512 & Feedforward & 83.03 \\  & Fbank+LIF & 512-512 & Feedforward & 85.24 \\  & Heidelberg[15] & 512-512 & Feedforward & 68.14 \\  & **Spiking-LEAF** & 512-512 & Feedforward & **92.24** \\  & Speech2spike [30] & 256-265-256 & Feedforward & 88.5 \\  & **Spiking-LEAF** & 256-256-256 & Feedforward & **90.47** \\ \cline{2-5}  & Fbank [3] & 512-512 & Recurrent & 93.58 \\  & Fbank+LIF & 512-512 & Recurrent & 92.04 \\  & **Spiking-LEAF** & 512-512 & Recurrent & **93.95** \\ \hline \multirow{7}{*}{SI} & Fbank & 512-512 & Feedforward & 29.42 \\  & Fbank+LIF & 512-512 & Feedforward & 27.23 \\  & **Spiking-LEAF** & 512-512 & Feedforward & **30.17** \\ \cline{1-1} \cline{2-5}  & Fbank & 512-512 & Recurrent & 31.76 \\ \cline{1-1}  & Fbank+LIF & 512-512 & Recurrent & 29.74 \\ \cline{1-1}  & **Spiking-LEAF** & 512-512 & Recurrent & **32.45** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of different auditory front-ends on KWS and SI tasks. The bold color denotes the best model for each network configuration.

\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline Acoustic features & Neuron type & \(I_{f}\) & \(I_{LI}\) & \(L_{SR}\) & Firing rate & Accuracy \\ \hline Fbank & LIF & - & - & - & 17.94\% & 85.24\% \\ Learnable & LIF & - & - & - & 18.25\% & 90.73\% \\ Learnable & TC-LIF & - & - & - & 34.21\% & 91.89\% \\ Learnable & TC-LIF & ✓ & - & - & 40.35\% & 92.24\% \\ Learnable & TC-LIF & ✓ & ✓ & - & 34.54\% & **92.43\%** \\ Learnable & TC-LIF & ✓ & - & ✓ & 15.03\% & 90.82\% \\ Learnable & TC-LIF & ✓ & ✓ & ✓ & **11.96\%** & 92.04\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation studies of various components of the proposed Spiking-LEAF model on the KWS task.

Figure 4: This figure illustrates the Fbank feature and spike representation generated by Spiking-LEAF without and with lateral inhibition and spike rate regularization loss.

Figure 3: Test accuracy on the KWS task with varying SNRs.