A Hybrid Neural Coding Approach for Pattern Recognition with Spiking
  Neural Networks
Recently, brain-inspired spiking neural networks (SNNs) have demonstrated
promising capabilities in solving pattern recognition tasks. However, these
SNNs are grounded on homogeneous neurons that utilize a uniform neural coding
for information representation. Given that each neural coding scheme possesses
its own merits and drawbacks, these SNNs encounter challenges in achieving
optimal performance such as accuracy, response time, efficiency, and
robustness, all of which are crucial for practical applications. In this study,
we argue that SNN architectures should be holistically designed to incorporate
heterogeneous coding schemes. As an initial exploration in this direction, we
propose a hybrid neural coding and learning framework, which encompasses a
neural coding zoo with diverse neural coding schemes discovered in
neuroscience. Additionally, it incorporates a flexible neural coding assignment
strategy to accommodate task-specific requirements, along with novel layer-wise
learning methods to effectively implement hybrid coding SNNs. We demonstrate
the superiority of the proposed framework on image classification and sound
localization tasks. Specifically, the proposed hybrid coding SNNs achieve
comparable accuracy to state-of-the-art SNNs, while exhibiting significantly
reduced inference latency and energy consumption, as well as high noise
robustness. This study yields valuable insights into hybrid neural coding
designs, paving the way for developing high-performance neuromorphic systems.
Xinyi Chen, Qu Yang, Jibin Wu, Haizhou Li, Kay Chen Tan
# A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks

###### Abstract

The biological neural systems evolved to adapt to ecological environment for efficiency and effectiveness, wherein neurons with heterogeneous structures and rich dynamics are optimized to accomplish complex cognitive tasks. Most of the current research of biologically inspired spiking neural networks (SNNs) are, however, grounded on a homogeneous neural coding scheme, which limits their overall performance in terms of accuracy, latency, efficiency, and robustness, etc. In this work, we argue that one should holistically design the network architecture to incorporate diverse neuronal functions and neural coding schemes for best performance. As an early attempt in this research direction, we put forward a hybrid neural coding framework that integrates multiple neural coding schemes discovered in neuroscience. We demonstrate that the proposed hybrid coding scheme achieves a comparable accuracy with the state-of-the-art SNNs with homogeneous neural coding on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets with less than eight time steps and at least \(3.90\times\) fewer computations. Furthermore, we demonstrate accurate, rapid, and robust sound source localization on SoClas dataset. This study yields valuable insights into the performance of various hybrid neural coding designs and hold significant implications for designing high performance SNNs.

 Spiking Neural Network, Neuromorphic Computing, Hybrid Coding Framework, Layer-wise Learning, Neural Coding.

## 1 Introduction

Human brains are the most sophisticated yet energy-efficient computational system. Spiking neural networks (SNNs), with rich neuronal dynamics and sparse spiking activities, are designed to mimic the remarkable information processing mechanism of biological neural systems _in silico_. The higher degree of biological details encompassed endows these artificial neural systems with a greater potential to reach the unprecedented accuracy, efficiency, and robustness of their biological counterparts. However, the state-of-the-art (SOTA) SNNs used in pattern recognition are primarily bootstrapped from homogeneous neurons and uniform network structures, and rely on extended learning with large-scale datasets to reach high performance. The findings in biological neural systems prompt us to look into diversified neural coding schemes and network structures in neural computation.

Over the last few decades, growing biological evidence indicates that perception and cognition are supported by diversified neural codes, which are believed to play an essential role in signal processing, learning, and memory in neural systems [1]. The earliest research on neural coding can be traced back to 1926 when Adrian and Zotterman first revealed that sensory nerves innervating the muscle discharge action potentials at a rate that is correlated with the weights hung from the muscle [2]. Since then, substantial evidence was accumulated from different neural systems and led to the conclusion that the firing rate of neurons is the key information carrier. However, the rate coding failed to describe many important features of neural computation.

Temporal coding schemes have been proposed as a complementary approach to explain the limitations of rate coding. For example, in the human visual system, object recognition can be accomplished within 150 ms since the stimulus onset [3], during which signals need to travel across multiple interconnected stages. The rank order coding has been proposed to account for how rapid information processing occurs in the visual system, which suggests the sensory information is represented as the chronological order of single spikes and more important features will be conveyed earlier [4]. Additionally, the spiking activities are found to be phase-correlated with the ongoing neural oscillations during memory formulation in the hippocampus [5]. Motivated by this, phase coding is introduced to explain how temporally correlated information has been stored and retrieved reliably in the memory system [6]. Moreover, as observed in the thalamus, hippocampus, and auditory systems, neurons are capable of firing a burst of spikes with a small inter-spike interval for rapid information transmission, which has led to the development of burst coding [7].

The large body of research on neural coding has provided a wealth of inspiration for the SNN design. Nevertheless, most of the existing SNNs are still grounded on a single, homogeneous neural coding schemes and rely on extensive learning to achieve task-related objectives. However, this design methodology typically leads to sub-optimal performance for the targeted tasks. For instance, direct coding uses the real value for the input or output layer, which is the least biological plausible scheme and identical to the coding scheme used in ANNs. The majority of research on ANN-to-SNN conversion uses the rate coding
and approximate the output of the ReLU activation function with the firing rate of spiking neurons [8, 9, 10, 11, 12]. As a more reliable coding scheme, burst coding encodes information into the inter-spike interval (ISI) and allows a burst of spikes to be transmitted within a short time interval for robust information transmission [13]. Despite superior performance, implementing burst coding comes at a high computational cost.

Some recent studies are looking at more efficient coding schemes. Phase coding, for example, embed information into the spike-phase based neural oscillations, which remarkably reduces the spike density [14]. However, the primitive phase coding is hard to represent a highly dynamical information stream. The time-to-first-spike (TTFS) coding, on the other hand, encodes information via the first spike time [15, 16, 17]. However, these TTFS-based SNNs face gradient instability and dead neuron problems during training [16], which limits the scale of networks.

In this paper, we argue that SNNs should be designed with multiple neural coding schemes to achieve optimal performance. To this end, we propose a hybrid coding and learning framework. As illustrated in Fig. 1, the proposed framework incorporates a neural coding zoo formed by a collection of bio-inspired neural coding schemes. According to the task requirements, the neural coding scheme is assigned for each network layer that provides the desired information representation basis. Moreover, we introduce a layer-wise learning method to ensure the desired neural representation can be achieved efficiently at each network layer with high levels of efficacy. We verify the effectiveness of the proposed framework on both image and audio perception tasks, and demonstrate desired attributes of high accuracy and robustness, as well as low latency and energy consumption. The main contributions of this paper are summarized as follows:

* We present a novel hybrid neural coding framework, which takes a holistic approach to designing the neural coding schemes of a SNN based on specific task requirements and environmental conditions.
* We propose a layer-wise learning method that can efficiently achieve the desired neural coding schemes with high levels of efficacy.
* We showcase the superiority and versatility of the proposed hybrid neural coding framework in performing image classification and sound localization tasks.
* Our research yields valuable insights into the performance of various hybrid neural coding designs and hold significant implications for designing optimal hybrid coding SNNs.

The remainder of this paper is organized as follows. In Section II, we present the proposed hybrid coding framework in detail. In Sections III and IV, we evaluate the performance of the proposed hybrid coding framework on image classification and sound localization tasks, respectively. Additionally, we conduct a series of ablation studies in Section IV to compare different hybrid coding designs. Finally, we conclude our findings in Section V.

## 2 A Hybrid Neural Coding Framework for Spiking Neural Networks

In this section, we will present our proposed hybrid neural coding framework that provides a holistic network representation basis design. As illustrated in Fig. 1, our framework consists of three key components: neural coding

Fig. 1: Illustration of the proposed hybrid neural coding framework for image classification task. According to the task requirements and environmental conditions, heterogeneous neural coding schemes are selected from the neural coding zoo and assigned to the Input, Hidden, and Output Layer. A layer-wise learning method is introduced to achieve the desired hybrid neural codes, where \(y^{L}\) represents the predicted outputs of the hybrid coding SNN, \(y^{label}\) denotes the one-hot labels, \(\hat{y}^{l}\) is the hidden states of the teacher ANN layer \(l\).
zoo, hybrid coding assignment strategy, and hybrid learning method. Firstly, we will provide a brief introduction of SNNs and then describe the collection of bio-inspired neural coding schemes that form our neural coding zoo. Secondly, we will introduce our proposed hybrid coding assignment strategy, with a focus on practical design for image classification and sound localization tasks. Finally, we will introduce a layer-wise learning method that can efficiently train hybrid coding SNNs.

### _Neural Coding Zoo_

Spiking neurons, unlike traditional artificial neurons, simulate the rich dynamic behaviors of biological neurons. Electrical impulses, or spikes, act as information carriers to transmit information between neurons. Spiking neurons generally transform input spike trains into their internal neuronal states and subsequently produce output spike trains according to specified neuronal functions. The frequency, precise spike timing, or synchrony patterns of spike trains could all be utilized to represent information, and the appropriate assignment of neural coding schemes could significantly impact the information processing efficiency, efficacy, and robustness of a SNN.

Inspired by the rich biological neural representation discovered in neuroscience, various neural coding schemes, with different merits and weaknesses, have been studied for SNNs. The neural coding zoo is a comprehensive collection of the most representative neural coding schemes, which provides a wealth of candidate options for designing hybrid coding SNNs. We selected direct coding, rate coding, phase coding, burst coding, and TTFS coding for our neural coding zoo to enable the construction of flexible and versatile SNNs that can be customized to meet specific task requirements and environmental conditions. In the following, we provide a summary of each selected neural code, with their merits and weaknesses depicted in Fig. 2. It is worth noting that our neural coding zoo can easily include other newly discovered or developed neural codes.

* **Direct coding** has been widely used in SNN input and output layers, which directly employs analog values to encode input stimuli or to decode outputs [18, 19]. Given its analog nature, it is the least biologically plausible and energy efficient neural code, as it abandons the binary spike-based information representation. Nevertheless, direct coding could avoid sampling errors and delays in other coding schemes as well as provide convenience for output decoding.
* **Rate coding** utilizes the mean firing rate within the given time window to represent the signal amplitude. Due to the similarity between ReLU activation values and firing rates of spiking neurons, rate coding has been widely adopted in ANN-to-SNN conversion methods [8, 9, 10, 11, 12] to construct SNNs that can well approximate the ReLU activation values. While rate coding is biologically plausible and can provide superior information representation capability, it is often criticized for high latency, poor energy efficiency, and being ineffective in representing fast-changing signals.
* **Phase coding** represents information by the phase of spiking activities with respect to the underlying neural oscillations [20]. It is proven to be more efficient in information transmission than rate coding and offers higher noise robustness than TTFS coding [21]. However, the representation capacity of phase coding is constrained by the frequency of underlying neural oscillations, which reduces performance when representing rapidly changing stimuli [14, 22].
* **Burst coding** makes effective use of the ISI to represent information [23] and is highly efficient in information transmission [24]. However, to keep track of the ISI has a high computational complexity. To address this issue, recent works propose using the neuron model that can emit multiple spikes within a single time step to represent graded ISIs [25, 13], resulting in a simple yet robust neural coding scheme.
* **TTFS coding** represents information using the timing of the first spike, where earlier spikes encode more significant information. TTFS coding leads to the favorable properties of fast inference, low spike density, and low energy consumption [26, 27]. However, the TTFS coding suffers from the notorious dead neuron problem that affects the training stability. Furthermore, spike timing is highly sensitive to the neuronal noise, which is pervasive in neural systems and analog computing substrates.

### _Hybrid Coding Assignment Strategy_

As discussed earlier, each homogeneous neural coding scheme has its own unique merits and weaknesses, and if applied alone, it will significantly impact the overall performance of SNNs. Inspired by the rich and hybrid

Fig. 2: Comparison of computational performance and biological plausibility among different candidate neural coding schemes.
neural codes observed in biological neural systems, we propose constructing hybrid coding SNNs with heterogeneous neural coding schemes. This can be achieved by selecting the neural coding schemes from the neural coding zoo and assign them to different parts of the network according to varying task requirements and environmental conditions. However, this will lead to a complex combinatory optimization problem that is intractable for large-scale networks. To reduce the design effort and accelerate training, we organize the network layers into three blocks and adopt a homogeneous neural coding scheme throughout each block. Specifically, three network blocks are considered in this work that correspond to the Input Layer, Hidden Layers, and Output Layer. In the following, we describe the key considerations when assigning neural coding schemes to these network blocks.

* **Input Layer** plays a significant role as the interface between the SNN and its external environment. It is imperative that this layer provides an efficient, real-time, and high-fidelity representation of the input stimuli. Therefore, to achieve these objectives, the input coding scheme must be chosen based on the nature of the input and the prevailing environmental conditions. This ensures a fast response while minimizing information loss during sampling and encoding.
* **Hidden Layers** play a crucial role in extracting rich and hierarchical feature representations from the input stimuli. Consequently, it is imperative that the chosen neural codes not only facilitate efficient feature extraction but also exhibit a degree of resource parsimony and robustness to neuronal noise.
* **Output Layer** functions as the output interface for for generating decisions and action, underscoring the need for a neural coding scheme that is both effective in facilitating accurate decision-making and swift in producing responses.

It is noteworthy that the proposed hybrid coding assignment strategy has the potential to provide guidance in the construction of a diverse range of models aimed at addressing various pattern recognition tasks, as well as designing task-specific models based on distinct objectives. To illustrate the efficacy of this approach in integrating the strengths of different coding schemes, we will use image classification and sound localization tasks as examples in the subsequent discussion.

### _Hybrid Coding Design for Image Classification_

#### 2.3.1 Direct Coding for High-Fidelity Input Representation

For image classification tasks, a high-fidelity representation of input images is typically required. However, rate coding may require a large sampling time window with high firing rates, leading to slow and inefficient during runtime. On the other hand, temporal coding schemes (e.g., TTFS coding) may suffer from dead neuron and gradient instability issues, negatively impacting the ease of training [16]. To address these challenges, we adopt a direct coding scheme that treats the intensity value of image pixels as the input current and directly injects it into post-synaptic neurons in the hidden layers [18, 19]. This approach bypasses the analog-to-spike conversion process required in other neural encoding schemes, eliminating the information loss during rate-based sampling as well as the quantization errors during temporal encoding.

#### 2.3.2 Burst Coding for Reliable and Rapid Hidden Feature Representation

In this task, we utilize burst coding to ensure rapid and efficient transfer of spike-based feature representation to subsequent layers. Unlike rate coding, which restricts neurons to emit at most one spike at a time, the bursting neurons can generate up to \(\Gamma\) spikes. This enables faster transmission of important information with greater robustness. In this work, \(\Gamma\) is set to five following the discovery that biological neurons rarely emit a spike burst with more than five spikes [28]. Specially, we use the bursting neuron model introduced in [25], which is defined as:

\[\mathbf{m}^{l}(t)=\mathbf{v}^{l}(t-1)+\mathbf{W}^{l}\mathbf{x}^{l-1}(t), \tag{1}\]

where \(\mathbf{x}^{l-1}(t)\) is the input current from layer \(l-1\). \(\mathbf{m}^{l}(t)\) and \(\mathbf{v}^{l}(t)\) are the membrane potential before and after spike generation, respectively. \(\mathbf{W}^{l}\) represents the weight matrix of layer \(l\). Once the membrane potential exceeds the firing threshold \(V_{th}^{l}\), an output spike is generated, and the membrane potential is reset to the resting potential using the reset-by-subtraction scheme [29]. This approach ensures that no information is lost during the resetting process. Mathematically, the spike generation process of bursting neurons can be expressed as:

\[\mathbf{s}^{l}(t) =\min\left(\max\left(\lfloor\frac{\mathbf{m}^{l}(t)}{V_{th}^{l}} \rfloor,0\right),\Gamma\right), \tag{2}\] \[\mathbf{v}^{l}(t) =\mathbf{m}^{l}(t)-\mathbf{s}^{l}(t)V_{th}^{l},\]

where \(\mathbf{s}^{l}(t)\) is an integer-valued vector that taken values from range \([0,\Gamma]\). For each input spike burst, it is transduced into the post-synaptic current as per:

\[\mathbf{x}^{l}(t)=\mathbf{s}^{l}(t)V_{th}^{l}. \tag{3}\]

#### 2.3.3 TTFS Coding for Fast and Efficient Output Decision-making

Ideally, the classification decision could be made as soon as sufficient evidence has been accumulated at the output layer \(L\). To achieve this objective, we adopt the TTFS decoding that reaches a decision once the first output spike has been detected, without waiting for all simulation time steps to finish. It thus leads to a significant reduction in inference time. Specifically, we adopt the leaky integrate-and-fire (LIF) neuron model [30] for the output layer, which offers richer temporal dynamics than the bursting neuron model used in hidden layers. The membrane potential of LIF neurons is defined as:

\[\mathbf{v}^{L}(t)=\mathbf{W}^{L}K^{L}\left(t\right), \tag{4}\]
where \(K^{L}(t)\) is the post-synaptic potential (PSP) kernel, which is induced from pre-synaptic spikes and iterated as follows:

\[\begin{split} K^{L}(t)&=K_{0}(\mathbf{m}^{L}(t)-\mathbf{I}^{L} (t)),\\ \mathbf{m}^{L}(t)&=\exp(-\frac{1}{\tau_{m}})\cdot(\mathbf{m} ^{L}(t-1)+s^{L-1}(t)),\\ \mathbf{I}^{L}(t)&=\exp(-\frac{1}{\tau_{s}})\cdot(\mathbf{I} ^{L}(t-1)+s^{L-1}(t)),\end{split} \tag{5}\]

where \(K_{0}\) is a normalization constant that ensures the maximum value of PSP kernel is equal to \(V_{th}^{L}\). \(\tau_{m}\) and \(\tau_{s}\) are the integration time constant of the membrane potential and the decay time constant of synaptic currents, respectively. The \(i^{th}\) output neuron fires when \(\mathbf{v}_{i}^{L}(t)\) crosses the set firing threshold \(V_{th}^{L}\). For classification problems with \(n\) classes, we treat the label \(i\) of the neuron that generates the first output spike as the classification result. Moreover, if multiple neurons fire at the same time, the result is defined as:

\[y^{L}=\max_{i}(\mathbf{v}_{i}^{L}(t_{f})). \tag{6}\]

Finally, in cases where no neuron fires until the end of the time window, we treat the last time step as the decision time and determine the classification result according to Eq. (6).

### _Hybrid Coding Design for Sound Localization_

#### 2.4.1 Phase Coding for High-fidelity Input Representation

Mammalian auditory systems possess a remarkable capability to localize the sound source from the time difference of arrival (TDOA) between their sensory receptors (i.e., ears) [31]. To be specific, the sound propagates through the air and reaches two ears with different time delays and energy attenuation. The peripheral auditory system first encodes the sound source into phase-locked spikes at each unilateral cochlea. Subsequently, the spike timing difference between bilateral cochleas is detected by the coincidence detection neurons in the medial superior olive (MSO), providing the important cue for localizing the sound source [32].

Considering the temporal characteristics of the sound localization task and the biological information transformation process described above, phase coding is an ideal candidate for extracting and encoding the localization cues. In particular, we adopted the Multi-Tone Phase Coding (MTPC) [33] for high-fidelity input representation in our model. As described in Fig. 3, MTPC first decomposes the received audio into single tone sinusoidal signals, with the first-peak of each single tone sinusoidal signal being encoded as a spike that represent the arrival time of the sound. Furthermore, the coincidence detection neurons detect the time delay between two microphones, triggering an output spike if a particular time delay is detected. Finally, the output spikes generated from sub-bands are aggregated based on the center frequency and bandwidth of Constant Q Transform (CQT).

#### 2.4.2 Burst Coding for Robust Hidden Feature Representation

In sound localization task, rapid information transmission and robust information representation are two desirable attributes when selecting a neural code for hidden layers. On the one hand, mammals need to respond rapidly to avoid moving predators or catch preys. On the other hand, phase-encoded inputs are sensitive to pervasive neuronal noises. To address these challenges, we select burst coding for hidden layers, as it demonstrate superior performance in these two aspects as shown in Fig. 2.

#### 2.4.3 TTFS Coding for Fast and Efficient Output Decision-Making

In addition to high localization accuracy, fast response to the moving sound source and high energy efficiency are crucial aspects of any auditory interface. These factors motivated our choice of TTFS coding for the output layer of the localization network. In Section 3.2, we will illustrate how TTFS decoding effectively reduces inference time without compromising the localization accuracy. Furthermore, in Section 4.2, we will compare TTFS coding with other coding schemes to validate its superiority in the context of sound localization tasks.

Fig. 3: Illustration of the neural encoding front-end designed for the sound localization task. a) The received audio is decomposed into \(n\) single tone sinusoidal signals by Fast Fourier Transform (FFT); b) The first-peak of each sinusoidal signal is encoded into a single spike to represent the arrival time of the sound; c) The coincidence detection network with \(N_{\tau}\) neurons is applied to detect the phase difference between two microphones. An output spike is trigger if a particular time delay is detected; d) The outputs of \(n\) single tones are grouped into \(N_{c}\) channels, following the Constant Q Transform (CQT), to reduce the computational cost of post-processing.
### _A Layer-wise Learning Method for Hybrid Coding_

The utilization of hybrid neural coding across various network stages renders the conventional end-to-end direct training approach unsuitable. To address this issue, we propose a novel layer-wise training method for the hidden and output layers, aimed at achieving the desired neural codes.

#### 2.5.1 Hidden Layer

We adopt the Teacher-Student learning approach that was initially introduced for training rate-based SNNs [34] and extend this training method to support other neural coding schemes. Specifically, our layer-wise learning method leverages a pre-trained ANN to supervise the training of a student SNN. We consider the intermediate feature representations of the ANN as the target and train the SNN to generate equivalent spike-count-based feature representations. For this purpose, we use mean square error (MSE) as the local loss function for each hidden layer, which is defined as:

\[\mathcal{L}^{l}\left(\hat{y}^{l},c^{l}(T)\right)=\left|\left|\hat{y}^{l}-r^{l} \cdot\frac{c^{l}(T)}{T}\right|\right|_{2}^{2}, \tag{7}\]

where \(\hat{y}^{l}\) denotes the activation of ANN layer \(l\). \(T\) is the simulation time window and \(c^{l}(T)=\sum_{t=1}^{T}s^{l}(t)\) is the total spike count. \(r^{l}\) is a layer-wise scaling factor that ensures the firing rate of spiking neurons (i.e., \(c^{l}(T)/T\)) matches the activation value of ANN neurons [10].

To eliminate the mismatch between training and testing, we train each hidden layer by taking into consideration the temporal dynamics of input spike trains rather than only considering the mean firing rate as in [10]. Specifically, we employ the backpropagation through time (BPTT) algorithm to perform temporal credit assignments at each layer. The weight gradients can be derived as follows:

\[\begin{split}\frac{\partial\mathcal{L}^{l}}{\partial w_{ij}^{l}}& =\sum_{t=1}^{T}\frac{\partial\mathcal{L}^{l}}{\partial m_{i}^{l}( t)}\frac{\partial m_{i}^{l}(t)}{\partial w_{ij}^{l}}\\ &=\sum_{t=1}^{T}\frac{\partial\mathcal{L}^{l}}{\partial m_{i}^{l} (t)}x_{j}^{l-1}(t),\end{split} \tag{8}\]

For the ease of expression, we let \(\delta_{i}^{l}(t)=\frac{\partial\mathcal{L}^{l}}{\partial s_{i}^{l}(t)}\) and yield

\[\frac{\partial\mathcal{L}^{l}}{\partial m_{i}^{l}(t)}=\begin{cases}\delta_{i}^ {l}(t+1)\frac{\partial s_{i}^{l}(t+1)}{\partial m_{i}^{l}(t+1)}+\delta_{i}^{l} (t)\frac{\partial s_{i}^{l}(t)}{\partial m_{i}^{l}(t)},&\text{ if }t<T,\\ \delta_{i}^{l}(T)\frac{\partial s_{i}^{l}(T)}{\partial m_{i}^{l}(T)},&\text{ if }t=T,\end{cases} \tag{9}\]

where

\[\delta_{i}^{l}(t)=\begin{cases}-V_{th}^{l}\delta_{i}^{l}(t+1)\frac{\partial s_ {i}^{l}(t+1)}{\partial m_{i}^{l}(t+1)}+\delta_{i}^{l}(T),&\text{ if }t<T,\\ -\frac{2}{T}\left(\hat{y}_{i}^{l}-r^{l}(\frac{1}{T}\Sigma_{t=1}^{T}s_{i}^{l}(t) )\right),&\text{ if }t=T.\end{cases} \tag{10}\]

For burst neurons, we adopt the straight-through estimator [35] (i.e., \(d\lfloor x\rfloor/dx=1\)) to address the discontinuity in Eq. (2) and result in

\[\frac{\partial s_{i}^{l}(t)}{\partial m_{i}^{l}(t)}=1. \tag{11}\]

We refer to this local learning method as local tandem learning (LTL) from here onwards.

#### 2.5.2 Output Layer

In order to achieve fast and yet accurate decision-making, the output layer is trained to make correct classification decisions based on the first output spike, which is referred to as TTFS learning hereafter. The TTFS learning is accomplished by carefully balancing two learning objectives. On the one hand, the weights of the target neuron should be adjusted to fire as early as possible. On the other hand, all the other neurons should fire later than the target one. These two learning objectives are achieved by adjusting the weights of output layer according to the following two loss functions, while all other hidden layer weights remain fixed.

To achieve the first objective, we select the cross-entropy (CE) loss as the first loss function, which can advance the first spike time of the target neuron by forcing the membrane potential to increase at that time. This learning process can be described by the following equation:

\[\mathcal{L}_{1}(y_{i},p_{i})=-\sum_{i=1}^{n}y_{i}^{label}\ln p_{i}, \tag{12}\]

where \(p_{i}=exp(\mathbf{v}_{i}^{L}(t_{f}))/\sum_{i=1}^{n}exp(\mathbf{v}_{i}^{L}(t_{f}))\) and \(y_{i}^{label}\) is the one-hot encoded target vector.

Considering the fixed firing threshold and growing postsynaptic membrane potential resulted from minimizing \(\mathcal{L}_{1}\), the decision time will continuously decrease until every decision is made at the earliest possible time step. This may, however, lead to the overfitting problem as the available input information is inadequately exploited.To address this issue, we introduce another loss function, \(\mathcal{L}_{2}\), which delays incorrect decisions to balance classification accuracy and decision speed. Specifically, long-term depression (LTD) is triggered when any other neuron fires earlier than the target one:

\[\mathcal{L}_{2}=\theta(t_{f})\cdot(\mathbf{v}^{L}(t_{f})-V_{th}^{L}), \tag{13}\]

where \(\theta(t_{f})\) is a scaling factor of LTD that is defined as:

\[\theta(t_{f})=max\left(1-\frac{acc(t_{f})}{acc(T)},0\right), \tag{14}\]

where \(acc(t_{f})\) and \(acc(T)\) correspond to the test accuracy at the first spike time and the last time step, respectively, whose value is updated at the end of each training epoch. For incorrect decisions chadie too early, the firing time will be delayed to later time steps depending on whether the accuracy can be improved if the decision is made later, that is \(\theta(t_{f})>0\).

The combination of the above two loss functions ensures those easy decisions can be made earlywhile difficult decisions are postponed to later and more informative time steps. The weight gradients can be derived as follows:

\[\frac{\partial\mathcal{L}}{\partial w^{L}}=\frac{\partial\mathcal{L}}{\partial \mathbf{v}^{L}(t_{f})}K^{L}(t_{f}). \tag{15}\]

where the total loss \(\mathcal{L}=\alpha\mathcal{L}_{1}+\beta\mathcal{L}_{2}\) is tuned by two hyperparameters \(\alpha\) and \(\beta\). The network parameters of hidden and output layers can be updated according to the stochastic gradients decent or their adaptive variants.
## 3 Accurate and Rapid Image classification with Hybrid coding

In image classification tasks, classification accuracy and inference speed serve as the two primary performance indicators. In this section, we compare the image classification accuracy and the required inference time of the proposed hybrid coding approach to other SOTA SNN implementations on CIFAR-10 [42], CIFAR-100 [42], and Tiny-ImageNet [43] datasets. For all datasets, we utilize VGG-16 [44] and ResNet-20 [45] network structures. Subsequently, we validate the effectiveness and training speed of our proposed layer-wise learning method. Lastly, we demonstrate that the proposed hybrid coding approach can achieve SOTA classification accuracy while significantly reducing the computational cost. For simplicity, direct, rate, burst, phase, and TTFS coding schemes are denoted as D, R, B, P, and T hereafter.

### _Superior Image Classification Performance_

In Tab. I, we report our results and compare them against other SOTA SNN models. To ensure a fair comparison, we select competitive SNN benchmarks based on their conversion errors for network conversion methods and classification accuracy for direct training methods. Overall, as the inference curves illustrated in Fig. 4(a) and 4(b), the proposed hybrid coding approach attains competitive classification accuracy while significantly reducing inference time across all model architectures and datasets.

For CIFAR-10 dataset, our VGG-16 model requires only \(5.21\) time steps on average to reach \(95.67\%\) classification

Fig. 4: Test accuracy as a function of inference time. The experiments are conducted on CIFAR-10, CIFAR-100, and Tiny ImageNet dataset with VGG-16 (a,c,e) and ResNet-20 (b,d,f). Note that for the proposed hybrid coding approach, we only report the average latency as the decision is made as soon as the first output spike is generated.
accuracy. Whereas the QCFS [9], considered the best among all ANN-to-SNN conversion methods, can only achieve a comparable accuracy with \(16\) time steps. Similar conclusions can be drawn from the converted ResNet-20 models. Furthermore, our ResNet-20 model outperforms other direct trained models, such as STBP-tdBN [39] and TET [40].

For the more challenging CIFAR-100 dataset, our model reaches a satisfactory accuracy of \(75.75\%\) and \(75.39\%\) using average inference time steps of \(6.88\) and \(3.94\) for VGG-16 and ResNet-20, respectively. As presented in Fig. 4(c) and 4(d), although the classification accuracy of our models is slightly worse than Calibration [10], Burst+LIPooling [25], and QCFS models, it is worth noting that their results are achieved with at least 16 time steps. Consequently, our models are more favorable in terms of inference time and energy consumption. Additionally, we evaluate our method on a large-scale Tiny-ImageNet dataset. Due to a lack of benchmarks on this dataset, we reproduce the results for Calibration, Burst+LIPooling, and QCFS methods using their publicly available codes. As the results are shown in Fig. 4(e) and 4(f) as well as Tab. 1, it is obvious that our proposed models are capable of achieving superior results with a shorter inference time.

To elucidate how early decision-making is accomplished with the proposed hybrid coding framework, we display the distribution of decision time for D+B+T and D+R+T models in Fig. 5. It is evident that the first spike time is densely distributed around 4 and 8 for D+B+T and D+R+T models, respectively. This result indicates that burst coding facilitates faster information transmission across hidden layers, which forms the basis for early decision-making by the TTFS classifier. Additionally, it is noteworthy that decisions made

\begin{table}
\begin{tabular}{l l l c c c c} \hline \hline
**Dataset** & **Method** & **Architecture** & **ANN Acc. (\%)** & **SNN Acc. (\%)** & \(\Delta\)**Acc. (\%)** & **Avg. Inference Time** \\ \hline \hline \multirow{8}{*}{CIFAR-10} & Burst Spikes [13] & VGG-16 & 91.41 & 91.41 & 0 & 793 \\  & RMP [36] & VGG-16 & 93.63 & 93.04 & -0.59 & 256 \\  & TSC [37] & VGG-16 & 93.63 & 93.27 & -0.36 & 128 \\  & Calibration [10] & VGG-16 & 95.72 & 95.14 & -0.58 & 64 \\  & Burst+LIPooling [25] & VGG-16 & 95.74 & 95.58 & -0.16 & 32 \\  & QCFS [9] & VGG-16 & 95.52 & 95.40 & -0.12 & 16 \\  & Diet-SNN [38] & VGG-16 & 93.72 & 93.44 & -0.28 & 10 \\  & **Ours (Hybrid Coding)** & **VGG-16** & **95.84** & **95.67** & **-0.17** & **5.21** \\  & RMP [36] & ResNet-20 & 91.47 & 91.36 & -0.11 & 2048 \\  & TSC [37] & ResNet-20 & 91.47 & 91.42 & -0.05 & 2048 \\  & Calibration [10] & ResNet-20 & 95.46 & 94.78 & -0.68 & 32 \\  & Burst+LIPooling [25] & ResNet-20 & 96.56 & 96.11 & -0.45 & 32 \\  & QCFS [9] & ResNet-20 & 91.77 & 91.62 & -0.15 & 16 \\  & Diet-SNN [38] & ResNet-20 & 92.79 & 92.54 & - & 10 \\  & STBP-tdBN [39] & ResNet-19* & - & 93.16 & - & 6 \\  & TET [40] & ResNet-19* & - & 94.50 & - & 6 \\  & **Ours (Hybrid Coding)** & **ResNet-20** & **95.45** & **95.24** & **-0.21** & **5.04** \\ \hline \multirow{8}{*}{CIFAR-100} & Burst Spikes [13] & VGG-16 & 68.77 & 68.69 & -0.08 & 3000 \\  & RMP [36] & VGG-16 & 71.22 & 70.93 & -0.29 & 2048 \\  & TSC [37] & VGG-16 & 71.22 & 70.97 & -0.25 & 2048 \\  & Calibration [10] & VGG-16 & 77.89 & 76.64 & -1.25 & 64 \\  & Burst+LIPooling [25] & VGG-16 & 78.49 & 78.26 & -0.23 & 64 \\  & QCFS [9] & VGG-16 & 76.28 & 76.24 & -0.04 & 16 \\  & Diet-SNN [38] & VGG-16 & 71.82 & 69.67 & -2.15 & 5 \\  & **Ours (Hybrid Coding)** & **VGG-16** & **77.15** & **75.75** & **-1.40** & **6.88** \\  & RMP [36] & ResNet-20 & 68.72 & 67.82 & -0.90 & 2048 \\  & TSC [37] & ResNet-20 & 68.72 & 68.18 & -0.54 & 2048 \\  & Calibration [10] & ResNet-20 & 77.16 & 76.32 & -0.84 & 32 \\  & Burst+LIPooling [25] & ResNet-20 & 80.69 & 79.83 & -0.86 & 64 \\  & QCFS [9] & ResNet-20 & 69.94 & 69.82 & -0.12 & 32 \\  & Diet-SNN [38] & ResNet-20 & 64.64 & 64.07 & -0.57 & 5 \\  & TET [40] & ResNet-19* & - & 74.72 & - & 6 \\  & **Ours (Hybrid Coding)** & **ResNet-20** & **76.56** & **75.39** & **-1.17** & **3.94** \\ \hline \multirow{8}{*}{Tiny-ImageNet} & Spike-thrift [41] & VGG-16 & 56.56 & 51.92 & -4.64 & 150 \\  & Calibration [10] & VGG-16\({}^{+}\) & 60.95 & 53.96 & -6.99 & 32 \\ \cline{1-1}  & Burst+LIPooling [25] & VGG-16\({}^{+}\) & 61.80 & 55.56 & -6.24 & 64 \\ \cline{1-1}  & QCFS [9] & VGG-16\({}^{+}\) & 54.91 & 54.92 & 0.01 & 16 \\ \cline{1-1}  & **Ours (Hybrid Coding)** & **VGG-16** & **58.75** & **56.95** & **-1.80** & **7.11** \\ \cline{1-1}  & Calibration [10] & ResNet-20\({}^{+}\) & 58.29 & 52.79 & -5.5 & 32 \\ \cline{1-1}  & QCFS [9] & ResNet-20\({}^{+}\) & 55.81 & 52.32 & -3.49 & 16 \\ \cline{1-1}  & **Ours (Hybrid Coding)** & **ResNet-20** & **58.34** & **56.50** & **-1.84** & **6.54** \\ \hline \hline \end{tabular}
* Only the results of ResNet-19 are reported in these works, which has comparable model capacity to other ResNet-20 models.

+ Our reproduced results using publicly available codes.

\end{table} TABLE I: Comparison of classification accuracy and inference time on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. \(\Delta\)Acc. = SNN Acc. = ANN Acc.
at earlier time steps exhibit higher accuracy than those made later, as indicated by the wrong decision rates. This implies early decisions correspond to easy samples and TTFS decoding has the capability to decide when sufficient evidence has been accumulated for different samples. Compared with direct decoding which makes all decisions at the final time step, TTFS decoding can significantly reduce average inference time yet achieve high classification accuracy.

### _Effective and Rapid Training with Layer-wise Learning Method_

To further explore the effectiveness and training speed of the proposed layer-wise learning method, we conduct a series of studies in this section. All experiments are performed on CIFAR-10 dataset using VGG-16 architecture.

Firstly, to assess the effectiveness of the LTL rule in producing the desired neural representation as specified by the neural coding scheme, we execute experiments by converting the hidden layers of teacher ANN into SNN layers, while maintaining the input and output layers with direct coding. In this set-up, the LTL-trained D+R+D and D+B+D models achieve minimal accuracy drops of \(0.56\%\) and \(0.31\%\) from the teacher ANN, suggesting the LTL rule is highly effective in producing rate- and burst-based neural representations. In the same vein of research, the Calibration method [10] proposes to fine-tune the hidden layer representations layer-wise to follow rate coding. However, this method neglects the temporal dynamics of spike trains and only performs training at the spike-train-level. As a result, their trained D+R+D model encounters a larger accuracy drop of \(1.21\%\).

For a more detailed analysis, we calculate the normalized MSE losses (i.e., Eq. (7)) between the convolutional layers of teacher ANN and student SNN and provide the results in Fig. 6(a). As expected, the LTL-trained D+R+D model achieves significantly lower MSE losses than those of the Calibration trained model. This finding suggests that fine-tuning the temporal dynamics of each hidden layer is crucial for achieving the desired neural representation. Furthermore, the D+B+D model outperforms the D+R+D models, indicating the burst coding can better approximate the neural representation of teacher ANN.

By leveraging the knowledge from teacher ANN, the layer-wise LTL rule can achieve faster training convergence compared to other end-to-end training rules. To illustrate this, we compare the learning curves of the LTL rule with the end-to-end TET rule [40]. As depicted in Fig. 6(b), our model reaches the accuracy of \(95.19\%\) rapidly within only \(5\) training epochs, whereas the TET model does not converge even at epoch \(100\). Coupling with the early decision-making capability as discussed earlier, our hybrid coding and learning framework significantly reduces the cost of training and deployment of SNNs.

In general, the quality of the decision improves when a longer decision time is provided to an SNN. To demonstrate how the proposed TTFS learning rule effectively balances the latency and classification accuracy, we plot the learning curves of the TTFS classifier in Fig. 6(c) and 6(d). The model performance is evaluated by considering both latency and accuracy. In particular, we use \(\Delta\)Acc.\(*\)\(t_{inf}\) as the evaluation criteria, where \(t_{inf}\) refers to the average inference time. As shown in Fig. 6(c), the average decision time is sharply advanced at the beginning, accompanied by a degree of accuracy loss due to overfitting. The learning dynamics during this period are mainly driven by the long-term potentiation (LTP) effect of loss function \(\mathcal{L}_{1}\) as described in Fig. 6(d). Fortunately, the training begins to be dominated by \(\mathcal{L}_{2}\) after the convergence of \(\mathcal{L}_{1}\), which improves the classification accuracy by slightly delaying wrong decisions, allowing more evidence to accumulate. The learning dynamics at these two stages perfectly align with the two objectives specified in the TTFS learning rule. Therefore, this ensures that an early yet accurate decision is made.

### _Superior Energy Efficiency with Hybrid Coding_

In the previous section, we demonstrated that the proposed hybrid coding framework can achieve rapid decision-making. Here, we analyze whether this attribute can translate into energy savings. To this end, we follow the convention of the Neuromorphic Computing community [46, 47] and calculate the synaptic operations (SynOps) ratio between equal-structured ANN and SNN use Multiply-and-Accumulate (MAC) and Accumulate (AC) operations, respectively. The total SynOps required in the ANN model can be calculated as below:

\[SynOps(ANN)=\sum_{l}^{L}f_{in}^{l}N^{l}, \tag{16}\]

where \(L\) is the total number of layers, \(f_{in}^{l}\) denotes the fan-in connections to the neuron in layer \(l\) from the previous layer, and \(N^{l}\) is the total number of neurons in layer \(l\). As the SNN model has an additional temporal dimension and only operates when spikes occur, the total SynOps calculation is different from its counterpart ANN model's. Hence, the total SynOps is formulated as below:

\[SynOps(SNN)=\sum_{t}^{T}\sum_{l}^{L-1}\sum_{j}^{N^{l}}f_{out,j}^{l}s_{j}^{l}(t), \tag{17}\]

where the \(T\) is the total inference time window, \(f_{out}^{l}\) represents the fan-out connections from neuron \(j\) in layer \(l\) to the

Fig. 5: The distribution of decision-making time for D+B+T and D+R+T models.
next layer, and \(s_{j}^{l}(t)\) denotes the spike occurrence in neuron \(j\).

To facilitate a fair comparison with other SOTA SNN models, we find recent works whose codes are publicly available and reproduce their results on CIFAR-10 dataset using VGG-16. Particularly, we report the SynOps ratios at the time step when a comparable classification accuracy has been achieved.

As the results summarised in Tab. II, our model achieves the lowest SynOps ratio of \(0.49\), which is at least 3.90\(\times\) lower than the other SOTA models. Notably, the total SynOps required by our model is much lower than that of the network conversion method QCFS, which achieves a SynOps ratio of 1.91 at T=16. This result again validates our earlier finding that the reduced inference time is not due to the compressed time window provided by burst coding. Instead, burst coding allows important information to be transmitted earlier, leading to early and efficient decision-making. Surprisingly, the SynOps ratio of the TET direct trained model is significantly higher than ours even at a comparable inference time. We suspect this is due to the high firing rate resulting from the activity regularization in TET training. According to a recent study [48] on 45 \(nm\) CMOS technology, the \(32\)-bit float MAC and AC operations consume 4.6 \(pJ\) and 0.9 \(pJ\) energy. As such, our SNN model can yield an order of magnitude energy saving over its ANN counterpart. The saving can be further boosted to 65.31\(\times\) when the cheaper \(32\)-bit int operations are used (3.1 \(pJ\) and 0.1 \(pJ\) for corresponding MAC and AC operations).

## 4 Efficient and Robust Sound Localization with Hybrid Coding

In this section, we evaluate the proposed hybrid coding approach on the SLoClas dataset [49] which is designed for real-world sound localization and classification tasks. Particularly, we perform a comprehensive comparison of the proposed hybrid coding SNN against other SOTA models based on factors such as localization precision, inference speed, energy consumption, and noise robustness. Moreover, an ablation study is carried out to gain insight into the impact that distinct hybrid neural coding designs exert on the model's overall performance. The detailed configurations of data preprocessing, MTPC front-end, and SNN architecture are summarized in Supplementary materials.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **SNN Acc. (\%)** & \(\Delta\)**Acc. (\%)** & **Avg. Inference Time** & **Ratio** & **Energy Saving** \\ \hline Calibration [10] & 95.04 & -0.65 & 64 & 2.10 & 2.43 (15.24) \\ QCFS [9] & 93.43 & -0.15 & 16 & 1.91 & 2.68 (11.94) \\ Burst+LIPooling [25] & 93.71 & -0.40 & 64 & 4.81 & 1.06 (6.65) \\ TET [40] & 94.94 & - & 6 & 2.06 & 2.48 (15.53) \\
**Ours (Hybrid Coding)** & **95.67** & **-0.17** & **5.21** & **0.49** & **10.43 (65.31)** \\ \hline \hline \end{tabular}
\end{table} TABLE II: Comparison of energy efficiency with other SNN implementations when achieving a comparable classification accuracy. \(\Delta\)**Acc**: the accuracy difference between equal-structured ANN and SNN. **Ratio**: SynOps(SNN)/SynOps(ANN). **Energy Saving**: Estimated energy saving of SNN against ANN using 32-bit float operations (32-bit int operations).

Fig. 6: (a) Comparison of final layer-wise MSE losses optimized by LTL and Calibration rules. (b) Learning curves of different training methods. (c) Illustration of learning dynamics during TTFS training. \(t_{inf}\) refers to the average inference time, and \(\Delta\)Acc. refers to the accuracy drop from baseline ANN. (d) The evolution of the two loss functions used for TTFS training.
### _Precise, Rapid, and Robust Sound Localization_

To furnish a comprehensive assessment of the localization precision, we report the mean absolute error (MAE) and the azimuth classification accuracy. As defined in Eqs. 18 and 19, the former computes the average difference between the predicted and the true azimuth angle, while the latter represents the proportion of accurately predicted test samples.

\[MAE(^{\circ})=\frac{1}{N_{t}}\sum_{i=1}^{N_{t}}|\hat{\theta}_{i}-\theta_{i}|, \tag{18}\]

\[Acc(\%)=\frac{1}{N_{t}}\sum_{n=1}^{N_{t}}(|\hat{\theta}_{i}-\theta_{i}|<\eta), \tag{19}\]

where \(\hat{\theta}_{i}\) and \(\theta_{i}\) are the estimated and the ground truth azimuth angle of sample \(i\). \(\eta\) is the acceptable absolute error when determining whether a sample has been correctly classified. \(N_{t}\) represents the total number of test samples. Since the SLoClas dataset is recorded at a resolution of \(5^{\circ}\), we set \(\eta\) to 2.5\({}^{\circ}\) and round the estimated angles to the nearest increment of 5 degrees in the classification task.

As presented in Tab. III, our hybrid coding SNN achieves an MAE of 0.60\({}^{\circ}\) with an average inference time of only 4.37 time steps, outperforming all other competitive baseline models. Furthermore, we compare the noise robustness of the proposed hybrid coding SNN with other baseline models using the directional background noises provided in the SLoClas dataset. To create noisy data samples, we randomly select a noise fragment from the SLoClas dataset and add it to the clean test samples at different signal-to-noise ratios (SNRs). As the results provided in Fig. 7, our hybrid coding SNN consistently outperform other models regardless of the noise levels. These results demonstrate the effectiveness and efficiency of the proposed hybrid coding approach.

### _Ablation Study of Hybrid Coding Design_

To understand why the proposed hybrid coding design can improve localization performance, we further conduct a series of ablation studies to investigate different hybrid coding designs.

#### 4.2.1 Localization Precision, Latency, and Energy Consumption

We first look into two sets of designs to investigate the impact of TTFS decoding: 1. P+B+T vs. P+B+D; 2. P+R+T vs. P+R+D. As the results presented in Tab. IV, direct decoding and TTFS decoding yield nearly identical MAEs; however, TTFS decoding requires only about one-third to one-half of the average inference time. Consequently, these results suggest that TTFS coding can maintain high localization precision while leading to reduced inference latency and energy consumption. It is worth noting that the energy efficiency index SynOps shows a similar ratio between the two output coding schemes regardless of what coding scheme used in hidden layers. These findings suggest that early decision-making supported by TTFS decoding contributes to energy saving.

Furthermore, the results offer valuable insights into the function of burst coding within our hybrid coding design. Our findings indicate that burst coding outperforms rate coding by 0.22 MAE and leads to a decision being made 3.25 time steps earlier. However, it is worth noting that burst coding is associated with a higher spike density. Interestingly, when burst coding is combined with TTFS coding, the energy consumption can be reduced by one-third, resulting in only half the energy consumption of an ANN. These findings suggest that choosing between burst coding and rate coding will depend on the specific requirements of the task at hand. For instance, if energy efficiency is a top

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **MAE(\({}^{\circ}\))** & **Avg. Inference Time** & **SynOps (Millions)** \\ \hline ANN & 1.12 & - & 5.15 \\ \hline P+R+D & 0.83 & 16 & 2.06 \\ \hline P+B+D & 0.61 & 16 & 10.70 \\ \hline P+R+T & 0.82 & 7.62 & 0.94 \\ \hline P+B+T & 0.60 & 4.37 & 2.86 \\ \hline \hline \end{tabular}
\end{table} TABLE IV: Comparison of MAE, inference time, and SynOps among different neural coding combinations.

\begin{table}
\begin{tabular}{l l c c c c c} \hline \hline
**Dataset** & **Method** & **ANN MAE(\({}^{\circ}\))** & **ANN Acc. (\%)** & **SNN MAE(\({}^{\circ}\))** & **SNN Acc.** & **Avg. Inference Time** \\ \hline \multirow{4}{*}{SLoClas} & GCC-PHAT-CNN [49] & 4.39 & 86.94 & - & - & - \\  & MTPC-CSNN [33]\({}^{*}\) & - & - & 1.34 & 93.16 & 2 \\ \cline{1-1}  & MTPC-CSNN [33]\({}^{*}\) & - & - & 1.23 & 93.95 & 4 \\ \cline{1-1}  & MTPC-CSNN [33]\({}^{*}\) & - & - & 1.02 & 94.72 & 8 \\ \cline{1-1}  & MTPC-RSNN [33]\({}^{*}\) & - & - & 1.48 & 94.30 & 51 \\ \cline{1-1}  & **Ours (Hybrid Coding)** & **1.12** & **95.19** & **0.60** & **95.61** & **4.37** \\ \hline \hline \multicolumn{8}{l}{* Our reproduced results.} \\ \end{tabular}
\end{table} TABLE III: Comparison of sound localization accuracy, MAE and inference time on SLoClas Dataset.

Fig. 7: Comparison of localization precision at different SNRs.
priority, P+R+T coding could be selected. Alternatively, if minimizing MAE and latency are key concerns, burst coding may be a better option, given its optimal balance between precision, latency, and energy consumption.

#### 4.2.2 Environmental Noise

In order to investigate the impact of hybrid coding design on the noise robustness of a model, two forms of environmental noise, specifically factory noise and babble noise, are employed to generate noise-corrupted test samples. The training is carried out on clean data samples, while the effect of noise is evaluated on noise-corrupted test samples at varying SNRs. The results of different hybrid coding designs are presented in Fig. 8(a) and 8(b). In general, the P+B+T hybrid coding design demonstrates superior noise robustness under most noise conditions, except for the factory noise with an SNR of 0, where P+B+D performs slightly better than ours P+B+T. This observation can be attributed to the fact that TTFS coding, which requires only about one-third of the time for decision-making, is less susceptible to cumulative noises when the noise level is low. However, in situations where environmental noise significantly impairs acoustic information, TTFS decoding becomes more susceptible to noise due to the limited evidence that can be leveraged, thereby hindering its performance. Furthermore, our findings suggest that the hybrid coding design employed in this study performs better under babble noise than under factory noise, owing to the latter's greater prevalence in dynamic frequency components.

#### 4.2.3 Neuronal Noise

We further conduct a detailed analysis of the impact of neuronal noise on the hybrid coding design. Neuronal noise is known to be ubiquitous in human brains. Investigating the performance of the hybrid coding design under such conditions provides an opportunity to identify the most promising design that closely mimics the human auditory system. In our experiments, we simulate neuronal noise by introducing spike deletion operations, whereby a fraction of hidden neurons are randomly contaminated by neuronal noise, thereby failing to generate output spikes. We present the localization results as a function of spike deletion rate in Fig. 8. Our findings indicate that TTFS coding is more reliable than direct coding, which is consistent with our observations on environmental noises. Interestingly, when a large percentage of information fails to transfer (\(>60\%\)), P+R+T outperforms P+B+T. This can be attributed to the fact that the decision time of P+B+T is approximately half that of P+R+T, which significantly impacts the amount of information delivered. Therefore, our design of P+B+T may not accumulate sufficient evidence during inference, hence trading noise robustness for inference speed.

## 5 Conclusion

In this paper, we proposed a hybrid neural coding approach to solve pattern recognition tasks with SNNs. In particular, we demonstrated that by holistically designing the SNN to encompass hybrid neural coding schemes, we could leverage the benefits of different neural codes to achieve optimal performance according to task requirements and environmental conditions. To overcome the challenges associated with training hybrid coding SNNs, we proposed a layer-wise learning method that was both highly effective and efficient in producing the desired neural coding schemes over existing SNN learning methods. Our extensive experiments demonstrated that state-of-the-art image classification accuracies and sound localization precision could be achieved with significantly improved inference speed, energy efficiency, and noise robustness compared to other SNNs constructed using homogeneous neural coding.

While our study primarily examined image classification and sound localization tasks, we emphasized that the design principles and training methods proposed in this work could be extended to other pattern recognition tasks and diverse environmental conditions. Notably, the successful implementation of our hybrid coding approach necessitated a comprehensive understanding of the strengths and limitations of various neural coding schemes as well as task-specific requirements. In future research, we aimed to develop a neural coding search method that could automatically identify the optimal hybrid neural coding scheme for each individual spiking neuron according to task demands and environmental conditions, thereby reducing the requisite level of expertise.

Fig. 8: The sound localization results under different noise conditions (a) babble noise, (b) factory noise, (c) neuronal noise. \(\delta\)**MAEs** is the MAE difference between noisy and clean conditions.