Unleashing the Potential of Spiking Neural Networks for Sequential
  Modeling with Contextual Embedding
The human brain exhibits remarkable abilities in integrating temporally
distant sensory inputs for decision-making. However, existing brain-inspired
spiking neural networks (SNNs) have struggled to match their biological
counterpart in modeling long-term temporal relationships. To address this
problem, this paper presents a novel Contextual Embedding Leaky
Integrate-and-Fire (CE-LIF) spiking neuron model. Specifically, the CE-LIF
model incorporates a meticulously designed contextual embedding component into
the adaptive neuronal firing threshold, thereby enhancing the memory storage of
spiking neurons and facilitating effective sequential modeling. Additionally,
theoretical analysis is provided to elucidate how the CE-LIF model enables
long-term temporal credit assignment. Remarkably, when compared to
state-of-the-art recurrent SNNs, feedforward SNNs comprising the proposed
CE-LIF neurons demonstrate superior performance across extensive sequential
modeling tasks in terms of classification accuracy, network convergence speed,
and memory capacity.
Xinyi Chen, Jibin Wu, Huajin Tang, Qinyuan Ren, Kay Chen Tan
# Unleashing the Potential of Spiking Neural Networks for

###### Abstract

The human brain exhibits remarkable abilities in integrating temporally distant sensory inputs for decision-making. However, existing brain-inspired spiking neural networks (SNNs) have struggled to match their biological counterpart in modeling long-term temporal relationships. To address this problem, this paper presents a novel Contextual Embedding Leaky Integrate-and-Fire (CE-LIF) spiking neuron model. Specifically, the CE-LIF model incorporates a meticulously designed contextual embedding component into the adaptive neuronal firing threshold, thereby enhancing the memory storage of spiking neurons and facilitating effective sequential modeling. Additionally, theoretical analysis is provided to elucidate how the CE-LIF model enables long-term temporal credit assignment. Remarkably, when compared to state-of-the-art recurrent SNNs, feedforward SNNs comprising the proposed CE-LIF neurons demonstrate superior performance across extensive sequential modeling tasks in terms of classification accuracy, network convergence speed, and memory capacity.

## Introduction

Driven by the recent advance in training algorithm development [22, 23] and model architecture design [10, 11], brain-inspired spiking neural networks (SNNs) have exhibited remarkable performance in tasks such as image classification [22, 24, 25], speech processing [26, 27], and robotic control [12, 10]. Furthermore, when implemented on ultra-low-power and densely-connected neuromorphic chips [1, 11, 12], these models demonstrate compelling energy efficiency and low latency, making them highly promising for achieving ultra-low-power artificial intelligence (AI) at the edge. However, despite these advancements, existing SNNs still face challenges in effectively modeling long-term temporal dependency when compared to their biological counterparts [24, 25].

Towards the goal of enhancing the sequential modeling capability of SNNs, recent research endeavors have focused on developing novel spiking neuron models that incorporate slow-decaying variables capable of storing long-term memory. One notable approach involves the integration of a threshold adaptation mechanism into spiking neurons, wherein the firing threshold of a neuron is elevated and gradually decays following each firing event [13]. This adaptive firing threshold mechanism not only helps suppress excessive neuronal firing but also demonstrates effectiveness in storing long-term memory. Furthermore, Yin et al. propose to use a learnable decaying time constant for the adaptive firing threshold, thereby enabling the storage and utilization of multi-scale temporal information [23]. Similarly, Shaban et al. introduce a novel adaptive spiking neuron model, wherein the threshold undergoes a double-exponential decay, enabling the storage of both short-term and long-term information [2]. However, despite the promising results observed with these models, their ability to establish long-term dependencies remains limited. This limitation can be attributed to the inadequate representation of the temporal structure of sensory inputs within the long-term memory created by the adaptive firing threshold.

In another vein of research, the integration of self-attention mechanisms along with position encoding (PE) has been proposed in Transformer [26] to enhance the efficacy of sequential modeling. Specifically, self-attention enables the direct modeling of temporal dependencies between distant inputs. Additionally, PE has been introduced to explicitly represent the temporal structure of inputs using a fixed-length vector, thereby addressing the position invariance issue associated with self-attention operations. In practice, PE can be generated through a fixed sinusoidal function [26, 27], or learned together with other model parameters [10]. Notably, these advancements have contributed to the remarkable progress witnessed in large language models such as BERT [12] and GPT [1]. Motivated by these developments, attention-based spiking neural networks [24, 25] have recently been proposed to enhance the sequential modeling capabilities of SNNs. However, these models encounter challenges related to high computational com
plexity, limitations in hardware implementation, and the inability to operate in real-time scenarios.

In this work, we present a novel spiking neuron model, dubbed **Contextual Embedding Leaky Integrate-and-Fire (CE-LIF)**. Our proposed model incorporates the concepts of adaptive firing threshold and position encoding to enhance the sequential modeling capability of SNNs. Specifically, we introduce a contextual embedding component to induce the firing threshold adjustments, which is produced by multiplying the membrane potential with a learnable temporal encoding (TE). Notably, the TE allows for the explicit representation of a neuron's preferred temporal pattern, thereby alleviating the challenge of preserving the temporal structure of the input signals within the adaptive firing threshold. To validate the effectiveness of the CE-LIF neuron, we conduct both theoretical analysis and extensive experimental studies, highlighting its superior sequential modeling capabilities. The key contributions of our work can be summarized as follows:

* We propose a novel spiking neuron model called CE-LIF that can significantly enhance the sequential modeling capability of SNNs.
* We provide a theoretical analysis of the proposed CE-LIF neuron model. Our analysis offers valuable insights into how error gradients can be effectively propagated to distant inputs, thereby establishing long-term temporal dependencies.
* We thoroughly validate the effectiveness of CE-LIF on a wide range of sequential modeling tasks. Notably, we demonstrate that the feedforward SNNs comprising CE-LIF neurons can surpass both state-of-the-art (SOTA) recurrent SNNs and LSTM models in terms of classification accuracy, network convergence speed, and memory capacity.

## Methods

### Leaky Integrate-and-Fire (LIF) Neuron Model

Inspired by the rich neuronal dynamics and firing behavior of biological neurons, a number of spiking neuron models have been proposed in the literature, offering varying levels of biological details [11]. For large-scale SNN simulation, the leaky integrate-and-fire (LIF) model has commonly been adopted. This simplified neuron model can describe the essential membrane integration and spike generation processes of biological neurons while remaining computationally efficient. The neuronal dynamic of LIF neurons could be described as follows:

\[\tau_{m}\frac{dv(t)}{dt}=-(v(t)-v_{reset})+I(t), \tag{1}\] \[\text{if}\quad v(t)\geq\theta,\quad v(t)\to v_{reset}.\]

where \(v_{reset}\) denotes the resting potential, \(\tau_{m}\) is the membrane decaying time constant, and \(I\) corresponds to the input current transduced from the input spikes \(S(t)\), that is, \(I(t)=\mathcal{W}S(t)\), where \(\mathcal{W}\) is the synaptic weight. \(\theta\) signifies

Figure 1: The comparison of neuronal structure and dynamics between biological neurons and spiking neurons. **(a)** Biological neurons transduce presynaptic spikes into input currents, which are subsequently integrated into the membrane potential. The spike generation happens once the membrane potential surpasses the firing threshold. Notably, the firing threshold of biological neurons undergoes adaptations in response to the firing activities of the neuron. **(b-d)** The functional block diagrams (bottom row) and neuronal dynamics (top row) of the LIF neuron, ALIF neuron, and our proposed CE-LIF neuron. The LIF neuron generates spikes at a fixed firing threshold. In contrast, the ALIF model refines the LIF model by incorporating a biologically inspired adaptive firing threshold, thereby enhancing neuronal memory capacity while suppressing excessive firing. Furthermore, our proposed CE-LIF model advances upon the ALIF model by replacing the postsynaptic-spike-induced firing threshold adjustment with a contextual-embedding-induced one. Specifically, this contextual embedding is produced by multiplying the membrane potential with a learnable temporal encoding. This refinement not only results in enhanced memory capacity compared to ALIF but also facilitates long-term temporal credit assignment.
the firing threshold. When the membrane potential reaches \(\Theta\), it triggers the generation of an output spike. Following the spike generation, membrane potential will be reset to \(v_{reset}\). In practice, this continuous-time formulation is usually discretized using the Euler method:

\[\left\{\begin{array}{l}V[t]=\alpha\cdot V[t-1]\cdot(1-S[t-1])+I[t],\\ S[t]=H\left(V[t]-\theta\right),\end{array}\right. \tag{2}\]

where \(\alpha=e^{-\tau_{m}}\) is the membrane potential decaying rate, \(H(\cdot)\) is the Heaviside step function. SNNs comprising LIF neurons have demonstrated exceptional performance in pattern recognition tasks with limited temporal context. However, their suitability for tasks necessitating extended temporal dependencies is currently constrained due to two contributing factors. Firstly, the internal memory of LIF neurons, represented by the membrane potential, diminishes rapidly over time and is reset upon neuron firing. It, therefore, presents a significant challenge in retaining long-term memory. Secondly, the backpropagation-through-time (BPTT) algorithm, which is commonly employed in SNN training, poses an additional obstacle to learning long-term dependency. This is attributed to the recursive nature of gradient updates, which can lead to problems such as gradient vanishing and exploding.

### Adaptive LIF Neuron Model

Research on biological neurons has revealed that the neuronal firing threshold undergoes adaptations in response to neuronal firing activities. Specifically, an increase in the frequency of neuronal firing results in an elevation of the firing threshold [16]. This phenomenon has been identified as playing a crucial role in enhancing memory retention. Inspired by these findings, the adaptive threshold LIF (ALIF) neuron model has been introduced with the aim of enhancing the memory capacity of spiking neurons [1]. The formulation of the ALIF model can be summarized as follows:

\[\left\{\begin{array}{l}\Theta[t]=\beta\cdot(\Theta[t-1]-\Theta_{0})+\gamma \cdot S[t-1]+\Theta_{0},\\ V[t]=\alpha\cdot V[t-1]\cdot(1-S[t-1])+I[t],\\ S[t]=H\left(V[t]-\Theta[t]\right),\end{array}\right. \tag{3}\]

where \(\Theta_{0}\) represents the initial firing threshold. Each postsynaptic spike contributes to a transient increase in the firing threshold \(\Theta\) with a gain of \(\gamma\), whose contribution decays at a slow rate of \(\beta\) (e.g., 0.99). This firing threshold adaptation mechanism encompasses two significant advantages. Firstly, the elevated firing threshold can effectively suppress excessive firing, thereby improving the efficiency of neural signal processing. Secondly, benefiting from its slow decay rate and non-resetting characteristics, the adaptive threshold can serve as a long-term memory. However, within such memory storage, the input information is compressed into low-resolution binary spikes with their temporal structures being inadequately represented, thereby presenting substantial obstacles when performing temporal credit assignments (see Supplementary Materials for more detailed analysis).

### Contextual Embedding LIF Neuron

To address the above-identified deficiencies in ALIF neurons, we redesign the adaptive firing threshold by replacing the postsynaptic-spike-induced firing threshold increment with a contextual-embedding-induced one, we refer to the resulted neuron model as **Contextual Embedding LIF (CE-LIF)**. The refined adaptive firing threshold function is expressed as:

\[\Theta[t]=\beta\cdot(\Theta[t-1]-\Theta_{0})+TE[t]\odot V[t-1]+\Theta_{0}, \tag{4}\]

where \(\odot\) denotes the Hadamard product. The proposed contextual embedding comprises two interacting components, namely the membrane potential \(V\) and the temporal encoding vector \(TE\). By replacing \(S[t-1]\) with \(V[t-1]\)

Figure 2: Illustration of the evolution of neuronal variables during the Seq-MNIST classification task. **(a)** The input image is processed pixel-by-pixel through a raster scan. The recorded **(b)** contextual embeddings, **(c)** adaptive firing thresholds, and **(d)** the output spike trains of all output neurons. In (d), the black color bars represent the generated spikes.
in the firing threshold update function, the issue of low-resolution spike-based information representation is effectively addressed. This refinement not only aligns more closely with existing biological findings [12, 13], but also facilitates the effective delivery of gradients to early timesteps (see analysis in Ablations and Supplementary Materials). Additionally, the temporal encoding vector \(TE\in\mathbb{R}^{T}\) hard codes the preferred temporal pattern of a neuron, ensuring every neuron within the same layer possesses its unique TE vector. Therefore, the proposed contextual embedding effectively circumvents the challenge of preserving the temporal structure of input signals within the firing threshold.

Notably, the inner product between \(V\) and \(TE\) acts as an indicator of the presence of the preferred temporal pattern. This information is subsequently integrated into the firing threshold to facilitate context-dependent processing. In order to elucidate the dynamic interaction between these two components in enabling context-dependent processing, we provide a visualization in Figure 2 to illustrate the evolution of \(TE\odot V\), \(\Theta\), and \(S\) throughout the Sequential MNIST (Seq-MNIST) classification task. In this task, input images from the MNIST dataset are processed on a pixel-by-pixel basis via a raster scan [10]. Notably, as depicted in Figure 2(b), the \(TE\odot V\) of target neurons often display negative values, indicating the detection of their preferred temporal patterns. Consequently, this leads to a consistent reduction in the firing threshold \(\Theta\) in Figure 2(c), ultimately resulting in the observed bursting behaviors exhibited by the target neurons, as illustrated in Figure 2(d). In contrast, non-target neurons more frequently exhibit positive values in their \(TE\odot V\) representation, leading to elevated firing thresholds that inhibit them from firing.

### Impact of CE on Gradient Propagation

Here, we provide a theoretical analysis to explain how gradients can be effectively delivered to earlier timesteps in CE-LIF neurons. Specifically, we adopt the BPTT algorithm along with surrogate gradient [14, 15] for parameter update, which can be summarised as follows:

\[\Delta\mathcal{W}^{l}\propto\frac{\partial\mathcal{L}}{\partial\mathcal{W}^{ l}}=\sum_{t=1}^{T}\frac{\partial\mathcal{L}}{\partial V^{l}[t]}S^{l-1}[t],\quad \Delta b^{l}\propto\frac{\partial\mathcal{L}}{\partial\mathcal{W}^{l}}=\sum_{ t=1}^{T}\frac{\partial\mathcal{L}}{\partial V^{l}[t]}, \tag{5}\]

where \(\mathcal{L}\) is the loss function. \(\mathcal{W}^{l}\) and \(b^{l}\) refer to weight and bias terms of layer \(l\), respectively. The partial derivative of \(\frac{\partial\mathcal{L}}{\partial V^{l}[t]}\) can be further decomposed and calculated as follows:

\[\begin{split}\frac{\partial\mathcal{L}}{\partial V^{l}[t]}& =\frac{\partial\mathcal{L}}{\partial S^{l}[t]}\frac{\partial S^{l }[t]}{\partial V^{l}[t]}+\frac{\partial\mathcal{L}}{\partial V^{l}[t+1]} \frac{\partial V^{l}[t+1]}{\partial V^{l}[t]}+\frac{\partial\mathcal{L}}{ \partial\Theta^{l}[t+1]}\frac{\partial\Theta^{l}[t+1]}{\partial V^{l}[t]}\\ &=\frac{\partial\mathcal{L}}{\partial S^{l}[t]}g^{\prime}[t]+ \frac{\partial\mathcal{L}}{\partial V^{l}[t+1]}\alpha(1-S^{l}[t])+\frac{ \partial\mathcal{L}}{\partial\Theta^{l}[t+1]}TE[t+1],\\ \frac{\partial\mathcal{L}}{\partial S^{l}[t]}&=\frac{ \partial\mathcal{L}}{\partial V^{l}[t+1]}\frac{\partial V^{l}[t+1]}{\partial S ^{l}[t]}+\frac{\partial\mathcal{L}}{\partial V^{l+1}[t]}\frac{\partial V^{l+ 1}[t]}{\partial S^{l}[t]}\\ &=-\frac{\partial\mathcal{L}}{\partial V^{l}[t+1]}\alpha V^{l}[t]+ \frac{\partial\mathcal{L}}{\partial V^{l+1}[t]}W^{l},\\ \frac{\partial\mathcal{L}}{\partial\Theta^{l}[t]}& =\frac{\partial\mathcal{L}}{\partial S^{l}[t]}\frac{\partial S^{l}[t]}{ \partial\Theta^{l}[t]}+\frac{\partial\mathcal{L}}{\partial\Theta^{l}[t+1]} \frac{\partial\Theta^{l}[t+1]}{\partial\Theta^{l}[t]}\\ &=-\frac{\partial\mathcal{L}}{\partial S^{l}[t]}g^{\prime}[t]+ \frac{\partial\mathcal{L}}{\partial\Theta^{l}[t+1]}\beta,\end{split} \tag{6}\]

where \(g^{\prime}[t]=H^{\prime}(V^{l}[t]-\Theta^{l}[t])\) refers to the surrogate gradient function, which is applied during error backpropagation to address the discontinuity of spike generation function given in Eq. (2). In particular, we adopt a boxcar function [14] that is defined as follows:

\[g^{\prime}[t]=\left\{\begin{array}{ll}1,&\text{if}\quad|V^{l}[t]-\Theta^{l }[t]|<\Gamma,\\ 0,&\text{otherwise},\end{array}\right. \tag{7}\]

where \(\Gamma\) is a hyperparameter that determines the permissible range for gradient propagation.

Subsequently, by transforming Eq. (6) into a cumulative form using recursive accumulation, we have:

\[\begin{split}\frac{\partial\mathcal{L}}{\partial\Theta^{l}[t]}& =\sum_{i=t}^{T}-\frac{\partial\mathcal{L}}{\partial S^{l}[i]}g^{ \prime}[t]\beta^{i-t}=\sum_{i=t}^{T}-\frac{\partial\mathcal{L}}{\partial S^{l}[ i]}\frac{\partial S^{l}[i]}{\partial V^{l}[i]}\beta^{i-t}\\ &=\sum_{i=t}^{T}-\frac{\partial\mathcal{L}}{\partial V^{l}[i]} \beta^{i-t},\end{split} \tag{8}\]

Finally, by substituting Eq. (8) into Eq. (6), we can rewrite \(\frac{\partial\mathcal{L}}{\partial V^{l}[t]}\) as follows:

\[\begin{split}\frac{\partial\mathcal{L}}{\partial V^{l}[t]}& =\underbrace{\frac{\partial\mathcal{L}}{\partial S^{l}[t]}g^{ \prime}[t]}_{\text{LIF (Spatial)}}+\underbrace{\frac{\partial\mathcal{L}}{\partial V^{l}[t+1]} \alpha(1-S^{l}[t])}_{\text{LIF (Temporal)}}\\ &\quad\quad-\underbrace{\sum_{i=t+1}^{T}\frac{\partial\mathcal{L}}{ \partial V^{l}[i]}\beta^{i-t}TE[t+1]}_{\text{CE}},\end{split} \tag{9}\]

where the first two terms have an identical format as in the LIF model. The first term, as illustrated in Figure 3, represents the spatial credit assignment (SCA) between consecutive layers. However, the SCA may encounter the dead gradient problem when the surrogate gradient value \(g^{\prime}[t]\) equals zero. Furthermore, the second term corresponds to the temporal credit assignment (TCA) that decays exponentially at a rate of \(\alpha\) as well as resets after each spike generation. Consequently, the gradient information cannot be effectively propagated over a long period of time. Notably, our proposed CE

Figure 3: Illustration of the error backpropagation in an SNN comprising CE-LIF neurons. The backward paths of CE (orange dotted lines) establish a highway that can backpropagate the error information directly to earlier timesteps. Better to view this figure in color.
can effectively address this notorious problem by establishing a gradient highway between all dependent timesteps. By incorporating appropriately learned values of \(TE\) alongside a slow decay rate of \(\beta\), the CE-LIF model can effectively propagate gradients to much more distant timesteps compared with the ALIF model, as will be demonstrated in the Experiments section.

## Experiments

In this section, we provide a comprehensive analysis of the proposed CE-LIF model in terms of the sequential modeling capability, network convergence speed, and efficacy in performing long-term TCA. Moreover, ablation studies among different implementations of contextual embedding are also provided to demonstrate the superiority of the proposed design. The detailed experimental setups are provided in the Supplementary Materials.

### Pixel-level Sequential Image Classification

We first evaluate the performance of our proposed CE-LIF model on two widely adopted sequential classification tasks, namely Seq-MNIST and Permuted Sequential MNIST (PS-MNIST) [13]. In these tasks, the input images are processed by a model pixel-by-pixel through a raster scan. As the results summarized in Table 1, the CE-LIF model outperforms all other SOTA SNN implementations with the same or a lesser number of parameters. These results are also competitive with other ANN models, such as LSTM [10], IndRNN [12], and TCN [1]. Notably, the results of our CE-LIF model are achieved with a feedforward network structure, while almost all other works employ a recurrent network structure to achieve effective sequential modeling. Considering that our feedforward SNNs have already achieved saturation in terms of classification accuracy on these datasets, introducing a recurrent network structure would lead to an unnecessary parameter increase. As a result, we restrict our analysis of CE-LIF to a feedforward SNN in the following sections. For a more comprehensive analysis of the spiking RNN utilizing CE-LIF neurons, we refer readers to the Supplementary Materials.

Moreover, the CE-LIF model exhibits significantly faster and more stable training convergence compared with LIF and ALIF models. As demonstrated in Figure 4, the CE-LIF model achieves rapid convergence on the Seq-MNIST task within a mere ten training epochs, whereas the LIF and ALIF models require more than 100 epochs to converge. Furthermore, the CE-LIF model exhibits a much smoother loss landscape, which facilitates training convergence and offers great potential for improved generalization. These discrepancies do not primarily stem from the ease of convergence in a feedforward structure. More crucially, they can be attributed to the more effective TCA established by the proposed contextual embedding. Therefore, these findings highlight the superiority of the proposed CE-LIF model in sequential modeling.

### Memory Capacity Analysis

We further analyze the memory capacity of the proposed CE-LIF model on the adding problem and copy memory problem that necessitates long-term memory [11]. Specifically, the comparative study is performed among the CE-LIF, LSTM, and ALIF models.

**Adding problem** is a regression task in which two input sequences of length \(T\) are provided, one representing a value sequence and the other an indicator sequence [1]. The value sequence comprises randomly sampled entries from the range of \((0,1)\), while the indicator sequence assigns a value of \(1\) to two randomly selected entries and \(0\) to the remaining entries. The indicator sequence works as a pointer to indicate which two entries in the value sequence should be added. In this task, input sequences are processed by models one entry after the

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **SNN** & **Architecture** & **Parameters** & **Seq-MNIST** & **PS-MNIST** \\ \hline LSTM [10] & No & Recurrent & 66.5k & 98.20\% & 88.00\% \\ IndRNN [12] & No & Recurrent & 99.2k & 99.50\% & 96.00\% \\ TCN [1] & No & Convolution & 70.0k & 99.00\% & 97.20\% \\ \hline LSNN [1] & Yes & Recurrent & 68.2k & 93.70\% & N.A. \\ DEEP-R LSNN [1] & Yes & Recurrent & 66.4k & 96.40\% & N.A. \\ DEXAT [1] & Yes & Recurrent & N.A. & 96.40\% & N.A. \\ AHP [1] & Yes & Recurrent & 68.4k & 96.00\% & N.A. \\ LIF* & Yes & Feedforward & 85.1k & 63.45\% & - \\ LTMD* [14] & Yes & Feedforward & 85.1k & 68.56\% & - \\ PLIF* [15] & Yes & Feedforward & 85.1k & 87.92\% & - \\ GLIF* [15] & Yes & Feedforward & 85.4k & 95.27\% & - \\
**CE-LIF (Ours)** & **Yes** & **Feedforward** & **83.5k** & **98.19\%** & **97.90\%** \\ \hline LIF* & Yes & Recurrent & 155.1k & 84.14\% & 55.13\% \\ LTMD* [15] & Yes & Recurrent & 155.1k & 84.62\% & 54.93\% \\ ALIF [16] & Yes & Recurrent & 156.3k & 98.70\% & 94.30\% \\ PLIF* [15] & Yes & Recurrent & 155.1k & 91.79\% & - \\ GLIF* [15] & Yes & Recurrent & 157.5k & 96.64\% & 90.47\% \\
**CE-LIF (Ours)** & **Yes** & **Feedforward** & **153.9k** & **98.84\%** & **98.14\%** \\ \hline \hline \multicolumn{6}{l}{* Our reproduced results based on publicly available} \\ - & These models are unable to converge & \multicolumn{1}{c}{} \\ N.A. & These results are not publicly available & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 1: Comparison of model performance on pixel-level sequential image classification tasks. For both Seq-MNIST and PS-MNIST datasets, the sequence lengths are set to 784.
other, and the model will output the sum after receiving all entries. The mean squared error (MSE) is used as the loss function \(\mathcal{L}\). It is worth noting that the baseline MSE for this task is 0.167, obtained by assuming both entries take an average value of 0.5. The learning curves for the sequence lengths of \(T=100\), \(200\), \(500\), and \(800\) is presented in Figure 5(a-d). We notice that feedforward SNNs comprising CELIF neurons can successfully converge across all sequence lengths. In contrast, the selected baseline models exhibit inferior performance, with LSTM performing better than recurrently connected ALIF. Despite the slower convergence speed, the LSTM model reaches an MSE comparable to that of the CE-LIF model when \(T=100\) and \(200\). However, both models fail to converge for longer sequence lengths. Notably, the CE-LIF model exhibits a slight increase in MSE with increasing sequence length, attributed to the accumulated quantization errors from spike-based representation.

**Copy memory problem** is a classification task with 10 output classes ranging from \(0\) to \(9\). The total input sequence length is \(T+20\), where the key input sequence \(x_{0:10}\) is fed into the model during the first \(10\) timesteps, with each entry uniformly sampled from integer \(1\) to \(8\). For the following \(T\) timesteps, the inputs are set to \(0\). During the last \(10\) timesteps, the model consistently receives an input of \(9\), signifying the decision-making phase, and the model is expected to output \(x_{0:10}\) sequentially during this period. To successfully accomplish this task, the model is anticipated to possess long-term memory that can hold the input \(x_{0:10}\) for a prolonged period of \(T\). To match the spike-based representation in SNNs, we employ one-hot encoding to encode the input values before feeding them into SNNs, with cross-entropy (CE) loss being used as the loss function. Assuming \(y_{0:T+10}=0\) and \(y_{T+10:\text{end}}\) are sampled uniformly at random from \(0\) to \(9\), the baseline loss can be calculated as \(\frac{10\ln(8)}{T+20}\) (Arjovsky, Shah, and Bengio 2016). Figure 5 (e-h) present the learning curves for \(T=100\), \(200\), \(400\), and \(500\). Remarkably, the feedforward SNNs comprising the proposed CE-LIF neurons can converge within 5,000 iterations, even for the challenging case of \(T=500\). In contrast, both the LSTM and ALIF models fail to surpass the baseline value even with the shortest time interval \(T=100\), indicating they can not preserve information beyond the \(T\) time interval.

Figure 4: Comparison of **(a)** learning curves and **(b-d)** loss landscapes among LIF, ALIF, and CE-LIF models. Note that the CE-LIF model adopts a feedforward structure, while LIF and ALIF models use a recurrent one. These structures are chosen to maximize the respective model’s performance. The results in (a) are obtained across three runs with different random seeds.

Figure 5: Learning curves for **(a-d)** adding problem and **(e-h)** copy memory problem.
### Efficacy of CE in Facilitating Long-term TCA

To provide more insights into how long-term temporal relationships have been established in CE-LIF neurons, we provide a visualization in Figure 6 to illustrate the error gradients calculated for CE-LIF, LSTM, and ALIF models during training. All models have been trained on the copy memory problem with \(T=400\), and the gradients \(Grad[t]=\partial\mathcal{L}/\partial S^{l}[t]\) of the last hidden layer are reported. In this task, the inputs that contain the most valuable information are located within the first \(10\) timesteps, while the memory recall occurs during the last \(10\) timesteps. Therefore, an ideal model for this task should be able to establish long-term dependencies between the output error and the distant inputs that are situated \(410\) timesteps away. Noticeably, the CE-LIF model demonstrates an impressive ability to backpropagate gradients to these distant inputs effectively. In contrast, both the ALIF and LSTM models encounter challenges in propagating gradients across such a lengthy interval, although the ALIF model exhibits slightly better performance compared to the LSTM model. This empirical observation aligns with the theoretical analysis presented in the earlier section.

### Ablation Study of Contextual Embedding Design

In the CE-LIF model, the contextual embedding is formulated as the product of \(TE\) and \(V\). Here, we conduct an ablation study to evaluate the effectiveness of this design. The results presented in Table 2 demonstrate that the proposed design outperforms alternative candidates on both benchmarks. Notably, a significant improvement is observed when changing from \(S[t-1]\) used in ALIF to \(V[t-1]\). Furthermore, the accuracy is further improved when multiplying \(V[t-1]\) with the temporal encoding \(TE[t]\) that facilitates the temporal structure modeling. Hence, this study provides compelling evidence regarding the significant effectiveness of the proposed contextual embedding design.

### Visualization of Temporal Encoding

While previous sections have highlighted the superior sequential modeling capabilities of the CE-LIF model, it remains unclear what information is stored in TE that contributes to effective sequential modeling. Therefore, we investigate the learned TE on the Seq-MNIST task. As the heat map presented in Figure 7, the color at each coordinate \((i,j)\) represents the cosine similarity between \(TE[i]\) and \(TE[j]\). Before training, there is no transparent correlation between different time steps. In contrast, the heat map exhibits periodic patterns after training, demonstrating higher similarities between neighboring time steps and time steps that are multiples of 28 apart. This pattern aligns well with the structure of the input image, suggesting that \(TE\)s contribute to the modeling of temporal structure. Furthermore, it is observed that later time steps exhibit broader connections with earlier ones, thus contributing to effective decision-making by incorporating a broader temporal context.

## Conclusion

In this work, we introduced a novel spiking neuron model called CE-LIF, which can enhance the sequential modeling capabilities of SNNs. The theoretical analysis and extensive experimental studies demonstrated the efficacy of the proposed contextual embedding in facilitating long-term TCA, thereby leading to rapid network convergence and superior temporal classification accuracies. Furthermore, our experiments on the copy memory problem demonstrated the significantly increased memory capacity of CE-LIF compared to SOTA ALIF and LSTM models. It, therefore, opens up a myriad of opportunities for solving challenging temporal signal processing tasks with neuromorphic solutions.

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**ID** & \(\mathbf{CE[t]}=\) & **Seq-MNIST** & **PS-MNIST** \\ \hline
1 & \(S[t-1]\) & 68.66\% & 36.64\% \\
2 & \(V[t-1]\) & 87.55\% & 72.60\% \\
3 & \(TE[t]\) & 95.36\% & 90.19\% \\
4 & \(V[t-1]+TE[t]\) & 95.55\% & 89.97\% \\
5 & \(S[t-1]\odot TE[t]\) & 98.61\% & 96.21\% \\
6 & \(\mathbf{V[t-1]\odot TE[t]}\) & **98.84\%** & **98.14\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of various contextual embedding options in sequential image classification tasks. Note that a feedforward SNN is adopted in this experiment.

Figure 6: Illustration of the \(Grad[t]\) calculated for the ALIF, LSTM, and CE-LIF models in performing long-term TCA.

Figure 7: The cosine similarity between different time steps of \(TE\)**(a)** before training and **(b)** after training.