Uncovering the Representation of Spiking Neural Networks Trained with
  Surrogate Gradient
Spiking Neural Networks (SNNs) are recognized as the candidate for the
next-generation neural networks due to their bio-plausibility and energy
efficiency. Recently, researchers have demonstrated that SNNs are able to
achieve nearly state-of-the-art performance in image recognition tasks using
surrogate gradient training. However, some essential questions exist pertaining
to SNNs that are little studied: Do SNNs trained with surrogate gradient learn
different representations from traditional Artificial Neural Networks (ANNs)?
Does the time dimension in SNNs provide unique representation power? In this
paper, we aim to answer these questions by conducting a representation
similarity analysis between SNNs and ANNs using Centered Kernel Alignment
(CKA). We start by analyzing the spatial dimension of the networks, including
both the width and the depth. Furthermore, our analysis of residual connections
shows that SNNs learn a periodic pattern, which rectifies the representations
in SNNs to be ANN-like. We additionally investigate the effect of the time
dimension on SNN representation, finding that deeper layers encourage more
dynamics along the time dimension. We also investigate the impact of input data
such as event-stream data and adversarial attacks. Our work uncovers a host of
new findings of representations in SNNs. We hope this work will inspire future
research to fully comprehend the representation power of SNNs. Code is released
at https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA.
Yuhang Li, Youngeun Kim, Hyoungseob Park, Priyadarshini Panda
# Uncovering the Representation of Spiking Neural Networks Trained with Surrogate Gradient

###### Abstract

Spiking Neural Networks (SNNs) are recognized as the candidate for the next-generation neural networks due to their bio-plausibility and energy efficiency. Recently, researchers have demonstrated that SNNs are able to achieve nearly state-of-the-art performance in image recognition tasks using surrogate gradient training. However, some essential questions exist pertaining to SNNs that are little studied: _Do SNNs trained with surrogate gradient learn different representations from traditional Artificial Neural Networks (ANNs)? Does the time dimension in SNNs provide unique representation power?_ In this paper, we aim to answer these questions by conducting a representation similarity analysis between SNNs and ANNs using Centered Kernel Alignment (CKA). We start by analyzing the spatial dimension of the networks, including both the width and the depth. Furthermore, our analysis of residual connections shows that SNNs learn a periodic pattern, which rectifies the representations in SNNs to be ANN-like. We additionally investigate the effect of the time dimension on SNN representation, finding that deeper layers encourage more dynamics along the time dimension. We also investigate the impact of input data such as event-stream data and adversarial attacks. Our work uncovers a host of new findings of representations in SNNs. We hope this work will inspire future research to fully comprehend the representation power of SNNs. Code is released at [https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA](https://github.com/Intelligent-Computing-Lab-Yale/SNNCKA).

## 1 Introduction

Lately, Spiking Neural Networks (SNNs) (Tavanaei et al., 2019; Roy et al., 2019; Deng et al., 2020; Panda et al., 2020; Christensen et al., 2022) have received increasing attention thanks to their biology-inspired neuron activation and efficient neuromorphic computation. SNNs process information with binary spike representation and therefore avoid the need for multiplication operations during inference. Neuromorphic hardware such as TrueNorth (Akopyan et al., 2015) and Loihi (Davies et al., 2018) demonstrate that SNNs can save energy by orders of magnitude compared to Artificial Neural Networks (ANNs).

Although SNNs can bring enormous energy efficiency in inference, training SNNs is notoriously hard because of their spiking activation function. This function returns a zero-but-all gradient (_i.e.,_ Dirac delta function) and thus makes gradient-based optimization infeasible. To circumvent this problem, various training techniques have been proposed. For example, spike-timing-dependent plasticity (STDP) (Rao and Sejnowski, 2001)
either strengthens or weakens the synaptic weight based on the firing time; time-to-first-spike (Mostafa, 2017) encodes the information into the time of spike arrival to get a closed-form solution of the gradients. However, these two methods are restricted to small-scale tasks and datasets. Surrogate gradient technique (Bengio et al., 2013; Bender et al., 2018; Wu et al., 2018; Bellec et al., 2018; Kim et al., 2022b; Li et al., 2022), on the other hand, can achieve the best task performance by applying an alternate function during backpropagation. Combined with surrogate gradient, SNNs can be optimized by Backpropagation Through Time (BPTT) (Neftci et al., 2019) algorithm, outperforming other learning rules in SNNs.

Despite increasing interest in pursuing high-performance SNNs with surrogate gradient training, there is limited understanding of how surrogate gradient training affects the representation of SNNs. Investigating this fundamental question is critical since surrogate gradient-based BPTT algorithm mimics the way how ANN learns and is less biological-plausible compared to other learning rules like STDP. Therefore, it would be intriguing to study whether surrogate gradient-based SNNs learn different representations than ANNs. Understanding the representation learned in SNN can also promote further research developments, e.g., designing spiking-friendly architectures (Kim et al., 2022; Na et al., 2022) and exploring other ways to optimize SNNs (Bellece et al., 2020; Zhang and Li, 2020; Kim and Panda, 2021).

More concretely, we ask, do SNNs optimized by surrogate gradient BPTT learn distinct representations from ANNs? How do the width and depth of the neural network affect the representation learned in SNNs and ANNs? Does the extra temporal dimension in SNNs yield unique intermediate features? On neuromorphic datasets, how does the SNN process event-based data? In this paper, we aim to answer these core questions through a detailed analysis of ResNets (He et al., 2016) and VGG-series (Simonyan and Zisserman, 2015) models using a representation similarity analysis tool. Specifically, we utilize the popular Centered Kernel Alignment (CKA) (Kornblith et al., 2019) to measure the similarity between SNNs and ANNs. Fig. 1 demonstrates the overall workflow of our representation similarity analysis framework. Our analysis spans both the spatial and temporal dimensions of SNNs, as well as the impact of network architecture and input data.

Our contributions and findings include:

* We analyze the representation similarity between SNNs and ANNs using the centered kernel alignment to determine whether SNNs produce different feature representations from ANNs. We examine various aspects of representation similarity between SNNs and ANNs, including spatial and temporal dimensions, input data type, and network architecture.
* Surprisingly, our findings show that SNNs trained with surrogate gradient have a rather similar representation to ANNs. We also find that residual connections greatly affect the representations in SNNs.

Figure 1: **The representation similarity analysis workflow. The test images are fed into both ANN and SNN, then we record the intermediate feature for computing the correlation matrix, which is used for inferring the CKA similarity (Kornblith et al., 2019).**
* Meanwhile, we find that the time dimension in SNNs does not provide much unique representation. We also find that shallow layers are insensitive to the time dimension, where the representation in each time step converges together.

## 2 Related Work

**Spiking Neural Networks (SNNs).** SNNs have gained increasing attention for building low-power intelligence. Generally, the SNN algorithms to obtain high performance can be divided into two categories: (1) ANN-SNN conversion (Rueckauer et al., 2016, 2017; Han et al., 2020; Sengupta et al., 2019; Han and Roy, 2020) and (2) direct training SNN from scratch (Wu et al., 2018, 2019). Conversion-based methods utilize the knowledge from ANN and convert the ReLU activation to a spike activation mechanism. This type of method can produce an SNN in a short time. For example, in Rueckauer et al. (2017), one can find the percentile number and set it as the threshold for spiking neurons. Authors in Deng and Gu (2021) and Li et al. (2021) decompose the conversion error to each layer and then propose to reduce the error by calibrating the parameters. However, achieving near-lossless conversion requires a considerable amount of time steps to accumulate the spikes. Direct training from scratch allows SNNs to operate in extremely low time steps, even less than 5 (Zheng et al., 2020). To enable gradient-based learning, direct training leverages surrogate gradient to compute the derivative of the discrete spiking function. This also benefits the choice of hyper-parameters in spiking neurons. Recent works (Fang et al., 2021; Rathi and Roy, 2020; Kim and Panda, 2021; Deng et al., 2022) co-optimize parameters, firing threshold, and leaky factor together via gradient descent. Our analysis is mostly based on directly trained SNNs, as converted SNNs only contain ANN features and may be misleading for representation comparison.

**Representation Similarity Analysis (RSA).** RSA (Kriegeskorte et al., 2008) was not originally designed for analyzing neural networks specifically. Rather, it is used for representation comparison between any two computational models. Prior works such as Khaligh-Razavi and Kriegeskorte (2014); Yamins et al. (2014) have used RSA to find the correlation between visual cortex features and convolutional neural network features. Authors of Seminar (2016); Raghu et al. (2017); Morcos et al. (2018); Wang et al. (2018) have studied RSA between different neural networks. However, recent work (Kornblith et al., 2019) argues that none of the above methods for studying RSA can yield high similarity even between two different initializations of the same architecture. They further propose CKA, which has become a powerful evaluation tool for RSA and has been successfully applied in several studies. For example, Nguyen et al. (2020) analyzes the representation pattern in extremely deep and wide neural networks, and Raghu et al. (2021) studies the representation difference between convolutional neural networks and vision transformers with CKA. In this work, we leverage this tool to compare ANNs and SNNs.

## 3 Preliminary

### ANN and SNN Neurons

In this paper, vectors/matrices are denoted with bold italic/capital letters (_e.g._\(\mathbf{x}\) and \(\mathbf{W}\) denote the input vector and weight matrix, respectively). Constants are denoted by small upright letters. For non-linear activation function in artificial neurons, we use the rectified linear unit (ReLU) (Krizhevsky et al., 2012), given by \(\mathbf{y}=\max(0,\mathbf{W}\mathbf{x})\). As for the non-linear activation function in spiking neurons, we adopt the well-known Leaky Integrate-and-Fire (LIF) model. Formally, given a membrane potential \(\mathbf{u}^{(t)}\) at time step \(t\) and a pre-synaptic input \(\mathbf{i}^{(t+1)}=\mathbf{W}\mathbf{x}^{(t+1)}\), the LIF neuron will update as

\[\mathbf{u}^{(t+1),\text{pre}}=\tau\mathbf{u}^{(t)}+\mathbf{i}^{(t+1)},\quad\mathbf{y}^{(t+1)} =\begin{cases}1&\text{if }\mathbf{u}^{(t+1),\text{pre}}>v_{th}\\ 0&\text{otherwise}\end{cases},\quad\mathbf{u}^{(t+1)}=\mathbf{u}^{(t+1),\text{pre}} \cdot(1-\mathbf{y}^{(t+1)}). \tag{1}\]

Here, \(\mathbf{u}^{(t+1),\text{pre}}\) is the pre-synaptic membrane potential, \(\tau\) is a constant leak factor within \((0,1)\). Let \(v_{th}\) be the firing threshold, the LIF neuron will fire a spike (\(\mathbf{y}^{(t+1)}=1\)) when the membrane potential exceeds the threshold; otherwise, it will stay inactive (\(\mathbf{y}^{(t+1)}=0\)). After firing, the spike output \(\mathbf{y}^{(t+1)}\) will propagate to
the next layer and become the input \(\mathbf{x}^{(t+1)}\) of the next layer. Note that here the layer index is omitted for simplicity. The membrane potential will be reset to 0 if a spike fires (refer to Eq.1 the third sub-equation).

### Optimize SNN with Surrogate Gradient

To enable gradient descent for SNN, we adopt the BPTT algorithm (Werbos, 1990). Formally, denote the loss function value as \(L\), the gradient of the loss value with respect to weights can be formulated by

\[\frac{\partial L}{\partial\mathbf{W}}=\sum_{t=1}^{T}\frac{\partial L}{\partial \mathbf{y}^{(t)}}\frac{\partial\mathbf{y}^{(t)}}{\partial\mathbf{u}^{(t),\text{pre}}} \mathbf{K}^{(t)},\quad\text{where }\mathbf{K}^{(t)}=\left(\frac{\partial\mathbf{u}^{(t), \text{pre}}}{\partial\mathbf{i}^{(t)}}\frac{\partial\mathbf{i}^{(t)}}{\partial\mathbf{ W}}+\frac{\partial\mathbf{u}^{(t),\text{pre}}}{\partial\mathbf{u}^{(t-1)}}\frac{ \partial\mathbf{u}^{(t-1)}}{\partial\mathbf{u}^{(t-1),\text{pre}}}\mathbf{K}^{(t-1)} \right). \tag{2}\]

Here, the gradient is computed based on the output spikes from all time steps. In each time step, we denote \(\mathbf{K}\) as the gradient of pre-synaptic membrane potential with respect to weights \(\frac{\partial\mathbf{u}^{(t)}}{\partial\mathbf{W}}\), which consists of the gradient of pre-synaptic input and the gradient of membrane potential in the last time steps.

As a matter of fact, all terms in Eq.2 can be easily differentiated except \(\frac{\partial\mathbf{y}^{(t)}}{\partial\mathbf{u}^{(t),\text{pre}}}\) which returns a zero-but-all gradient (Dirac delta function). Therefore, the gradient descent ends up either freezing the weights or updating weights to infinity. To address this problem, the surrogate gradient is proposed (Bender et al., 2018; Wu et al., 2018; Bellec et al., 2018; Neftci et al., 2019; Li et al., 2021b) to replace the Dirac delta function with another function:

\[\frac{\partial\mathbf{y}^{(t)}}{\partial\mathbf{u}^{(t),\text{pre}}}=\frac{1}{\alpha} \mathbbm{1}_{|\mathbf{u}^{(t),\text{pre}}-v_{th}|<\alpha}, \tag{3}\]

where \(\alpha\) is a hyper-parameter for controlling the sharpness and \(\alpha=1\) the surrogate gradient becomes the Straight-Through Estimator (Bengio et al., 2013).

Compared to other methods such as ANN-SNN conversion (Deng and Gu, 2021) or spike-timing-dependent plasticity (Caporale et al., 2008), BPTT using surrogate gradient learning yields the best performance in image recognition tasks. However, from a biological perspective, BPTT is implausible: for each weight update, BPTT requires the use of the transpose of the weights to transmit errors backward in time and assign credit for how past activity affected present performance. Running the network with transposed weights requires the network to either have two-way synapses or use a symmetric copy of the feedforward weights to backpropagate the error (Marschall et al., 2020). Therefore, the question remains whether the representation in SNNs learned with surrogate gradient-based BPTT actually differs from the representation in ANNs.

### Centered Kernel Alignment

Let \(\mathbf{X}_{s}\in\mathbb{R}^{m\times Tp_{1}}\) and \(\mathbf{X}_{a}\in\mathbb{R}^{m\times p_{2}}\) contain the representation in an arbitrary layer of SNN with \(p_{1}\) hidden neurons across \(T\) time steps and the representation in an arbitrary layer of ANN with \(p_{2}\) hidden neurons, respectively. Here \(m\) is the batch size and we concatenate features from all time steps in the SNN altogether. We intend to use a similarity index \(s(\mathbf{X}_{s},\mathbf{X}_{a})\) to describe how similar they are. We use the Centered Kernel Alignment (CKA) (Kornbith et al., 2019) to measure this:

\[\text{CKA}(\mathbf{K},\mathbf{L})=\frac{\text{HSIC}(\mathbf{K},\mathbf{L})}{ \sqrt{\text{HSIC}(\mathbf{K},\mathbf{K})\text{HSIC}(\mathbf{L},\mathbf{L})}}, \quad\text{HSIC}(\mathbf{K},\mathbf{L})=\frac{1}{(m-1)^{2}}tr(\mathbf{KHLH}). \tag{4}\]

Here, \(\mathbf{K}=\mathbf{X}_{s}\mathbf{X}_{s}^{\top},\mathbf{L}=\mathbf{X}_{a} \mathbf{X}_{a}^{\top}\) are the Gram matrices as shown in Fig.1. Each Gram matrix has the shape of \(m\times m\), reflecting the similarities between a pair of examples. For example, \(\mathbf{K}_{i,j}\) indicates the similarity between the \(i^{th}\) and \(j^{th}\) example in the SNN feature \(\mathbf{X}_{s}\). Further measuring the similarity between \(\mathbf{K}\) and \(\mathbf{L}_{i}\) one can measure whether SNN has a similar inter-example similarity matrix with ANN. Let \(\mathbf{H}=\mathbf{I}-\frac{1}{m}\mathbf{1}\mathbf{1}^{\top}\) be the centering matrix, the Hilbert-Schmidt Independence Criterion (HSIC) proposed by Gretton et al. (2005) can conduct a test statistic for determining whether two sets of variables are independent. HSIC = 0 implies independence. The CKA further normalizes HSIC to produce a similarity index between 0 and 1 (the higher the CKA, the more similar the input pair) which is invariant to isotropic scaling. In our implementation, we use the unbiased estimator of HSIC (Song et al., 2012; Nguyen et al., 2020) to calculate it across mini-batches.
## 4 Do SNNs Learn Different Representation from ANNs?

In this section, we comprehensively compare the representation learned in SNNs and ANNs. Our primary study case is ResNet with identity mapping block (He et al., 2016) on the CIFAR10 dataset, which is the standard architecture and dataset in modern deep learning for image recognition. 1 There are two differences between our SNNs and ANNs. First, ANNs adopt the Batch Normalization layer (Ioffe and Szegedy, 2015), and SNNs use the time-dependent Batch Normalization layer (Zheng et al., 2020), which normalizes the feature across all time steps (_i.e._\(\mathbf{X}s\)). Second, the ANNs use ReLU activation, and SNNs leverage the LIF spiking neurons. For default SNN training, we use direct encoding, \(\tau=0.5\) for the leaky factor, \(vth=1.0\) for the firing threshold, \(T=4\) for the number of time steps, and \(\alpha=1.0\) for surrogate gradient, which are tuned for the best training performance on SNNs. Detailed training setup and codes can be found in the supplementary material.

Footnote 1: We also provide RSA on VGG-series networks in Sec. A.1 and RSA on CIFAR100 dataset in Sec. A.2.

### Scaling up Width or Depth

We begin our study by studying how the spatial dimension of a model architecture affects internal representation structure in ANNs and SNNs. We first investigate a simple model: ResNet-20, and then we either increase its number of layers or increase its channel number to observe the effect of depth and width, respectively. In the most extreme cases, we scale the depth to 164 and the width to \(16\times\) (see detailed network configuration in Table D.1). For each network, we compute CKA between all possible pairs of layers, includ

Figure 2: **CKA heatmap between SNNs and ANNs with different depth and width on the CIFAR-10 dataset. Top: the CKA cross-layer heatmap across different depths from 20 layers to 164 layers. Middle: the CKA cross-layer heatmap across different widths from the original channel number to 16 times. Bottom: visualizing only the corresponding layer, which is the diagonal of the CKA heatmap. We find generally SNNs and ANNs have relatively high similarity, and deeper/wider networks have positive/negative effects on the representation similarity.**
ing convolutional layers, normalization layers, ReLU/LIF layers, and residual block output layers. Therefore, the total number of layers is much greater than the stated depth of the ResNet, as the latter only accounts for the convolutional layers in the network. Then, we visualize the result as a heatmap, with the \(x\) and \(y\) axes representing the layers of the network, going from the input layer to the output layer. Following Nguyen et al. (2020), our CKA heatmap is computed on 4096 images from the test dataset.

As shown in Fig. 2, the CKA heatmap emerges as a checkerboard-like grid structure, especially for the deeper neural network. In ResNet-20, we observe a bright block in the middle and deep layers, indicating that ANNs and SNNs learn overlapped representation. As the network goes deep, we find the CKA heatmap becomes darker, meaning that representations in ANNs and those in SNNs are diverging. Notably, we find a large portion of layers in artificial ResNet-164 exhibit significantly different representations than spiking ResNet-164 (\(<\)0.2 CKA value) which demonstrates that deeper layers tend to learn disparate features.

In Fig. 2 middle part, we progressively enlarge the channel number of ResNet-20. In contrast to depth, the heatmap of wider neural networks becomes brighter, which indicates the representations in SNNs and ANNs are converging. Interestingly, although the majority of layers are learning more similar representations between ANN and SNN in wide networks, the last several layers still learn different representations.

We further select only the diagonal elements in the heatmap and plot them in Fig. 2 bottom part. Because SNNs and ANNs have the same network topology, this visualization is more specific and may accurately reveal the similarity between SNNs and ANNs at each corresponding layer. First, we can find that in Fig. 2 the CKA curve of ResNet-20 shows relatively high values. Most layers go above 0.5 and some of them can even reach nearly 1.0. Interestingly, we observe that deeper networks tend to derive a curve with a jagged shape. This means some layers in SNN indeed learn different representations when compared to ANN, however, _the difference is intermittently mitigated_. In later sections, we will show that the mitigation of dissimilarity is performed by residual connection. As for width, we generally notice that CKA curves mostly become higher for wider networks, especially when comparing ResNet-20 and ResNet-20 \(\times\)8, where most layers have above 0.8 CKA value.
similarity since it is a linear transformation. These results substantiate that _the convolutional layers and LIF layers in SNNs are able to learn different representations than ANNs. However, the representation in the residual branch still dominates the representation in post-residual layers and leads to the junction of ANN's and SNN's representation_.

To further explore why residual connections can restore the representations in SNNs to ANN-like, we conduct an ablation study. We select one of the three stages in the spiking ResNet-56 where the residual connections are disabled selectively. In Fig. 4, we visualize the CKA heatmaps of SNN itself, which means both \(x\) and \(y\) axes are the same layers in SNN. The first heatmap demonstrates the full residual network, while the remaining three heatmaps show the partial residual networks, with the 1st, 2nd, and 3rd stages disabled, respectively. Our observations can be summarized as follows: (1) In terms of inter-stage similarity, residual connections can preserve the input information from the previous stage. In the 1st and 2nd heatmaps in Fig. 4, we find residual blocks can have high similarity with their former stage. The non-residual block, however, does not have this property. In the 3rd and 4th heatmaps, we can see that blocks without residual connections exhibit significantly different representations when compared to their former stage. Therefore, we can find that residual connections preserve the representation in early layers. As such, if ANN and SNN learn similar representations in the first layer, the similarity can propagate to very deep layers due to residual connections. (2) In terms of intra-stage similarity, the non-residual stage's heatmap appears with a uniform representation across all layers, meaning that layers in this stage are similar. In contrast, residual stages share a grid structure.

Next, we verify the accuracy of SNNs and ANNs when both are equipped with residual connections or not, under different network depths. As shown in Table 2, both the SNNs and ANNs can successfully train very

Figure 4: **The effect of residual connections in SNNs.** We remove residual connections in one of three stages in the ResNet-56 and show the CKA heatmaps. The non-residual stage is annotated with green square \(\square\).

Figure 3: **Emergence of periodic jagged CKA curve. Left**: CKA curve of ResNet-110. We subplot the 10-\(th\) and the 34-\(th\) residual blocks in ResNet-110, which forms a periodic jagged curve. **Top right**: The averaged CKA value in all blocks, **Bottom right**: The architecture specification of the residual block we used.
deep networks if the residual connections are enabled. In this case, though SNNs do not surpass the accuracy of ANNs, the gap is relatively small, with 1\(\sim\)2% accuracy degradation. However, if the residual connections are removed from SNNs, the gap between the accuracies of ANNs and SNNs significantly enlarges, ranging from 5\(\sim\)56%. Therefore, we can conclude that _the residual connections help the gradient descent optimization in SNNs and regularize the representations in SNNs to be similar to those in ANNs so that SNNs can have similar task performances with ANNs._

### Scaling up Time Steps

The results of the previous sections help characterize the effects of spatial structure on internal representation differences between SNNs and ANNs. Next, we ask whether the time dimension helps SNN learn some unique information. To verify this, we train several spiking ResNet-20 with 4/8/16/32/64/128 time steps and calculate the ANN-SNN CKA similarity. In Fig. 5, we envision the CKA heatmaps and curves respectively between artificial ResNet-20 and spiking ResNet-20 with various time steps. Notably, we cannot find significant differences among these heatmaps. Looking at the CKA curves, we also discover that many layers are overlapped, especially when we focus on the residual block output (the local maximums). We find similarities between different time steps reaching the same point, meaning that the time step variable does not provide much unique representation in SNNs.

To further analyze the representation along the time dimension in SNNs, we compare the CKA among various time steps. Concretely, for any layer inside an SNN, we reshape the feature \(\mathbf{X}_{s}\) to \([\mathbf{X}^{(1)},\mathbf{X}^{(2)},\dots,\mathbf{X}^{(T)}]\) where \(\mathbf{X}^{(i)}\) is the _i-th_ time step's output. By computing the CKA similarity between arbitrary two time steps, _i.e.,_\(\text{CKA}(\mathbf{X}^{(i)},\mathbf{X}^{(j)})\), we are able to construct a CKA heatmap with \(x,y\) axes being the time dimension, which demonstrates whether the features are similar across different time steps. Fig. 6 illustrates such CKA heatmaps of outputs from all residual blocks in the spiking ResNet-20, with time steps varying from 4 to 32. In general, deeper residual block output exhibits darker CKA heatmaps and the shallower layers tend to become yellowish. In particular, all the residual blocks from the first stage have an all-yellow CKA heatmap,

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Type**} & \multicolumn{4}{c}{**Depth**} \\ \cline{3-6}  & & 20 & 38 & 56 & 74 \\ \hline \multirow{2}{*}{ANN} & w/. residual connection & 91.06 & 92.34 & 92.98 & 92.85 \\  & w/o residual connection & 91.32 & 91.17 & 89.62 & 21.07 \\ \hline \multirow{2}{*}{SNN} & w/. residual connection & 89.63 & 91.14 & 91.94 & 91.83 \\  & w/o residual connection & 86.50 & 82.64 & 33.61 & 10.00 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **The impact of residual connections on accuracy.**

Figure 5: **The effect of time steps in SNNs. Left: CKA heatmaps between ANNs and SNNs with the different number of time steps. Right: The CKA curve of corresponding layers (diagonal values as in left).**
indicating extremely high similarity in these blocks. The second stage starts to produce differences across time steps, but they still share \(>\)0.8 similarities between any pair of time steps. The last stage, especially the last block, demonstrates around 0.5 similarities between different time steps. To summarize, the impact of time in SNN is gradually increased as the feature propagates through the network. In Appendix A.4, we provide the heatmap of convolutional/LIF layers and find a similar trend.

We further conduct an empirical verification to verify the findings in Fig. 6. More specifically, we define a sensitivity metric and measure it by reducing the number of time steps to 1 in certain layers of an SNN and recording the corresponding accuracy degradation. In Fig. 6 we find the first stage (s1) has the same representation in time dimension while the last stage (s3) exhibits a more diverse representation. Therefore, we choose to reduce the number of time steps to 1 either in s1 or in s3. To achieve this "mixed-time-step SNN", we repeat/average the activation in time dimension after s1/before s3 to match the dimension. Table 3 summarizes the sensitivity results. We observe that the first stage can have much lower accuracy degradation (\(<\)1%), while the last stage drop 2\(\sim\)4% accuracy. Moreover, if the last stage only uses 1 time step, then increasing the time steps for the other two stages does not benefit the accuracy at all. This indicates that LIF neurons are more effective in deep layers than in shallow layers.

### Representation under Event Data

In this section, we evaluate the CKA similarity on the event-based dataset. We choose CIFAR10-DVS (Li et al., 2017), N-Caltech 101 (Orchard et al., 2015) and train spiking/artificial ResNet-20. Since the ANN cannot process 4-dimensional spatial-temporal event data easily, we integrate all events into one frame for ANN training and 10 frames for SNN training. Fig. 7 provides the CKA heatmaps/curves on the event dataset, showing a different similarity distribution than the previous CIFAR-10 dataset. The heatmaps have a different pattern and the curves also do not appear with a periodic jagged shape. In addition, the similarity distribution differs at the dataset level, i.e., the CIFAR10-DVS and N-Caltech 101 share different

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Type**} & \multicolumn{4}{c}{**\# Time Steps**} \\ \cline{3-6}  & & 4 & 8 & 16 & 32 \\ \hline \multirow{3}{*}{SNN} & Full time steps & 89.67 & 90.44 & 90.98 & 90.99 \\  & Reduce the time steps in the first stage & **88.81** & **89.70** & **89.91** & **90.47** \\  & Reduce the time steps in the last stage & 87.41 & 87.78 & 87.38 & 87.73 \\ \hline \hline \end{tabular}
\end{table}
Table 3: **The sensitivity of time steps in SNNs.**

Figure 6: **The similarity across times in SNN.** Each heatmap shows the CKA among different time steps in the output of the residual block. “s” means stage, and “b” means block. The top/middle/bottom rows are spiking ResNet-20 with 4/8/32 time steps, respectively.
CKA curves and heatmaps. On N-Caltech 101, the SNN learns different feature representations compared with ANN in shallow and deep layers, but similar representation in intermediate layers. For CIFAR10-DVS, the similarity continues to decrease from 0.9 to 0.5 as the layers deepen. In summary, with the event-based dataset, SNNs and ANNs share a different CKA pattern in comparison to the natural image dataset, which implies that SNNs may have further optimization space in this type of dataset. We put more CKA results on various models for the CIFAR10-DVS dataset in Appendix A.3.

### Representation under Adversarial Attack Data

We next study the adversarial robustness of SNN and ANN using CKA. Inspired by quantized ANNs that are robust to adversarial attack Lin et al. (2019), the SNNs could also inherit this property since their activations are also discrete. Previous works have explored understanding the inherent robustness of SNNs (Sharmin et al., 2020; Kundu et al., 2021; Liang et al., 2021; Kim and Panda, 2021). However, they either evaluate on converted SNN or using rate-encoded images. Here, we test Projected Gradient Descent (PGD) attack (Madry et al., 2017) on the directly trained SNN and ANN using direct encoding. Formally, we generate the adversarial images by restricting the \(L\)-infinity norm of the perturbation, given by

\[\mathbf{x}_{adv}^{k+1}=\Pi_{P_{e}(\mathbf{x})}(\mathbf{x}_{adv}^{k}+\alpha\text{sign}( \nabla_{\mathbf{x}_{adv}^{k}}L(\mathbf{x}_{adv}^{k},\mathbf{w},\mathbf{y}))), \tag{5}\]

where \(\mathbf{x}_{adv}^{k}\) is the generated adversarial sample at the \(k\)-\(th\) iteration. \(\Pi_{P_{e}(\mathbf{x})}(\cdot)\) projects the generated sample onto the projection space, the \(\epsilon-L_{\infty}\) neighborhood of the clean sample. \(\alpha\) is the attack optimization step size. With higher \(\epsilon\), the adversarial image is allowed to be perturbed in a larger space, thus degrading task performance.

We evaluate the performance of spiking and artificial ResNet-20 on the CIFAR-10 dataset with \(\epsilon\) values ranging from \(0.001,0.005,0.01,0.02,0.05\) to generate adversarial images. We then compute the CKA value between the features of the clean images and the adversarially corrupted images. The results are summarized in Fig. 8 (left). We find that although the clean accuracy of ANN is higher than that of SNN, SNN has higher robustness against attacks. For example, the PGD attack with 0.01 \(L\)-infinity norm perturbation reduces the accuracy of ANN by 43%, while only reducing the accuracy of SNN by 22%. We also investigate the CKA similarity between clean and adversarial images, as shown in the second and third subplots of Fig. 8. We observe that the higher the robustness against adversarial attacks, the higher the similarity between clean and corrupted images. This intuition is confirmed by the CKA curves, which show that SNN has a higher similarity than ANN. We also observe several interesting phenomena. For example, the ANN

Figure 8: **The robustness against adversarial attack. Left: The accuracy of SNN and ANN after attack under different \(\epsilon\). Right: The CKA curve between clean images and adversarial images of ANN and SNN, respectively.**

Figure 7: **CKA Similarity on Event Dataset. We train spiking and artificial ResNet-20 on CIFAR10-DVS and N-Caltech 101, respectively. Left: the CKA heatmaps. Right: The CKA curves of corresponding layers between ANN and SNN.**
suffers a large decrease in similarity in the first block, even with a small \(\epsilon\) value of 0.001. Additionally, when we focus on the purple line (\(\epsilon=0.02\)), we notice that ANN and SNN have similar perseverance in earlier layers, but ANN drops much more similarity than SNN in the last block. These results provide insight into model robustness and suggest that SNNs are more robust than ANN, especially in their shallow and deep layers.

## 5 Discussion and Conclusion

Given that SNNs are drawing increasing research attention due to their bio-plausibility and recent progress in task performance, it is necessary to verify if SNNs, especially those trained with the surrogate gradient algorithms, can or have the potential to truly develop desired features different from ANNs. In this work, we conduct a pilot study to examine the internal representation of SNNs and compare it with ANNs using the popular CKA metric. This metric measures how two models respond to two different examples. Our findings can be briefly summarized as follows:

1. Generally, the layer-wise similarity between SNNs and ANNs is high, suggesting SNNs trained with surrogate gradient learn similar representation with ANNs. Moreover, wider networks like ResNet-20 8\(\times\) can even have \(>0.8\) similarities for almost all layers.
2. For extremely deep ANNs and SNNs, the CKA value would become lower, however, the residual connections play an important role in regularizing the representations. By conducting ablation studies, we demonstrate that the residual connections make SNNs learn similar representations with ANNs and help SNNs achieve high accuracy.
3. The time dimension does not provide much additional representation power in SNNs. We also demonstrate that the shallow layers learn completely static representation along the time dimension. Even reducing the number of time steps to 1 in shallow layers does not significantly affect the performance of SNNs.
4. On other types of datasets, SNNs may develop less similar representations with ANNs, _e.g.,_ event-data.

Our results show that SNNs optimized by surrogate gradient algorithm do not learn distinct spatial-temporal representation compared to the spatial representation in ANNs. Current SNN learning relies on the residual connection and wider neural networks (for example, Zheng et al. (2020) use ResNet-19 which is similar to our ResNet-20 8\(\times\)) to obtain decent task performance. However, our study suggests that this task performance is highly credited to the similar representation with ANN. Furthermore, the time dimension brings limited effect to the SNN representation on static datasets like CIFAR10 and CIFAR100. In particular, the first stage of ResNets results in quite similar representation across time steps.

Nonetheless, our study is not a negation against SNNs. Our results are based on the surrogate-gradient BPTT optimization, which, as aforementioned, is inherently bio-implausible and resembles the optimization method for ANNs. Therefore, it may not be surprising to see SNNs and ANNs have similar representations under a similar optimization regime. Additionally, we find that input data is also important in developing the representations. Indeed, the direct encoding used in SNNs inputs the same static images for each time step, again leading to less gap between the representations of ANNs and SNNs.

Here, we provide several directions worth studying in the future: a) Bio-plausible learning rule for SNNs: surrogate gradient training tends to learn an ANN-like representation in SNN, thus it is necessary to develop an optimization method that suits SNN better. b) Spiking architecture design: a specialized SNN network architecture may avoid learning similar representation, _e.g.,_ Kim et al. (2022). c) Understanding the robustness of SNN: adversarial attack is inconsequential for human visual systems, which may be reflected in SNN as well. We believe the SNN robustness can be significantly improved. To conclude, our work tries to understand the representation of SNNs trained with surrogate gradient and reveals some counter-intuitive observations. We hope our work can inspire more research in pushing the limit of SNNs.

**Acknowledgements.** This work was supported in part by CoCoSys, a JUMP2.0 center sponsored by DARPA and SRC, Google Research Scholar Award, the National Science Foundation CAREER Award, TII (Abu Dhabi), the DARPA AI Exploration (AIE) program, and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198).