Deep recurrent spiking neural networks capture both static and dynamic
  representations of the visual cortex under movie stimuli
In the real world, visual stimuli received by the biological visual system
are predominantly dynamic rather than static. A better understanding of how the
visual cortex represents movie stimuli could provide deeper insight into the
information processing mechanisms of the visual system. Although some progress
has been made in modeling neural responses to natural movies with deep neural
networks, the visual representations of static and dynamic information under
such time-series visual stimuli remain to be further explored. In this work,
considering abundant recurrent connections in the mouse visual system, we
design a recurrent module based on the hierarchy of the mouse cortex and add it
into Deep Spiking Neural Networks, which have been demonstrated to be a more
compelling computational model for the visual cortex. Using Time-Series
Representational Similarity Analysis, we measure the representational
similarity between networks and mouse cortical regions under natural movie
stimuli. Subsequently, we conduct a comparison of the representational
similarity across recurrent/feedforward networks and image/video training
tasks. Trained on the video action recognition task, recurrent SNN achieves the
highest representational similarity and significantly outperforms feedforward
SNN trained on the same task by 15% and the recurrent SNN trained on the image
classification task by 8%. We investigate how static and dynamic
representations of SNNs influence the similarity, as a way to explain the
importance of these two forms of representations in biological neural coding.
Taken together, our work is the first to apply deep recurrent SNNs to model the
mouse visual cortex under movie stimuli and we establish that these networks
are competent to capture both static and dynamic representations and make
contributions to understanding the movie information processing mechanisms of
the visual cortex.
Liwei Huang, ZhengYu Ma, Huihui Zhou, Yonghong Tian
Deep recurrent spiking neural networks capture both static and dynamic representations of the visual cortex under movie stimuli

###### Abstract

In the real world, visual stimuli received by the biological visual system are predominantly dynamic rather than static. A better understanding of how the visual cortex represents movie stimuli could provide deeper insight into the information processing mechanisms of the visual system. Although some progress has been made in modeling neural responses to natural movies with deep neural networks, the visual representations of static and dynamic information under such time-series visual stimuli remain to be further explored. In this work, considering abundant recurrent connections in the mouse visual system, we design a recurrent module based on the hierarchy of the mouse cortex and add it into Deep Spiking Neural Networks (SNNs), which have been demonstrated to be a more compelling computational model for the visual cortex. Using a similarity metric, Time-Series Representational Similarity Analysis (TSRSA), we measure the representational similarity between networks and mouse cortical regions under natural movie stimuli. Subsequently, we conduct a comparison of the representational similarity across recurrent/feedforward networks and image/video training tasks. Trained on the video action recognition task, the recurrent SNN achieves the highest representational similarity and significantly outperforms the feedforward SNN trained on the same task by 15% and the recurrent SNN trained on the image classification task by 8%. Based on the similarity experiments, we further investigate how static and dynamic representations of SNNs influence the similarity, as a way to explain the importance of these two forms of representations in biological neural coding. Taken together, our work is the first to apply deep recurrent SNNs to model the mouse visual cortex under movie stimuli and we establish that these networks are competent to capture both static and dynamic representations and make contributions to understanding the movie information processing mechanisms of the visual cortex.

## 1 Introduction

When observing the natural world, the biological visual system receives not only static but also dynamic information and integrates this information in both spatial [22; 23] and temporal [19] dimensions to encode and transmit information and perform visual tasks. Although bottom-up (feedforward) connections dominate the processing and transmission of visual information [13; 9], top-down (feedback) and lateral connections, which are also widespread in the biological visual cortex, play a crucial role [16; 31; 47]. Recurrences in the visual cortex not only provide diverse coding mechanisms for the visual system and enrich the temporal representation protocol [51; 14; 24; 52], but also facilitate the performance in complex and challenging visual tasks [18; 49; 55; 56].
Deep neural networks have become the tool of choice in the visual neuroscience community [28; 32; 58], compared with traditional computational models. They have shown significant utility in investigating various aspects of the biological visual cortex, such as rivaling the neural representations [59; 5; 4; 7; 6; 40], revealing the functional hierarchy [17; 44; 8; 54; 45], and understanding the processing mechanisms [1; 10]. As research has progressed, more brain-inspired structures and mechanisms have been introduced into the neural networks to better model the visual cortex. On the one hand, incorporating recurrences such as lateral and feedback connections in networks effectively improves the ability of networks to capture more brain-like representations and behavioral patterns [39; 35; 29; 41]. On the other hand, spiking neural networks [37] with computational mechanisms more similar to the brain have been developed as a more biologically plausible alternative [20; 15; 25; 3]. [21] has demonstrated that deep SNNs outperform their counterparts of traditional ANNs on several neural datasets.

In this work, we combine deep SNNs and recurrences to exploit the biological potential of SNNs and design a series of experiments to compare different deep SNNs based on representational similarity, aiming to elucidate the importance of static and dynamic representations of the visual cortex under movie stimuli (Figure 1). We summarize our main contributions in four points as follows.

* We design a recurrent module for deep SNNs inspired by certain properties of SNN and the functional hierarchy of the mouse visual cortex. We incorporate this recurrent module into SEW-ResNet [12] and pretrain it on the UCF101 dataset. Notably, this network achieves the highest level of representational similarity to the mouse visual cortex under movie stimuli.
* By quantifying the effect of dynamic representations on representational similarity, we demonstrate that the recurrent module largely stimulates the ability of SNNs to extract temporal features and capture the dynamic representations of the mouse visual cortex.
* By quantifying the effect of static representations on similarity, we reveal that the recurrent SNN also captures static representations of the mouse visual cortex, and its ability to represent dynamic information improves its robustness to disparate static information.
* In the two quantification experiments above, the representational similarity drops significantly in the case that either static or dynamic information is corrupted, which shows that both static and dynamic representations are important components of the neuronal population responses in the mouse visual cortex.

Overall, to the best of our knowledge, we are the first to use deep recurrent SNNs to investigate the representations of the mouse visual cortex under movie stimuli. We provide computational evidence for the importance of static and dynamic representations in the mouse visual cortex, and that the recurrent SNNs we design are capable of representing both types of information well and become an effective and novel computational model for revealing the information processing mechanisms of the visual system.

## 2 Related Work

Modeling the visual cortex with recurrent ANNsSince recurrent connections are critical structures in the brain, [26; 27] provided physiological and computational evidence that recurrences contribute to the temporal properties of visual coding. Either by automated search or by manual design, some work has constructed novel networks with recurrent connections [39; 35; 29; 53], which not only better fit the neural responses to static image stimuli but also reveal the neural dynamics of the visual cortex. What's more, as for movie stimuli, [46; 48] accurately predicted the neural representations using recurrent neural networks. However, most work has focused on investigating neural representations to static stimuli, a limited aspect of naturalistic stimuli. The exploration of static and dynamic representations to movie stimuli is still scarce. Our work is dedicated to the analysis of static and dynamic representations in networks to shed light on visual processing mechanisms.

The SNNs with recurrent connectionsMost studies applied spiking neurons and recurrent connections in single-layer networks to perform some simple temporal tasks in the early days [30; 2]. Recently, [60; 42] introduced diverse recurrent connections within layers and within spiking neurons in multi-layer networks, yielding considerable performance on sequential tasks. Moreover, recurrent
SNNs have also been used to explain the regime in the brain associated with cognition and working memory [50; 57]. With recurrent SNN models, [36] emphasized the criticality of homeostatic regulation in biological neurons. Although these recurrent SNN models have made a large contribution to the study of the brain, they have been limited to shallow or even single-layer networks while focusing on some local and detailed biological properties. In contrast, our work is the first to add long-term feedback connections to deep SNNs and provide a comprehensive comparison with large-scale neuronal population representations.

## 3 Methods

### Neural dataset

In this work, we conduct analysis using a subset of the Allen Brain Observatory Visual Coding dataset [47]. This dataset, recorded by Neuropixel probes, consists of neural spikes with high temporal resolution from 6 mouse visual cortical regions (VISp, VISl, VISrl, VISal, VISpm, VISam). Each cortical region contains hundreds of recorded neurons to minimize the effects of individual neuronal variability, facilitating the analysis of neural population representations. The visual stimulus presented to mice is a 30-second natural movie with 900 frames and the stimulus is repeated for 20 times. For each neuron, we sum the number of spikes in each movie frame and take the average across all repeats as the neural response. What's more, we exclude neurons that fire less than 0.5 spikes/s.

### Deep spiking neural networks with recurrent connections

Although deep SNNs achieve higher representational similarity to biological visual neural responses compared to traditional ANNs, the experiments are all based on static stimuli [21]. Consequently, in order to better match the representations of the mouse visual cortex under movie stimuli, we introduce feedback connections in deep SNNs to enhance their temporal encoding capability and design a recurrent module suitable for them based on some hierarchical properties of the mouse cortex.

The recurrent module consists of three components: a _feedforward_ module (\(O_{t}^{l}\)), a _feedback_ module (\(R_{t}^{l}\)), and a _fusion_ module (\(A_{t}^{l}\)). The _feedforward_ module is a submodule of the backbone network that plays a main role in abstracting spatial features from visual stimuli and encoding the visual content. Unlike feedforward networks, where the submodule receives the outputs of the previous stage directly, the _feedforward_ module receives the fused features of the outputs from the _feedback_ module and the previous stage. The _feedback_ module is composed of depthwise transposed convolution, batch normalization, and spiking neurons. On the one hand, depthwise transposed convolution effectively

Figure 1: The overview of our experiments. Six mouse visual cortex and recurrent spiking neural networks receive the same movie stimuli to generate the representation matrices. The metric TSRSA is applied to two representation matrices to quantify the representational similarity. In addition, two more experiments are used to quantify the effect of static and dynamic representations on similarity.
reduces the number of network parameters and upsamples the feature map to match the inputs. On the other hand, some work demonstrated that such a structure might mimic parallel information processing streams in mouse cortical regions and improves the representational similarity between the model and visual cortex [21]. The _fusion_ module first concatenates the inputs of the current module and the outputs of the _feedback_ module in the channel dimension, and then integrates the feedforward and feedback information through pointwise convolution, batch normalization, and spiking neurons. The recurrent module can be formulated as:

\[R_{t}^{l} =\mathrm{SN}(\mathrm{BN}(\mathrm{DW}(O_{t-1}^{l}))), \tag{1}\] \[A_{t}^{l} =\mathrm{SN}(\mathrm{BN}(\mathrm{PW}(\mathrm{CONCAT}(I_{t}^{l},R_ {t}^{l})))),\] (2) \[O_{t}^{l} =\mathrm{F}^{l}(A_{t}^{l}), \tag{3}\]

where \(\mathrm{SN}\) is spiking neurons, \(\mathrm{BN}\) is batch normalization, \(\mathrm{DW}\) is depthwise transposed convolution, \(\mathrm{PW}\) is pointwise convolution, and \(\mathrm{F}^{l}\) denotes all operations in the _feedforward_ module. \(O_{t}^{l}\) is not only the outputs of _feedforward_ module, but also the outputs of the entire recurrent module.

In this work, we use SEW-ResNet18 [12] as the backbone of our networks and apply recurrent connections to each stage of SEW-ResNet. For controlled experiments, networks with and without recurrent connections are all pretrained on both the ImageNet dataset and the UCF101 dataset by SpikingJelly [11]. Specifically, as for the training of the object recognition task on the ImageNet, each sample (an image) is input into SNNs for 4 times (the simulating time-steps \(T=4\)). As for the training of the video action recognition task on the UCF101, each sample (a video clip) contains 16 frames and one frame is input at each time step (the simulating time-steps \(T=16\)).

In order to extract network representations for comparison with the mouse visual cortex, we feed the same movie used in the neural dataset to pretrained SNNs and obtain features from all selected layers. It is worth noting that each movie frame is considered as a separate image to be fed to ImageNet-pretrained SNNs for 4 times (consistent with the training procedure), while the entire movie is continuously and uninterruptedly fed to UCF101-pretrained SNNs.

### Analysis of representation

#### 3.3.1 Representational similarity metric

To analyze the representational similarity between neural networks and the mouse visual cortex at the population level under time-series stimuli, we should solve two main problems. Firstly, although the Allen Brain Observatory Visual Coding dataset is a massive dataset, neurons recorded in a cortical region are far fewer than units of a network layer's feature maps, making it difficult to

Figure 2: SEW-ResNet (the left), R-SEW-ResNet (the centre), and the recurrent module (the right).
directly compare representations of two systems. Secondly, we should use a metric that not only analyzes static properties in representations, but also preserves sequential relationships of time-series representations and analyzes dynamic properties. Here, we use the Time-Series Representational Similarity Analysis (TSRSA) based on Representational Similarity Analysis (RSA) [34, 33] which has been widely used for the comparison of neural representations [29, 38, 44, 1, 7]. The original RSA focuses on the similarity between the neural representations to each pair of independent stimuli, whereas TSRSA quantifies the similarity among representations to time-series stimuli, taking into account temporal sequential relationships. We summarize the implementation of TSRSA as follows. First, through the preprocessing procedure for the neural dataset and the feature extraction procedure for neural networks, we obtain representation matrices \(R\in\mathbb{R}^{N\times M}\) from every layer of networks and every cortical region, where \(N\) is the number of units/neurons and \(M\) is the number of movie frame stimuli in chronological order. The column \(r_{t}\) of the representation matrix represents population responses to the movie frame \(t\). Secondly, for each column, using the Pearson correlation coefficient, we calculate the similarity between it and all subsequent columns one by one, yielding the representational similarity vector \(s_{t}\). The element \(s_{tp}\) of the similarity vector is \(Corr(r_{t},r_{t+p})\), where \(0<p<M-t\). Subsequently, we concatenate all vectors to obtain the complete representational similarity vector, which characterizes both the static and dynamic representations carried by a network layer or a cortical region. Finally, The Spearman rank correlation coefficient is used to quantify the similarity between two given vectors, which is also regarded as the representational similarity between two systems.

Using this metric, we apply a layer-by-layer comparison between a network and the mouse visual cortex, yielding the representational similarity score between each layer and each region. As for a network and a cortical region, we take the maximum score across network layers as the level of representational similarity between them. As for a network and the mouse visual cortex, we take the average similarity across all cortical regions as the final similarity.

#### 3.3.2 Quantification of the effect of dynamic representations on representational similarity

To explore the impact of dynamic representations on the representational similarity between SNNs and the mouse visual cortex, we disrupt the frame order of the movie used in the neural dataset, feed the shuffled movie into SNNs, reorder the frames back together with their corresponding network responses, and calculate the representational similarity while the stimuli are aligned in the representation matrices of the mouse and networks. As a result, the obtained network features differ from the original ones due to distinct dynamic sequential information. In order to obtain frame order with different levels of chaos while avoiding extreme chaos, such as the original first frame being shifted to the last frame, we divide the entire movie into multiple windows with the same number of frames and randomly shuffle the frames only within each window. Considering that the entire movie contains 900 frames, we conduct 10 sets of experiments, and the number of frames per window used in each set is 10, 20, 30, 40, 50, 80, 100, 150, 200 and 300 respectively. Each set comprises 10 trials to gain enough statistical power. What's more, we calculate the level of chaos for every trial, which is defined as \(1-r\), where \(r\) is the Spearman rank correlation coefficient between the disrupted frame order and the original frame order. Obviously, even in the same set of experiments, the level of chaos may vary across trials.

Also, although the network is fed a movie with shuffled frames to extract features, the representation matrix of the network is rearranged to the original frame order to ensure that it matches the order of the mouse representation matrix. In this way, we maintain the correspondence between the static representations of two systems, while modifying the dynamic representations of the network. This allows us to focus specifically on evaluating the effect of dynamic information on representational similarity.

#### 3.3.3 Quantification of the effect of static representations on representational similarity

In addition to analyzing the dynamic representations' impact, we also investigate the effect of static representations. Firstly, some movie frames are randomly selected and replaced with Gaussian noise images. Then, we feed the new movie into SNNs to obtain different features from the original movie. Similar to the experiments described in Section 3.3.2, the movie is divided into multiple windows with the same number of frames, and then we randomly replace one frame in each window with a noise image to avoid dense local replacement and to preserve as much dynamic information as
possible before and after the replaced frames. There are also 10 sets of experiments, each with 10 trials. The number of frames per window used in each set is 5, 10, 20, 30, 50, 60, 90, 100, 150 and 300 respectively. The ratio of replacement is the inverse of the number of frames per window.

Replacing movie frames results in a change in the static representations of the network, while the overall frame order remains the same as the original movie, allowing the network to maintain the original dynamic representations. Admittedly, noise images may lead to more disparate dynamic representations when the ratio of replacement is large. We attenuate this influence by sporadically distributing the noise images as much as possible and emphasis on how the static representations affect the representational similarity.

## 4 Results

### Comparisons of representational similarity

We compare the representational similarity across different network structures and different training tasks (Figure 3). For SNNs trained on the ImageNet, SEW-ResNet18 and R-SEW-ResNet18 achieve comparable representational similarity across all mouse cortical regions, which suggests that the recurrent module does not have a significant effect on the representations of networks trained on static images. However, when trained on the UCF101, recurrent SNNs perform consistently better than feedforward SNNs trained on the same task and all SNNs trained on the ImageNet. Specifically, R-SEW-ResNet18 trained on the UCF101 outperforms SEW-ResNet18 trained on the same dataset by 15% on average across all cortical regions and outperforms R-SEW-ResNet18 trained on the ImageNet by 8%. On the one hand, these results show that recurrent SNNs are better at extracting temporal features than feedforward SNNs when trained on a movie dataset. On the other hand, training with a movie dataset results in richer dynamic representations of recurrent SNNs compared to training with an image dataset, which is effective in improving the representational similarity with

Figure 4: The curves of representational similarity with SNN layer depth. The SNN layer depth is normalized from 0 (the first layer) to 1 (the last layer). To obtain smoother curves, we apply the cubic spline interpolation to calculate the similarity on 50 discrete depth points.

Figure 3: The representational similarity of SNNs for six mouse cortical regions. Each bar indicates the score of an SNN pretrained on a given dataset. The prefix “R” indicates that the SNN is embedded with the recurrent module.
the visual cortex. Nevertheless, feedforward SNNs trained on the UCF101 instead perform worse than those trained on the ImageNet. One explanation is that feedforward SNNs trained on a movie dataset not only fail to effectively capture dynamic information, but also compromise the ability to characterize static information. Furthermore, SEW-ResNet50 with the recurrent module yields the same positive effect (see Appendix).

We further analyze the curves of representational similarity across the SNN layers (Figure 4). It can be seen that the similarity curves of each network are very similar across six cortical regions, suggesting that the functional hierarchy of the mouse visual cortex may be organized in parallel, in line with the findings of some previous work [44, 21]. As for SNNs trained on the ImageNet, the similarity keeps trending upwards, reaching the maximum in almost the last layer. Differently, the similarity of R-SEW-ResNet18 trained on the UCF101 reaches the maximum in the middle layers and then decreases. As the results show, high-order features of SNNs trained on the ImageNet have similar static representations for neural responses of the mouse visual cortex under movie stimuli, while the middle-order features of recurrent SNNs trained on the UCF101 better capture both static and dynamic representations. Unexpectedly, the similarity of SEW-ResNet18 trained on the UCF101 shows a weird curve, peaking in the early layers, then dropping to a trough, and gradually rising again. Combined with the poor performance, SEW-ResNet18 trained on the UCF101 is bad at capturing the static and dynamic representations of the mouse visual cortex and is unable to match the functional hierarchy.

Taken together, the results suggest that static and dynamic representations are indeed included in the population responses of the mouse visual cortex to movie stimuli. The ability of feedforward SNNs to extract temporal features is far from adequate when relying only on the dynamics of spiking neurons, while the recurrent module endows SNNs with the greater capability to extract temporal features. What's more, after pretraining on the UCF101, recurrent SNNs capture both static and dynamic representations as well as achieve the highest representational similarity to the mouse visual cortex.

Figure 5: The curves of representational similarity (the main plot) and the curves of drop rate of the experimental similarity compared to the original similarity (the subplot). The horizontal coordinates of both plots are the level of chaos. In the main plot, the dashed horizontal lines are the representational similarity between SNNs and the mouse visual cortex under the original frame order. Each set of experiments contains 10 trials and each trial is indicated by one point. There is one ellipse for each set, and the height and width of the ellipse indicate the 95% confidence interval of the similarity and the level of chaos across 10 trials, respectively. The curves pass the average point of each set. In the subplot, the curves show the average drop rate and the average level of chaos across 10 trials for all experiment sets.
### Effect of dynamic representations on representational similarity

To investigate the importance of dynamic representations on representational similarity, following Section 3.3.2, we compare the experimental results of SEW-ResNet18 and R-SEW-ResNet18 trained on the UCF101 (Figure 5). As the curves in the main plot show, the representational similarity between SNNs and the mouse visual cortex is lower than the original similarity whenever the frame order is disrupted and the similarity decreases as the level of chaos increases. The order shuffling makes the movie frames discontinuous and breaks the original temporal relationships, leading to an alteration in the dynamic representations of SNNs. Considering that in TSRSA we align the frame stimuli of the network and visual cortex representation matrices, the decrease in similarity is mostly caused by the dissimilar dynamic representations and is not related to the static representations.

Furthermore, comparing the curves of drop rate between SEW-ResNet18 and R-SEW-ResNet18, we find out that the drop rate of R-SEW-ResNet18 increases with the level of chaos and eventually reaches a staggering 45.8%. However, the drop rate of SEW-ResNet18 is consistently lower than that of R-SEW-ResNet18 and the maximum is even less than 9%. These results reveal two significant findings. Firstly, the mouse visual cortex does not view movie frames as independent visual stimuli, and its neuronal population responses contain copious dynamic representations, rather than being dominated by static representations. R-SEW-ResNet18 trained on the UCF101 captures biological dynamic representations very well and is sensitive to the dynamic information. Even small perturbations on temporal features can cause large changes in its representations. Secondly, although SEW-ResNet18 is trained on a movie dataset, the vast majority of its representations depend on static information. Therefore, its representational similarity is not much affected when the temporal relationships of the input stimuli are ruined. In conclusion, the recurrent module greatly improves the ability of SNNs to capture temporal features and may provide new insights into the movie information processing mechanisms of the biological visual system.

### Effect of static representations on representational similarity

In addition to the importance of dynamic representations, the role of static representations in representational similarity should not be overlooked. We follow Section 3.3.3 to compare the representational similarity of R-SEW-ResNet18 trained on the ImageNet and the UCF101 (Figure 6). Similar to

Figure 6: The curves of representational similarity (the main plot) and the curves of drop rate of the experimental similarity compared to the original similarity (the subplot). The horizontal coordinates of both plots are the ratio of replacement. In the main plot, the dashed horizontal lines are the representational similarity between SNNs and the mouse visual cortex under the original movie. The error bar is the 95% confidence interval of the similarity across 10 trials. Except for these elements, the representations of other elements in the plots are the same as those in Figure 5.
the results in Section 4.2, the curves in the main plot show that the representational similarity between SNNs and the mouse visual cortex is mostly lower than the original similarity when some frames are replaced with noise images and the similarity decreases as the ratio of replacement increases. Obviously, the static representations of SNNs change a lot due to the totally different spatial features between the original movie frames and the noise images, resulting in the decrease of the representational similarity. What's more, the drop rate of R-SEW-ResNet18 trained on the UCF101 is slightly higher than that of R-SEW-ResNet18 trained on the ImageNet at low ratios of replacement, while the opposite result is observed at high ratios of replacement. The maximum drop rate of R-SEW-ResNet18 trained on the ImageNet is 62.4% while the maximum of that trained on the UCF101 is 53.7%, which could be explained by two possible reasons. On the one hand, the network trained on an image dataset treats the movie frames as independent individuals, so the network representations completely depend on static information and there is no temporal correlation between the representations of two frames. When the ratio of replacement is high, the corrupted static information leads to a large drop in the representational similarity. On the other hand, the network trained on a movie dataset represents both the static and dynamic information. Consequently, although the high ratio of replacement also damages its static representations, its dynamic representations to natural movie frames may moderate the drop of the representational similarity to some extent. In summary, the recurrent SNN trained on a movie dataset is able to capture both the static and dynamic representations of the mouse visual cortex, and the ability to represent dynamic information may make it preserve the brain-like representations when receiving corrupted static information.

## 5 Discussion

In this work, we introduce long-term feedback connections into deep spiking neural networks for the first time and use such SNNs to model the mouse visual cortex under movie stimuli, showing that deep recurrent SNNs trained on movies perform best on the representational similarity. We extend the analysis to both the static and dynamic representations of networks and the visual cortex with two meticulously designed experiments, providing computational evidence that the static and dynamic information of natural visual input is well encoded and transmitted by the neuronal population responses, and that recurrent SNNs is able to capture such representations. Specifically, in the movie frame order disruption experiment, we observe that the feedforward SNN not only has a lower representational similarity than the recurrent SNN, but also is almost unaffected by the disrupted frame order. The results demonstrate that the recurrent module effectively promotes SNNs' faculty of extracting temporal features, which facilitates the modeling of visual cortex representations. On the other hand, in the noise image experiment, we find out that the ability to represent the dynamic information may help recurrent SNNs mitigate the ruin of representations when the spatial information of visual stimuli is compromised. These findings aid us in better understanding the mechanisms of information processing and representation in the mouse visual system.

Biological studies suggest that the effect of recurrent connections in the visual cortex is variant across time, which may be a key factor acting on dynamic representations. [43] discovered that the neuronal population interactions between cortical regions are feedforward-dominated shortly after stimulus onset while feedback-dominated during spontaneous activity. As our work shows, deep recurrent SNNs are an excellent candidate for modeling the visual cortex and have the potential to explore the effects of feedforward and feedback connections on dynamic representations in biological neural responses.

In conclusion, with recurrent connections, deep SNNs yield the more powerful ability to represent spatio-temporal features more like the brain and become a new paradigm for studying both static and dynamic information processing in the biological visual system.