An Efficient Knowledge Transfer Strategy for Spiking Neural Networks
  from Static to Event Domain
Spiking neural networks (SNNs) are rich in spatio-temporal dynamics and are
suitable for processing event-based neuromorphic data. However, event-based
datasets are usually less annotated than static datasets. This small data scale
makes SNNs prone to overfitting and limits their performance. In order to
improve the generalization ability of SNNs on event-based datasets, we use
static images to assist SNN training on event data. In this paper, we first
discuss the domain mismatch problem encountered when directly transferring
networks trained on static datasets to event data. We argue that the
inconsistency of feature distributions becomes a major factor hindering the
effective transfer of knowledge from static images to event data. To address
this problem, we propose solutions in terms of two aspects: feature
distribution and training strategy. Firstly, we propose a knowledge transfer
loss, which consists of domain alignment loss and spatio-temporal
regularization. The domain alignment loss learns domain-invariant spatial
features by reducing the marginal distribution distance between the static
image and the event data. Spatio-temporal regularization provides dynamically
learnable coefficients for domain alignment loss by using the output features
of the event data at each time step as a regularization term. In addition, we
propose a sliding training strategy, which gradually replaces static image
inputs probabilistically with event data, resulting in a smoother and more
stable training for the network. We validate our method on neuromorphic
datasets, including N-Caltech101, CEP-DVS, and N-Omniglot. The experimental
results show that our proposed method achieves better performance on all
datasets compared to the current state-of-the-art methods. Code is available at
https://github.com/Brain-Cog-Lab/Transfer-for-DVS.
Xiang He, Dongcheng Zhao, Yang Li, Guobin Shen, Qingqun Kong, Yi Zeng
# Improving the Performance of Spiking Neural Networks

###### Abstract

Spiking neural networks (SNNs) have rich spatial-temporal dynamics, which are suitable for processing neuromorphic, event-based data. However, event-based datasets are usually less annotated than static datasets used in traditional deep learning. Small data scale makes SNNs prone to overfitting and limits the performance of the SNN. To enhance the generalizability of SNNs on event-based datasets, we propose a knowledge-transfer framework that leverages static images to assist in the training on neuromorphic datasets. Our method proposes domain loss and semantic loss to exploit both domain-invariant and unique features of these two domains, providing SNNs with more generalized knowledge for subsequent targeted training on neuromorphic data. Specifically, domain loss aligns the feature space and aims to capture common features between static and event-based images, while semantic loss emphasizes that the differences between samples from different categories should be as large as possible. Experimental results demonstrate that our method outperforms existing methods on all mainstream neuromorphic vision datasets. In particular, we achieve significant performance improvement of 2.7% and 9.8% when using only 10% training data of CIFAR10-DVS and N-Caltech 101 datasets, respectively.

Machine Learning, Neural Networks, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning Learning, Neural Networks, Learning Learning,
Compared to DVS dataset, it is relatively easy to obtain RGB data. When exploring a new environment, humans use their learned knowledge to compare it with new environment to better adapt to it. This inspires us leveraging static images to assist in the SNN training on neuromorphic datasets.

In this paper, we propose the knowledge transfer framework for SNN. Since DVS focuses on the variation of light intensity, we convert the widely available RGB static images into HSV space and take the luminance channel to help SNN learning on DVS data. To extract information useful for DVS, we propose domain loss and semantic loss. Specifically, domain loss learns domain-invariant features by reducing the joint distribution distance between the source domain (static images) and the target domain (DVS data), while semantic loss learns specific features by separating the distances between different classes of samples. As shown in Fig. 1, by introducing these two types of losses, the locations that are important for producing target are shifted to features of objects, such as the body of a bird, or the head of a dog and cat. Fig. 1 shows that our method learns the common features and obtains DVS features that facilitate correct classification.

The main contributions of this paper can be summarized as follows:

1. We propose a single-stage knowledge transfer framework based on annotated static datasets. To our best knowledge, we are the first to transfer the knowledge of spiking neural networks on static datasets to the DVS datasets, which enhances the generalization of SNNs.
2. We propose domain loss and semantic loss to enhance the learning of invariant and unique features between static and event data. Our results show that the learned features facilitate the training of SNN on DVS data, and the models trained with domain loss and semantic loss are flatter around local minima.
3. We conduct experiments on commonly used neuromorphic datasets CIFAR10-DVS, NCALTECH101, and N-Omniglot to verify the effectiveness of our method. The experimental results show that the proposed method outperforms the state-of-the-art methods on all datasets. Significantly when gradually reducing the number of training samples in the neuromorphic datasets, our method brings more significant performance improvements to SNNs.

## 2 Related Work

In order to solve the problem of limited labeled DVS data, previous works have tried domain adaptation, data augmentation and efficient training methods.

**Domain adaptation using static data.** Messikommer et al. (2022) use a generative event model to classify event features into content and motion features, enabling efficient matching between the latent space of events and images, and training the model directly with labeled images and unlabeled event data. Li et al. (2018) combine temporal coding and deep representation. The deep representation learned by the initially optimized CNN is effectively transferred to the event stream classification task, and the performance is further improved by fine-tuning the CNN with a certain number of training samples. Zhao et al. (2022) train a convolutional transformer network for event-based classification tasks using large-scale labeled image data via a passive unsupervised domain adaptation (UDA) algorithm. These works are related to ours; differently, we exploit the invariant and unique features between static and event data by cross-domain loss. Such features can provide a general

Figure 1: Class Activation Mapping of CIFAR10 and CIFAR10-DVS. Three categories are selected for display, the first two rows under each category represent static pictures, and the last row represents neuromorphic data integrated into frames; the three columns from left to right represent the results of original picture, baseline and our method, respectively.
ized prior knowledge for the SNN, thus facilitating further training of the network. This can enhance the original SNN structure instead of pre-training a new network model with more parameters.

Event-based augmentation.Li et al. (2022) propose neuromorphic data augmentation to stabilize the training of SNNs and improve generalization. Shen et al. (2022) design an enhancement strategy for event stream data, and perform the mixing of different event streams by Gaussian mixing model, while assigning labels to the mixed samples by calculating the relative distance of event streams.

SNN efficient trainingZhan et al. (2021) analyzes the plausibility of central kernel alignment (CKA) as a domain distance measure relative to maximum mean difference (MMD) in deep SNNs. Deng et al. (2022) introduce the time-efficient training (TET) method, which allows the surrogate gradient method to SNN to converge to a flatter minimum.

## 3 Preliminaries

Neuron model.We choose Leaky Integrate-and-Fire (LIF) model (Dayan and Abbott, 2005), the most commonly used neuron model. Equationally, the update of the membrane potential \(\mathbf{u}\) can be written in the following discrete form

\[\mathbf{u}^{t+1,l}=\tau\mathbf{u}^{t,l}+\mathbf{W}^{l}\mathbf{s}^{t,l-1}, \tag{1}\]

where \(\tau\) is leaky factor and \(\mathbf{u}^{t,l}\) denotes membrane potential of the neurons in layer \(l\) at time step \(t\). \(\mathbf{W}^{l}\) and \(\mathbf{s}^{l}\) represent the weight parameters of the layer \(l\) and the fired spikes in layer \(l\), respectively.

The membrane potential accumulates with the input until a given threshold \(V_{th}\) is exceeded, then the neuron delivers a spike and the membrane potential \(\mathbf{u}^{t,l}\) is reset to zero. The equation can be expressed as

\[\mathbf{s}^{t,l}=H\left(\mathbf{u}^{t,l}-V_{th}\right) \tag{2}\] \[\mathbf{u}^{t+1,l}=\tau\mathbf{u}^{t,l}\cdot\left(1-\mathbf{s}^{t,l}\right)+ \mathbf{W}^{l}\mathbf{s}^{t+1,l-1}, \tag{3}\]

where \(H\) denotes Heaviside step function. In this paper, leaky factor \(\tau\) is set to 0.5 and threshold \(V_{th}\) to 0.5.

Processing of neuromorphic data.The Dynamic Vision Sensor (DVS) triggers an event at a specific pixel point when it detects a significant change in brightness. Formuliaically, it can be expressed as

\[L(x,y,t)-L(x,y,t-\Delta t)\geq pC, \tag{4}\]

where \(x\) and \(y\) denote pixel location and \(\Delta t\) means the time since last triggered event at \((x,y)\). \(p\) is polarity of brightness change and \(C\) is a constant contrast threshold. In this way, DVS triggers a number of events \(\varepsilon\) during a time interval in the form \(\varepsilon=\left\{\left(x_{i},y_{i},t_{i},p_{i}\right)\right\}_{i=1}^{N}\). Due to the large number of events, we integrate them into frames in order to facilitate processing as previous works (Wu et al., 2019; He et al., 2020; Fang et al., 2021; Shen et al., 2022). Specifically, the events are divided into T slices, and all events in each slice are accumulated. The \(j\)-th (\(0\leq j\leq T-1\)) slice event after integration, \(E(j,x,y,p)\), can be defined as

\[E(j,x,y,p)=\sum_{j_{s}}^{j_{s}-1}\mathbf{1}_{x,y,p}\left(x_{i},y_{i},p_{i}\right) \tag{5}\]

\[j_{s}=\lfloor\frac{N}{T}\rfloor\cdot j,\quad j_{e}=\lfloor\frac{N}{T}\rfloor \cdot(j+1), \tag{6}\]

where \(\mathbf{1}_{x,y,p}\left(x_{i},y_{i},p_{i}\right)\) is an indictor function. \(j_{s}\) and \(j_{e}\) are the start and end index of event in \(j\)-th slice.

Feature Similarity Measurements.In order to measure the difference between static image and DVS data, we need to calculate the distance and similarity between them. The Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005) can be used to measure whether two sets of data are independent. For a Reproducing Kernel Hilbert Space \(\mathcal{H}\), all eigenfunctions of the kernel function constitute a set of orthogonal bases for the space. So we obtain the matrix form of HSIC under finite samples as follows

\[\mathrm{HSIC}(K,L)=\frac{1}{(n-1)^{2}}\operatorname{tr}(KJLJ), \tag{7}\]

where \(J\) is the centering matrix \(J_{n}=I_{n}-\frac{1}{n}11^{\mathrm{T}}\), here \(I_{n}\) is an \(n\) order unit matrix. \(\operatorname{tr}\) means trace of matrix. A normalized approach based on HSIC, i.e., central kernel alignment (CKA) (Cortes et al., 2012; Cristianini et al., 2001), can make HSIC invariant to isotropic scaling. The formula can be written as:

\[\mathrm{CKA}(K,L)=\frac{\mathrm{HSIC}(K,L)}{\sqrt{\mathrm{HSIC}(K,K)\, \mathrm{HSIC}(L,L)}}. \tag{8}\]

Kornblith et al. (2019) introduced CKA as a similarity index to better measure neural network representation similarity. Let \(K_{ij}=k\left(\mathbf{x}_{i},\mathbf{x}_{j}\right)\) and \(L_{ij}=l\left(\mathbf{y}_{i},\mathbf{y}_{j}\right)\), where k and l are kernels and can be simply chosen as linear kernel. Zhan et al. (2021) demonstrates the feasibility of using CKA as a distance metric in deep SNN. And in this paper, we use CKA as the distance criterion between static image domain and DVS data domain.

## 4 Methods

We expect to use the source domain (static images) data to help learn better SNN model for the target domain (DVS data). This is a supervised domain adaptation problem since we can make use of labeled target domain data despite the
fact that they are fewer. In this section, we first formalize the method in Sec. 4.1, and then introduce the color space transformation and dimension alignment in Sec. 4.2. The design of the loss function in Sec. 4.3. Sec. 4.4 and 4.5 elaborate the pipeline and training strategy of proposed method.

### Methodology formulation

A labeled source domain \(\mathcal{D}_{s}=\left\{x_{t}^{i},y_{s}^{i}\right\}_{i=1}^{N}\) and small labeled target domain \(\mathcal{D}_{t}=\left\{x_{t}^{i},y_{t}^{i}\right\}_{i=1}^{M}\) with feature space \(\mathcal{X}_{s}\) and \(\mathcal{X}_{t}\) respectively. The feature \(x_{s}^{i}\in\mathcal{X}_{s}\) and \(x_{t}^{i}\in\mathcal{X}_{t}\). In our task, two domain have the same category space and conditional probability distribution, i.e., \(\mathcal{Y}_{s}=\mathcal{Y}_{t}\) and \(Q_{s}\left(y_{s}\mid\mathbf{x}_{s}\right)=Q_{t}\left(y_{t}\mid\mathbf{x}_{t}\right)\). Let us use \(\mathbf{x}\) as a vector representation of the domain data and \(\mathbf{X}\) as the whole domain data. For static image domain and DVS data domain, the feature space and marginal distribution of these two domains is different due to the difference in sensor type, i.e., \(\mathcal{X}_{s}\neq\mathcal{X}_{t}\) and \(P_{s}\left(X_{s}\right)\neq P_{t}\left(X_{t}\right)\).

We aim to leverages \(\mathcal{D}_{s}\) to assist in learning a better classifer \(f:\mathbf{x}_{t}\mapsto\mathbf{y}_{t}\) to predict \(\mathcal{D}_{t}\) label \(\mathbf{y}_{t}\in\mathcal{Y}_{t}\). The model for function \(f\) involves a composition of two functions, i.e., \(f=h\circ g\). Here \(g:\mathcal{X}\rightarrow\mathcal{Z}\) represents an embedding of the input space \(\mathcal{X}\) into a feature space \(\mathcal{Z}\), and \(h:\mathcal{Z}\rightarrow\mathcal{Y}\) is a function that predicts outputs from the feature space. With this notation we would have \(f_{s}=h_{s}\circ g_{s}\) and \(f_{t}=h_{t}\circ g_{t}\). In this paper we utilize the final classification head of the original model as \(h_{t}\). This function is learned solely through supervised signal update gradients. There are a large number of labeled image that can be studied directly for static image so \(h_{s}\) are not the object of this study. Critically, we want to provide a generalization of \(g_{t}\) which can pave the way for learning of \(h_{t}\) to improve the generalizability of SNN.

As mentioned before, due to the different sensory devices, the two data domains have different feature spaces and they have different dimensions. This type of domain adaptation (DA) is heterogeneous and unlike homogeneous DA approaches, there is not much work focused on heterogeneous DA as far as depth approaches are concerned, and the solution for heterogeneous deep DA is still similar to some homogeneous DA approaches (Wang and Deng, 2018).

In this paper we use a discrepancy-based approach. The embedding function \(g\) would be modeled by network sharing between the source and target domains using all layers before the last classification layer. At this point the shared \(g_{t}=g_{s}=g\), the objective of optimization is finding the satisfied \(g\) in its hypothetical space \(\mathcal{G}\):

\[\operatorname*{arg\,min}_{g\in\mathcal{G}}\left(d\left(p\left(g\left(X_{s}^{ a}\right)\right),p\left(g\left(X_{t}^{a}\right)\right)\right)-d\left(p\left(g \left(X_{s}^{a}\right)\right),p\left(g\left(X_{t}^{c}\right)\right)\right) \right), \tag{9}\]

where \(X_{s}^{a}\) and \(X_{t}^{a}\) refer to the same classes of data in the source and target domains while \(X_{s}^{a}\) and \(X_{t}^{c}\) mean the data from different classes. The \(d\) is a metric for judging similarity between two domain, we choose CKA here. One of the purpose of Eq. 9 is to align the distributions of the features in the embedding space \(\mathcal{Z}\), whose features are assumed to be domain invariant. The other is learning unique features of DVS data. These two aspects facilitate finding a more

Figure 2: Proposed knowledge transfer framework for spiking neural network. Static image and DVS data are input simultaneously and share the network weights except for the last layer. The membrane potential of the neurons in the second-last layer is used to calculate the domain loss and semantic loss. Static data is replaced with DVS in certain probability.
generalized \(g\).

### HSV color space

The event data can be generated by rich local intensity changes in continuous time. One way is to use the relative motion of the image and the camera, e.g., Orchard et al. (2015) employed the camera movement of saccade, Li et al. (2017) employed image movement instead of camera movement; Li et al. (2022) reconstruct the written record of strokes into a video of writing tracks. All these approaches use DVS to sense the change of brightness of each pixel point in the image in an asynchronous manner and output a stream of events. The static images we use directly are in RGB color space, and the three components (red, green, blue) are sensitive to luminance, i.e., whenever the luminance changes, all three components change accordingly. Therefore, using RGB to reflect light intensity is not intuitive enough. For the above reason, we convert the still RGB image to HSV color space with three components (hue, saturation, value), as expressed in the following equation.

\[H=\begin{cases}0^{\circ},&\text{if}\ \ \Delta=0\\ 60^{\circ}\times\frac{G^{\prime}-B^{\prime}}{\Delta}+0^{\circ},&\text{if}\ \ C_{ \text{max}}=R^{\prime}\\ 60^{\circ}\times\frac{B^{\prime}-R^{\prime}}{\Delta}+120^{\circ},&\text{if}\ \ C_{ \text{max}}=G^{\prime}\\ 60^{\circ}\times\frac{R^{\prime}-G^{\prime}}{\Delta}+240^{\circ},&\text{if}\ \ C_{ \text{max}}=B^{\prime}\end{cases} \tag{10}\]

\[S=\begin{cases}0,&\text{if}\ \ C_{\text{max}}=0\\ \frac{\Delta}{C_{\text{max}}},&\text{if}\ \ C_{\text{max}}\neq 0\end{cases} \tag{11}\]

\[V=C_{\text{max}} \tag{12}\]

where \(R^{\prime},G^{\prime},B^{\prime}\) here are the normalized values of \(R,G,B\), \(C_{\text{max}}=\max\{R^{\prime},G^{\prime},B^{\prime}\},C_{\text{min}}=\min\{R ^{\prime},G^{\prime},B^{\prime}\}\) and \(\Delta=C_{\text{max}}-C_{\text{min}}\). \(V\) indicates the degree of brightness of the color, for the light source color, the brightness value \(V\) and the luminosity of the luminous body are related. To match with 2-channel, i.e., positive and negative polarity DVS data, we duplicated V channel and then input to the network simultaneously with DVS data.

### Loss function

Our loss contains three parts, classification loss for optimizing \(h_{t}\), domain loss and semantic loss for optimizing shared \(g\). We describe them in detail as following.

**Classification loss.** For the classification loss, we choose the TET loss (Deng et al., 2022), which is proven to compensate the momentum loss of surrogate gradient and make SNN have better generalizability. The formula is described as follows

\[\mathcal{L}_{TET}=\frac{1}{T}\sum_{t=1}^{T}\left((1-\lambda)\ell_{ce}\left( \mathbf{O}(t),\mathbf{y}\right)+\lambda\operatorname{MSE}\left(\mathbf{O}(t ),V_{th}\right)\right), \tag{13}\]

where \(\mathbf{O}(t)\) represents the presynaptic input current to the output layer. The two items before and after the plus symbol are the cross-entropy loss and the mean-squared error, with the combination of coefficients of \(1-\lambda\) and \(\lambda\) respectively.

**Domain loss.** The inputs to dual streams come from static image and DVS data respectively. The closer the value of CKA is to 1 indicates that the two vectors are more correlated. For this reason, we subtract the CKA from 1, minimizing the loss i.e., maximizing the correlation of the two inputs. We express the samples \(\mathbf{x}_{s},\mathbf{x}_{t}\) drawn from the whole data \(\mathbf{X}_{s},\mathbf{X}_{t}\), in this way, domain loss can be expressed as

\[\mathcal{L}_{d}(g)=1-\frac{1}{T}\sum_{t=1}^{T}\underset{y_{i}=y_{j},y\in \mathcal{Y}}{CKA}\left(g\left(\mathbf{x}_{s}^{i},t\right),g\left(\mathbf{x}_{ t}^{j},t\right)\right). \tag{14}\]

Here we use \(g\left(\mathbf{x}_{s}^{i},t\right)\) indicates the value of the input after the shared parameter function \(g\), \(t\) is brought in to emphasize that here is the output of \(g\) at time \(t\). Two samples \(\mathbf{x}_{s}^{i},\mathbf{x}_{t}^{j}\) are sampled from the same class, expressed by the formula \(y_{i}=y_{j}\).

**Semantic loss.** The representation of semantic loss is the opposite of domain loss, which is equal to the similarity between samples belonging to different categories in two domains, minimizing semantic loss means maximizing the difference between samples of different classes. The formula is expressed as

\[\mathcal{L}_{s}(g)=max\left(0,\frac{1}{T}\sum_{t=1}^{T}\underset{y_{i}\neq y_{j },y\in\mathcal{Y}}{CKA}\left(g\left(\mathbf{x}_{s}^{i},t\right),g\left(\mathbf{ x}_{t}^{j},t\right)\right)-m\right), \tag{15}\]

Here \(m\) is the margin of similarity, that is, the loss is considered zero when the similarity is below \(m\), and there is no further optimization. The meaning of \(m\) is that the similarity of two inputs from different domains can be controlled arbitrarily. For example, when the differences are well characterized, i.e., they are far enough apart to have low similarity, there is no need to waste time reducing similarity in that sample pair, so further training will focus on other sample pairs that are more difficult to separate.

**Total loss.** With the above statement, the total training loss is the sum of the classification loss, domain loss and semantic loss. That is:

\[\mathcal{L}_{all}=\mathcal{L}_{ce}+\lambda_{d}\mathcal{L}_{d}+\lambda_{s} \mathcal{L}_{s}. \tag{16}\]

One thing to mention is that domain loss and semantic loss work only within pre-set epochs while classification loss works throughout the training process.
### Single-stage training pipeline

During the training process, we gradually replace a portion of the static input with DVS data probabilistically, so that the role of domain loss is smoothly decreased and the semantic loss gradually moves from distinguishing different classes of sample distances across domains to different classes of distances within the DVS domain. Separately, with \(b_{i}\) denoting index of training batch, \(b_{l}\) denoting total length of training batch; \(e_{c}\) means current epoch and \(e_{m}\) means maximum training epoch, then probability of making a substitution \(P_{replacement}\) could be expressed by the following equation

\[P_{replacement}=\left(\frac{b_{i}+e_{c}*b_{l}}{e_{s}*b_{l}}\right)^{3}, \tag{17}\]

where \(e_{s}\) is a manual settings epoch for the end of the domain loss and semantic loss effects. The value of \(e_{s}\) if usually set to half of the total training epoch.

In summary, our framework first converts the static images from RGB to HSV and aligns the dimensions. And then for dual-stream inputs, the model shares the network parameters of all layers before the final classification layer to learn domain-invariant features by domain loss and unique features by semantic loss. Static image inputs are gradually replaced with DVS data inputs by increasing probability with the number of training epochs until the set epochs is reached, all static images are replaced with DVS data, and the feature extraction period ends. Finally, we fine-tune the SNN on the DVS data to get better performance. The whole process is shown in Fig. 2.

### Training strategy

We use gradient descent to directly train the SNN. For classification loss, the last layer of neurons only receives and accumulates presynaptic input currents and does not fire spikes, so the derivative of \(\mathcal{L}_{\mathrm{TET}}\) with respect to \(W\) is

\[\frac{\partial\mathcal{L}_{\mathrm{TET}}}{\partial W^{l}}=\sum_{t=1 }^{T}\frac{\partial\mathcal{L}_{\mathrm{TET}}}{\partial O(t)}\frac{\partial O (t)}{\partial W^{l}} \tag{18}\] \[\frac{\partial\mathcal{L}_{\mathrm{TET}}}{\partial O(t)}=\frac{1 }{T}\left(\left(1-\lambda\right)\left(S(O(t))-\hat{y}\right)+\lambda(O(t)-V_{ th})\right) \tag{19}\]

where \(\hat{y}\) means one-hot and \(S\) means softmax function.

The partial derivative of the loss \(\mathcal{L}_{d}\) with respect to \(W\) can be expressed as

\[\frac{\partial\mathcal{L}_{d}}{\partial W^{l}}=\sum_{t=1}^{T}\frac{\partial \mathcal{L}_{d}}{\partial s^{t,l}}\frac{\partial s^{t,l}}{\partial u^{t,l}} \frac{\partial u^{t,l}}{\partial W^{l}}. \tag{20}\]

Denote the kernel matrix of source and target domains by \(K_{S}\), \(K_{T}\), and the membrane potential inputs from the source

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Model** & **Methods** & **Architecture** & **Simulation Length** & **Accuracy** \\ \hline \multirow{10}{*}{CIFAR10-DVS} & Zheng et al. (2021) & STBP-tdBN & ResNet-19 & 10 & 67.80 \\  & Kugele et al. (2020) & Streaming Rollout & DenseNet & 10 & 66.80 \\  & Wu et al. (2021) & LIAF & LIAF-Net & 10 & 70.40 \\  & Li et al. (2021) & Dspike & ResNet-18 & 10 & 75.4 \\  & Fang et al. (2021) & PLIF & SNN4 & 20 & 74.8 \\  & Yao et al. (2021) & TA-SNN & SNN5 & 10 & 72.0 \\  & Li et al. (2022d) & NDA & VGGSNN & 10 & 81.70 \\  & Deng et al. (2022) & TET & VGGSNN & 10 & \(83.17\pm 0.15\) \\  & Zhu et al. (2022) & TJCA-TET & VGGSNN & 10 & 83.3 \\  & Duan et al. & TEBN & VGGSNN & 10 & 84.9 \\ \cline{2-6}  & **Our model** & Transfer & VGGSNN & 10 & \(84.60\pm 0.14\)\(\mathbf{(84.8)}\) \\  & **Our model** & Transfer & VGGSNN & 10 & \(84.67\pm 0.39\)\(\mathbf{(85.2)}^{2}\) \\ \hline \multirow{10}{*}{N-Caltech101} & Li et al. (2022d) & NDA & VGGSNN & 10 & 78.2 \\  & Deng et al. (2022) & TET & VGGSNN & 10 & \(79.77\pm 0.77^{1}\) \\ \cline{1-1}  & Shen et al. (2022b) & EventMixer & ResNet-18 & 10 & 79.5 \\ \cline{1-1}  & Zhu et al. (2022) & TJCA-TET & CombinedSNN & 14 & 82.5 \\ \cline{1-1} \cline{2-6}  & **Our model** & Transfer & VGGSNN & 10 & \(81.46\pm 0.82\)\(\mathbf{(82.30)}\) \\ \cline{1-1}  & **Our model** & Transfer & VGGSNN & 10 & \(81.57\pm 0.71\)\(\mathbf{(82.52)}^{2}\) \\ \hline \multirow{10}{*}{N-Omniglot} & Li et al. (2022b) & plain & SCNN & 12 & 60.0 \\ \cline{1-1}  & Li et al. (2022b) & plain & SCNN & 12 & \(61.73\pm 0.41\)\(\mathbf{(62.23)}^{1}\) \\ \cline{1-1} \cline{2-6}  & **Our model** & Transfer & SCNN & 12 & \(63.03\pm 0.14\)\(\mathbf{(63.22)}\) \\ \hline \multicolumn{6}{l}{\({}^{1}\) Our implement} \\ \multicolumn{6}{l}{\({}^{2}\) Using label smoothing} \\ \end{tabular}
\end{table}
Table 1: Experimental results compared with existing works. The best accuracy is shown in parentheses.
and target domains by \(u_{S}\), \(u_{T}\), respectively. The first term can be expanded as

\[\begin{split}\frac{\partial L_{d}}{\partial s_{S}^{t,l}}=& -\frac{1}{T}\sum_{t=1}^{T}\frac{\partial\text{CKA}\left(K_{S},K_{T} \right)}{\partial s_{S}^{t,l}}\\ =&-\frac{1}{T}\sum_{t=1}^{T}\frac{\partial\,\text{ CKA}\left(K_{S},K_{T}\right)}{\partial\,\text{HSIC}\left(K_{S},K_{T} \right)}\frac{\partial\,\text{HSIC}\left(K_{S},K_{T}\right)}{\partial s_{S}^{t,l}}\\ &-\frac{1}{T}\sum_{t=1}^{T}\frac{\partial\,\text{CKA}\left(K_{S},K _{T}\right)}{\partial\,\text{HSIC}\left(K_{S},K_{S}\right)}\frac{\partial\, \text{HSIC}\left(K_{S},K_{S}\right)}{\partial s_{S}^{t,l}}\end{split} \tag{21}\]

Calculating \(\frac{\partial\text{CKA}\left(K_{S},K_{T}\right)}{\partial\,\text{HSIC}\left( K_{S},K_{T}\right)}\) and \(\frac{\text{HSIC}\left(K_{S},K_{T}\right)}{\partial s_{S}^{t,l}}\) as an example, from Eq. 8

\[\begin{split}\frac{\partial\text{CKA}\left(K_{S},K_{T}\right)}{ \partial\,\text{HSIC}\left(K_{S},K_{T}\right)}=\frac{1}{\left[\text{HSIC} \left(K_{S},K_{S}\right)\text{HSIC}\left(K_{T},K_{T}\right)\right]^{1/2}} \end{split} \tag{22}\]

\(\frac{\text{HSIC}\left(K_{S},K_{T}\right)}{\partial s_{S}^{t,l}}\) in Eq. 21 can be expressed as

\[\begin{split}&\left(\sum_{j=1}^{n}\frac{\partial k_{S}\left(u_{S }^{t,l},u_{S_{j}}^{t,l}\right)}{\partial u_{S_{i}}^{t,l}}A(k_{T})_{ji}+\sum_{j= 1}^{n}\frac{\partial k_{S}\left(u_{S_{j}}^{t,l},u_{S_{i}}^{t,l}\right)}{ \partial u_{S_{i}}^{t,l}}A(k_{T})_{ij}\right.\\ &-\left.\frac{\partial k_{S}\left(u_{S_{i}}^{t,l},u_{S_{i}}^{t,l} \right)}{\partial u_{S_{i}}^{t,l}}A(k_{T})_{ii}\right)\frac{\partial u_{S_{i}}^ {t,l}}{(n-1)^{2}\partial s_{S_{i}}^{t,l}}\end{split} \tag{23}\]

where \(A(k_{T})_{ij}\) should be

\[\begin{split} k_{T}\left(u_{T_{i}}^{t,l},u_{T_{j}}^{t,l} \right)+&\sum_{k=1}^{n}\sum_{m=1}^{n}k_{T}\left(u_{T_{k}}^{t,l},u_ {T_{m}}^{t,l}\right)-& 2\sum_{m=1}^{n}k_{T}\left(z_{T_{i}}^{l},z_{T_{m}}^{l} \right)\end{split} \tag{24}\]

For second term \(\frac{\partial s^{t,l}}{\partial u^{t,l}}\), we use Piece-wise linear function (Xu et al., 2022) surrogate gradient, the formula as follows

\[\frac{\partial s^{t,l}}{\partial u^{t,l}}=max(0,1-|u^{t,l}|) \tag{25}\]

As for third term in Eq. 20, from Eq. 3, It can be expressed cyclically as

\[\begin{split}\frac{\partial u^{t,l}}{\partial W^{l}}=& \tau\left(1-s^{t-1,l}\right)\frac{\partial u^{t-1,l}}{\partial W^{l}}\\ &-\tau u^{t-1,l}\frac{\partial s^{t-1,l}}{\partial u^{t-1,l}} \frac{\partial u^{t-1,l}}{\partial W^{l}}+s^{t,l-1}.\end{split} \tag{26}\]

The derivation of \(\partial\mathcal{L}_{s}/\partial W^{l}\) is similar to Eq. 21, 22, 23. More detail please refer to supplementary materials.

## 5 Experiments

In this section, we conduct experiments on three mainstream neuromorphic datasets: CIFAR10-DVS (Li et al., 2017), N-Caltech 101 (Orchard et al., 2015) and N-Omniglot (Li et al., 2022b) to evaluate the effectiveness of the proposed method. We choose AdamW (Loshchilov and Hutter, 2017) as the optimizer with an initial learning rate of 1e-3. The total training epoch is 600 and the warmup epoch is 5. For a fair comparison, for the first two datasets, the network model is chosen as VGGSNN (64C3-128C3-AP2-AP2-256C3-256C3-AP2-512C3-512C3-AP2-512C3-AP2-FC) with step 10. For the N-omniglot, we use the network structure SCNN (15C5-AP2-40C5-AP2-FC-FC) from the original paper with step 12. More detail please refer to supplementary materials. All experiments are based on the Pytorch framework.

### Comparison with the State-of-the-Art

We first evaluate the proposed method on the CIFAR10-DVS dataset and compare the proposed method with TdBN (Zheng et al., 2021), Streaming Rollour (Kugele et al., 2020), LIAF (Wu et al., 2021), Dspike (Li et al., 2021), PLIF (Fang et al., 2021), TA-SNN (Yao et al., 2021), NDA (Li et al., 2022d), TET (Deng et al., 2022), TJCA-TET (Zhu et al., 2022), TEBN (Duan et al.). The results are presented in Tab. 1. In fairness, we perform three independent experiments using different random seeds and report the mean and standard deviation of the results. The best accuracy is indicated in parentheses. The experimental results demonstrate that the proposed method can achieve state-of-the-art performance

Figure 3: Classification accuracy with epoch for CIFAR10-DVS data on the whole test data with different amounts of training data. (a) training with 10% training data on CIFAR10-DVS, (b) training with 100% training data on CIFAR10-DVS
on CIFAR10-DVS dataset compared with existing methods. In addition, we observe that the trick of label smoothing is beneficial to alleviate the overfitting, leading to a 0.4% accuracy improvement.

As for N-Caltech 101 dataset, there are fewer available results. We use TET, NDA, EventMixer (Shen et al., 2022b), and TJCA-TET as baseline. The results in Tab. 1 show that our method outperforms all baselines and similarly, label smoothing brings a modest improvement in accuracy. For N-omniglot, we reproduce the original result and compare it with the proposed method. Experimental results show that our proposed method improves 1% accuracy over the original method.

### Ablation study

In our approach, the first step is to extract domain-invariant features using domain loss, followed by adding semantic loss to better extract features from the DVS data. Thus, the ablation experimental section shows the results of baseline, with domain loss, with domain loss and semantic loss. As shown in Fig. 3, baseline has overfitted at about 100-200 epochs earlier, i.e., the accuracy on the test set does not increase in subsequent training rounds. In contrast, the introduction of domain loss and semantic loss provides a good model at the end of the action period, and the green line is always at the top in the later fine-tuning phase, indicating that the best results can be achieved with these two losses.

Moreover, we conduct a detailed evaluation of our proposed approach on CIFAR10-DVS and N-Caltech101 datasets using varying amounts of training data, as presented in Tab. 2. Our results show that regardless of training data amount, with domain loss results in higher performance than the baseline method, while the combination of both domain and semantic losses outperforms the method with only domain loss. Importantly, our proposed approach achieves a remarkable performance improvement of nearly 10% on N-Caltech101 when using only 10% training data. This indicates that the fewer data available, the easier it is to overfit existing methods. Providing generalizability knowledge helps to alleviate this problem.

### Analysis

**Loss losslandscape** To evaluate the impact of domain and semantic loss in providing generalization weight for SNN fine-tuning on DVS data, we utilize 2D lossscape visualization (Li et al., 2018). For this purpose, we selected the model weights at the 300th epoch, which is the moment when the effect of loss ends, for CIFAR10-DVS and N-Caltech101 at 10% of the training data. As depicted in Fig. 4(b) and Fig. 4(d), the maximum loss decreases compared to Fig. 4(a) and Fig. 4(c) and the lowest loss area becomes flatter, which indicates that the SNN obtains better weights with domain loss and semantic loss.

**Visual Explanations from Deep Networks** To assess

\begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{2}{c}{NetWork} & Dataset & Methods & Accuracy \\ \hline \multicolumn{2}{c}{**Data with no Noise**} & & \\ \hline \multirow{4}{*}{VGG\_SNN} & \multirow{4}{*}{CIFAR10-DVS} & origin & 83.60\% \\  & & domainLoss & 84.10\% \\  & & domainLoss + semanticLoss & **84.50\%** \\ \cline{2-4}  & & origin & 79.54\% \\  & & domainLoss & 80.46\% \\  & & domainLoss + semanticLoss & **81.72\%** \\ \hline \multicolumn{2}{c}{**70\% Training Data**} & & \\ \hline \multirow{4}{*}{VGG\_SNN} & \multirow{4}{*}{CIFAR10-DVS} & origin & 80.70\% \\  & & domainLoss & 81.50\% \\  & & domainLoss + semanticLoss & **82.30\%** \\ \cline{2-4}  & & origin & 76.78\% \\ \cline{2-4}  & & domainLoss & 78.51\% \\  & & domainLoss + semanticLoss & **79.20\%** \\ \hline \multicolumn{2}{c}{**40\% Training Data**} & & \\ \hline \multirow{4}{*}{VGG\_SNN} & \multirow{4}{*}{CIFAR10-DVS} & origin & 76.50\% \\  & & domainLoss & 76.80\% \\ \cline{1-1}  & & domainLoss + semanticLoss & **77.90\%** \\ \cline{1-1} \cline{2-4}  & & origin & 67.93\% \\ \cline{1-1}  & & domainLoss & **71.84\%** \\ \cline{1-1}  & & domainLoss + semanticLoss & 71.49\% \\ \hline \multicolumn{2}{c}{**10\% Training Data**} & & \\ \hline \multirow{4}{*}{VGG\_SNN} & \multirow{4}{*}{CIFAR10-DVS} & origin & 58.60\% \\  & & domainLoss & 60.50\% \\ \cline{1-1}  & & domainLoss + semanticLoss & **61.30\%** \\ \cline{1-1} \cline{2-4}  & & origin & 45.40\% \\ \cline{1-1}  & & domainLoss & 54.71\% \\ \cline{1-1}  & & domainLoss + semanticLoss & **55.17\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Ablation experimental results with VGGSNN.

Figure 4: The loss landscape of VGGSNN on CIFAR10-DVS and N-Caltech101 at the 300th epoch.
whether domain loss and semantic loss effectively learn common and unique features between static images and neuromorphic data, we employ grad-cam++ (Chattopadhay et al., 2018). This method enables the visualization of the degree of similarity between each location in an image and a category by generating a heat map. Such a visualization allows us to understand which local locations of an original image contributed most significantly to the model's final classification decision. Ideally, static pictures and event data that are integrated into frames have similarity in object contour features when they are in the same class, while they should be as distinct as possible between different classes. This is well illustrated in Fig. 1, where by introducing domain and semantic loss, for both static pictures and DVS data, the network pays attention to the contour features of the category. In particular, the results on DVS show that our approach facilitates the correct classification of DVS data.

### The effect and selection of hyperparameters

In this section, we show the effect of hyperparameters in the proposed method and the principles of their selection.

The coefficient of domain loss for learning common features, \(\lambda_{d}\), is set fixed at 1. Our method is insensitive to the semantic loss coefficient and performs well within its reasonable range. As shown in Tab. 3, \(\lambda_{s}=0.0\) means only domain loss works, with the addition of semantic loss there is a performance degradation only when its coefficients \(\lambda_{s}\) are particularly large. This means that semantic loss should not dominate, and that the learning of common features should be satisfied before the learning of unique features.

As for the marginal coefficient \(m\), it should not be chosen too small, resulting in not working, or too large, limiting the role of semantic loss. As shown in Tab. 3, the choice of the m = 0.1 is appropriate.

## 6 Conclusion

In this paper, we propose a single-stage transfer learning framework based on a large number of annotated static datasets to improve the generalizability of SNN networks on DVS data. We propose domain loss and semantic loss to learn domain-invariant and unique features between static image and DVS data. Visualizations on loss-landscape and Class Activation Mapping show that the learned features benefit the performance of SNN on DVS data. Experimental results show that our method achieves the best performance on mainstream DVS datasets.
