A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets
  Spiking Neural Networks
While contrastive self-supervised learning has become the de-facto learning
paradigm for graph neural networks, the pursuit of higher task accuracy
requires a larger hidden dimensionality to learn informative and discriminative
full-precision representations, raising concerns about computation, memory
footprint, and energy consumption burden (largely overlooked) for real-world
applications. This work explores a promising direction for graph contrastive
learning (GCL) with spiking neural networks (SNNs), which leverage sparse and
binary characteristics to learn more biologically plausible and compact
representations. We propose SpikeGCL, a novel GCL framework to learn binarized
1-bit representations for graphs, making balanced trade-offs between efficiency
and performance. We provide theoretical guarantees to demonstrate that SpikeGCL
has comparable expressiveness with its full-precision counterparts.
Experimental results demonstrate that, with nearly 32x representation storage
compression, SpikeGCL is either comparable to or outperforms many fancy
state-of-the-art supervised and self-supervised methods across several graph
benchmarks.
Jintang Li, Huizhe Zhang, Ruofan Wu, Zulun Zhu, Baokun Wang, Changhua Meng, Zibin Zheng, Liang Chen
# A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks

###### Abstract

While contrastive self-supervised learning has become the de-facto learning paradigm for graph neural networks, the pursuit of high task accuracy requires a large hidden dimensionality to learn informative and discriminative full-precision representations, raising concerns about computation, memory footprint, and energy consumption burden (largely overlooked) for real-world applications. This paper explores a promising direction for graph contrastive learning (GCL) with spiking neural networks (SNNs), which leverage sparse and binary characteristics of SNNs to learn more biologically plausible and compact representations. We propose SpikeGCL, a novel GCL framework to learn binarized 1-bit representations for graphs, making balanced trade-offs between efficiency and performance. We provide theoretical guarantees to demonstrate that SpikeGCL has comparable expressiveness with its full-precision counterparts. Experimental results demonstrate that, with nearly 32x representation storage compression, SpikeGCL is either comparable to or outperforms many fancy state-of-the-art supervised and self-supervised methods across several graph benchmarks.

## 1 Introduction

Graph neural networks (GNNs) have demonstrated remarkable capabilities in learning representations of graphs and manifolds that are beneficial for a wide range of tasks. Especially since the advent of recent self-supervised learning techniques, rapid progress toward learning universally useful representations has been made [23]. Graph contrastive learning (GCL), which aims to learn generalizable and transferable graph representations by contrasting positive and negative sample pairs taken from different graph views, has become the hotspot in graph self-supervised learning. As an active area of research, numerous variants of GCL methods have been proposed to achieve state-of-the-art performances in graph-based learning tasks, solving the dilemma of learning useful representations from graph data without end-to-end supervision [20; 37; 36; 44].

In recent years, real-world graph data are scaling and even benchmark datasets are getting larger. For example, Amazon's product recommendation graph owns over 150M users and 350M products [4], Microsoft Academic Graph consists of more than 120M publications and related authors, venues, organizations, and fields of study [35]. New challenges beyond label annotations have arisen in terms of processing, storage, and deployment [5]. While GNNs-based contrastive learning has advanced the frontiers in many applications, current state-of-the-arts mainly requires large hidden dimensions (e.g.,
512 or even 1024) to learn generalizable _full-precision_ representations [26; 20], making them memory inefficient, storage excessive, and computationally expensive especially for resource-constrained edge devices [45].

In parallel, biological neural networks continue to inspire breakthroughs in modern neural network performance, with prominent examples including spiking neural networks (SNNs) [25]. SNNs are a class of brain-inspired networks with asynchronous discrete and sparse characteristics, which have increasingly manifested their superiority in low energy consumption and inference latency [8]. Unlike traditional artificial neural networks (ANNs), which use floating-point outputs, SNNs use a more biologically plausible approach where neurons communicate via sparse and binarized representations, referred to as'spikes'. Such characteristics make them a promising choice for low-power, mobile, or otherwise hardware-constrained settings. In literature [46; 48], SNNs are proven to consume \(\sim\)100x less energy compared with modern ANNs on the neuromorphic chip (e.g. ROLLs [13]).

Inspired by the success of visual research [16; 46; 15], some recent efforts have been made toward generalizing SNNs to graph data. For example, SpikingGCN [48] and SpikeNet [21] are two representative works combining SNNs with GNN architectures, in which SNNs are utilized as the basic building blocks to model the underlying graph structure and dynamics. Despite the promising potentiality, SNNs have been under-appreciated and under-investigated in graph contrastive learning. This has led to an interesting yet rarely explored research question:

_Can we explore the possibilities of SNNs with contrastive learning schemes to learn sparse, binarized yet generalizable representations?_

Present work.Deviating from the large body of prior works on graph contrastive learning, in this paper we take a new perspective to address self-supervised and binarized representation learning on graphs. We present SpikeGCL, a principled GCL framework built upon SNNs to learn binarized graph representations in a compact and efficient manner. Instead of learning full-precision node representations, we learn sparse and compact 1-bit representations to unlock the power of SNNs on graph data and enable fast inference. In addition, we noticed the problem of vanishing gradients (surprisingly overlooked) during direct training of SNNs and proposed a blockwise training approach to tackle it. Although SpikeGCL uses binarized 1-bit spikes as representations, it comes with both theoretical guarantees and comparable performance in terms of expressiveness and efficacy, respectively. Compared with floating point counterparts, SpikeGCL leads to less storage space, smaller memory footprint, lower power consumption, and faster inference speed, all of which are essential for practical applications such as edge deployment.

Overall, our contributions can be mainly summarized as follows:

* We believe our work is timely. In response to the explosive data expansion, we study the problem of self-supervised and binarized representations learning on graphs with SNNs, whose potential is especially attractive for low-power and resource-constrained settings.
* We propose SpikeGCL to learn binarized representations from large-scale graphs. SpikeGCL exhibits high hardware efficiency, significantly reducing memory consumption by \(\sim\)32x for node representations and energy consumption by up to \(\sim\)7kx. SpikeGCL is a theoretically guaranteed framework with powerful capabilities in learning node representations.
* We address the challenge of training deep SNNs with a simple blockwise learning paradigm. By limiting the backpropagation path to a single block of time steps, we prevent SNNs from suffering from the notorious problem of vanishing gradients.
* Extensive experiments demonstrate that SpikeGCL performs on par with or even sometimes better than state-of-the-art full-precision competitors, across multiple graph datasets of various scales. Our results hold great promise for accurate and fast online inference of graph-based systems.

Figure 1: Comparison between conventional GCL and our proposed SpikeGCL. Instead of using continuous full-precision representations, e.g., 32-bit floating points, SpikeGCL produces sparse and compact 1-bit representations, making them more memory-friendly and computationally efficient for cheap devices with limited resources.
We leave two additional remarks: (1) SpikeGCL is designed to reduce computation, energy, and storage costs from the perspective of compressing graph representations using biological networks, which is orthogonal to existing advances on sampling [11], data condensation [14] or pruning [19], and architectural simplifications [36; 26; 44]. (ii) To the best of our knowledge, our work is the first to explore the feasibility of implementing graph self-supervised learning with SNNs for learning binarized representations. We hope our work will inspire the community to further explore more powerful and efficient algorithms.

## 2 Related Work

**Graph contrastive learning.** Self-supervised representation learning from graph data is a quickly evolving field. The last years have witnessed the emergence of a promising self-supervised learning strategy, referred to as graph contrastive learning (GCL) [23]. Typically, GCL works by contrasting so-called positive samples against negative ones from different graph augmentation views, which has led to the development of several novel techniques and frameworks such as DGI [37], GraphCL [42], GRACE [47], and GGD [44]. Recent research has also suggested that negative samples are not always necessary for graph contrastive learning, with prominent examples including BGRL [36] and CCA-SSG [43]. Negative-sample-free approach paves the way to a simple yet effective GCL method and frees the model from intricate designs. We refer readers to [23] for a comprehensive review of graph contrastive learning. Despite the achievements, current GCL designs mainly focus on task accuracy with a large hidden dimensionality [26; 20], and lack consideration of hardware resource limitations to meet the real-time requirements of edge application scenarios [45].

**Spiking neural networks.** SNNs, which mimic biological neural networks by leveraging sparse and event-driven activations, are very promising for low-power applications. SNNs have been around for a while, and although they have not gained the same level of popularity as GNNs, they have steadily increased their influence in recent years. Over the past few years, SNNs have gained significant popularity in the field of vision tasks, including image classification [46], objection detection [15], and segmentation [16]. Efforts have been paid to incorporate their biological plausibility and leverage their promising energy efficiency into GNN learning. SpikingGCN [48] is a recent endeavor on encoding the node representation through SNNs. As SNNs are intrinsically dynamic with abundant temporal information conveyed by spike timing, SpikeNet [22] then proceeds to model the temporal evolution of dynamic graphs via SNNs. In spite of the increasing research interests, the benefits of SNNs have not been discovered in GCL yet.

**Binarized graph representation learning.** Model size, memory footprint, and energy consumption are common concerns for many real-world applications [1]. In reaction, binarized graph representation learning has been developed in response to the need for more efficient and compact representations for learning on large-scale graph data. Currently, there are two lines of research being pursued in this area. One line of research involves directly quantizing graphs by using discrete hashing techniques [41; 1] or estimating gradients of non-differentiable quantization process [4]. Another research line to produce binary representations using deep learning methods are those binarized networks, which are designed for fast inference and small memory footprint, in which binary representation is only a by-product [39]. On one hand, binarized graph representation learning is a novel and promising direction that aims to encode graphs into compact binary vectors that can facilitate efficient storage, retrieval, and computation. On the other hand, SNNs that use discrete spikes rather than real values to communicate between neurons are naturally preferred for learning binary representations. Therefore, it is intuitively promising to explore the marriage between them.

## 3 Preliminaries

**Problem formulation.** Let \(\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})\) denote an attributed undirected graph where \(\mathcal{V}=\{v_{i}\}_{i=1}^{N}\) and \(\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}\) are a set of nodes and edges, respectively. We focus primarily on undirected graphs though it is straightforward to extend our study to directed graphs. \(\mathcal{G}\) is associated with an \(d\)-dimensional attribute feature matrix \(\mathbf{X}=\{x_{i}\}_{i=1}^{N}\in\mathbb{R}^{N\times d}\). In the self-supervised learning setting of this work, our objective is to learn an encoder \(f_{\theta}:\mathbb{R}^{d}\rightarrow\{0,1\}^{d}\) parameterized by \(\theta\), which maps between the space of graph \(\mathcal{G}\) and their low-dimensional and _binary_ latent representations
\(\mathbf{Z}=\{z_{i}\}_{i=1}^{N}\), such that \(f_{\theta}(\mathcal{G})=\mathbf{Z}\in\{0,1\}^{N\times d}\) given \(d\) the embedding dimension. Note that for simplicity we assume the feature dimensions are the same across all layers.

Spiking neural networks.Throughout existing SNNs, the integrate-fire (IF) [31] model and its variants are commonly adopted to formulate the spiking neuron and evolve into numerous variants with different biological features. As the name suggests, IF models have three fundamental characteristics: (i) **Integrate**. The neuron integrates current by means of the capacitor over time, which leads to a charge accumulation; (ii) **Fire**. When the membrane potential has reached or exceeded a given threshold \(V_{\text{th}}\), it fires (i.e., emits a spike). (iii) **Reset**. After that, the membrane potential is reset, and here we introduce two types of reset [30]: _reset to zero_ which always sets the membrane potential back to a constant value \(V_{\text{reset}}<V_{\text{th}}\), typically zero, whereas _reset by subtraction_ subtracts the threshold \(V_{\text{th}}\) from the membrane potential at the time where the threshold is exceeded: We use a unified model to describe the dynamics of IF-based spiking neurons:

**Integrate:** \[V^{t} =\Psi(V^{t-1},I^{t}),\] (1) **Fire:** \[S^{t} =\Theta(V^{t}-V_{\text{th}}),\] (2) **Reset:** \[V^{t} =\left\{\begin{array}{rl}S^{t}V_{\text{reset}}+(1-S^{t})V^{t},& \text{reset to zero},\\ S^{t}(V^{t}-V_{\text{th}})+(1-S^{t})V^{t},&\text{reset by subtraction}, \end{array}\right.\] (3)

where \(I^{t}\) and \(V^{t}\) denote the input current and membrane potential (voltage) at time-step \(t\), respectively. The decision to fire a spike in the neuron output is carried out according to the Heaviside step function \(\Theta(\cdot)\), which is defined by \(\Theta(x)=1\) if \(x\geq 0\) and \(0\) otherwise. The function \(\Psi(\cdot)\) in Eq. (1) describes how the spiking neuron receives the resultant current and accumulates membrane potential. We have IF [31] and its variant Leaky Integrate-and-Fire (LIF) [10] model, formulated as follows:

**IF:** \[V^{t} =V^{t-1}+I^{t},\] (4) **LIF:** \[V^{t} =V^{t-1}+\frac{1}{\tau_{m}}\left(I^{t}-(V^{t-1}-V_{\text{reset}} )\right),\] (5)

where \(\tau_{m}\) in Eq. (5) represents the membrane time constant to control how fast the membrane potential decays, leading to the membrane potential charges and discharges exponentially in response to the current inputs. Typically, \(\tau_{m}\) can also be optimized automatically instead of manually tuning to learn different neuron dynamics during training, which we referred to as Parametric LIF (PLIF) [7]. In this paper, the surrogate gradient method is used to define \(\Theta^{\prime}(x)\triangleq\sigma^{\prime}(\alpha x)\) during error back-propagation, with \(\sigma(\cdot)\) denote the surrogate function such as Sigmoid and \(\alpha\) the smooth factor [21].

## 4 Spiking Graph Contrastive Learning (SpikeGCL)

In this section, we introduce our proposed SpikeGCL framework for learning binarized representations. SpikeGCL is a simple yet effective derivative of the standard GCL framework with minimal but effective modifications on the encoder design coupled with SNNs. In what follows, we first shed light on building sequential inputs for SNNs from a single non-temporal graph (SS 4.1) and depict how to binarize node representations using SNNs (SS 4.2). Then, we explain the architectural overview and detailed components of SpikeGCL one by one (SS 4.3). Finally, we employ blockwise learning for better training of deep SNNs (SS 4.4).

### Grouping node features

Typically, SNNs require sequential inputs to perform the integrated-and-fire process and emit spikes. One major challenge is how to formulate such inputs from a single non-temporal graph. A common practice introduced in the literature is to repeat the graph multiple times (i.e., a given time window \(T\)), typically followed by probabilistic encoding methods (e.g., Bernoulli encoding) to generate diverse input graphs [48; 40]. However, this will inevitably introduce high computational and memory overheads, becoming a major bottleneck for SNNs to scale to large graphs.

In this work, we adopt the approach of partitioning the graph data into different groups rather than repeating the graph multiple times to construct the sequential inputs required for SNNs. Given a time window \(T\)\((T>1)\) and a graph \(\mathcal{G}\), we uniformly partition the node features into the following
groups1: \(\mathbf{X}=[\mathbf{X}^{1},\ldots,\mathbf{X}^{T}]\) along the feature dimension, where \(\mathbf{X}^{t}\in\mathbb{R}^{N\times\frac{d}{t}}\) consists of the group of features in the \(t\)-th partition. In cases where \(d\) cannot be divided by \(T\), we have \(\mathbf{X}^{t}\in\mathbb{R}^{N\times(d//T)}\) for \(t<T\) and \(\mathbf{X}^{T}\in\mathbb{R}^{N\times(d\ mod\ T)}\). Thus we have \(T\) subgraphs as

Footnote 1: Exploring non-uniform partition with clustering methods is an important direction for future work.

\[\hat{\mathcal{G}}=[\mathcal{G}^{1},\ldots,\mathcal{G}^{T}]=[(\mathcal{V}, \mathcal{E},\mathbf{X}^{1}),\ldots,(\mathcal{V},\mathcal{E},\mathbf{X}^{T})]. \tag{6}\]

Each subgraph in \(\hat{\mathcal{G}}\) shares the same graph structure but only differs in node features. Note that \([\mathbf{X}^{1},\ldots,\mathbf{X}^{T}]\) are _non-overlapping_ groups, which avoids unnecessary computation on input features. Our strategy leads to huge memory and computational benefits as each subgraph in the graph sequence \(\hat{\mathcal{G}}\) only stores a subset of the node features instead of the whole set. This is a significant improvement over previous approaches such as SpikingGCN [48], as it offers substantial benefits in terms of computational and memory complexity.

### Binarizing graph representations

In GCL, GNNs are widely adopted as encoders for representing graph-structured data. GNNs generally follow the canonical _message passing_ scheme in which each node's representation is computed recursively by aggregating representations ("messages") from its immediate neighbors [17; 11]. Given an \(L\) layer GNN \(f_{\theta}\), the updating process of the \(l\)-th layer could be formulated as:

\[h_{u}^{(l)}=\mathrm{COMBINE}^{(l)}\left(\left\{h_{u}^{(l-1)},\mathrm{AGGREGATE} ^{(l)}\left(\left\{h_{v}^{(l-1)}:v\in\mathcal{N}_{u}\right\}\right)\right\}\right) \tag{7}\]

where \(\mathrm{AGGREGATE}(\cdot)\) is an aggregation function that aggregates features of neighbors, and \(\mathrm{COMBINE}(\cdot)\) denotes the combination of aggregated features from a central node and its neighbors. \(h_{u}^{(l)}\) is the embedding of node \(u\) at the \(l\)-th layer of GNN, where \(l\in\{1,\ldots,L\}\) and initially \(h_{u}^{(0)}=x_{u}\); \(\mathcal{N}_{u}\) is the set of neighborhoods of \(u\). After \(L\) rounds of aggregation, each node \(u\in V\) obtains its representation vector \(h_{u}^{(L)}\). The final node representation is denoted as \(\mathbf{H}=[h_{1}^{(L)},\ldots,h_{N}^{(L)}]\).

We additionally introduce SNNs to the encoder for binarizing node representations. SNNs receive continuous values and convert them into binary spike trains, which opens up possibilities for mapping graph structure from continuous Euclidian space into discrete and binary one. Given a graph sequence \(\tilde{\mathcal{G}}=[\mathcal{G}^{1},\ldots,\mathcal{G}^{T}]\), we form \(T\) graph encoders \([f_{\theta_{1}}^{1},\ldots,f_{\mathcal{G}_{T}}^{T}]\) accordingly such that each encoder \(f_{\theta_{\varepsilon}}^{t}\) maps a graph \(\mathcal{G}_{t}\) to its hidden representations \(\mathbf{H}^{t}\). Here we denote \(f_{\theta}=[f_{\theta_{1}}^{1},\ldots,f_{\theta_{T}}^{T}]\) with slightly abuse of notations. Then, a spiking neuron is employed on the outputs to generate binary spikes in a dynamic manner:

\[\mathbf{S}^{t}=\Theta\left(\Psi(V^{t-1},\mathbf{H}^{t})-V_{\text{th}}\right), \quad\mathbf{H}^{t}=f_{\theta}^{t}(\mathcal{G}^{t}), \tag{8}\]

where \(\Psi(\cdot)\) denotes a spiking function that receives the resultant current from the encoder and accumulates membrane potential to generate the spikes, such as IF or LIF introduced in SS 3. \(\mathbf{S}^{t}\in\{0,1\}^{N\times\frac{d}{t}}\) is the output spikes at time step \(t\). In this regard, the binary representations are derived by taking historical spikes in each time step with a _concatenate_ pooling [21], i.e., \(\mathbf{Z}=\left(\mathbf{S}^{1}||\cdots||\mathbf{S}^{T}\right)\).

### Overall framework

We decompose our proposed SpikeGCL from four dimensions: (i) augmentation, (ii) encoder, (iii) predictor head (a.k.a. decoder), and (v) contrastive objective. These four components constitute the design space of interest in this work. We present an architectural overview of SpikeGCL in Figure 2(a), as well as the core step of embedding generation in Figure 2(b).

**Augmentation.** Augmentation is crucial for contrastive learning by providing different graph views for contrasting. With a given graph \(\mathcal{G}\), SpikeGCL involves bi-level augmentation techniques, i.e., topology (structure) and feature augmentation, to construct its corrupted version \(\tilde{\mathcal{G}}\). To be specific, we harness two ways for augmentation: _edge dropping_ and _feature shuffling_. Edge dropping randomly drops a subset of edges from the original graph, while feature shuffling gives a new permutation of the node feature matrix along the feature dimension. In this way, we obtain the corrupted version of an original graph \(\tilde{\mathcal{G}}=\{\mathcal{V},\tilde{\mathcal{E}},\tilde{\mathbf{X}}\}\), where \(\tilde{\mathcal{E}}\subseteq\mathcal{E}\) and \(\tilde{\mathbf{X}}\) denotes the column-wise shuffled feature matrix such that \(\tilde{\mathbf{X}}=\mathbf{X}[:,\mathcal{P}]\) with \(\mathcal{P}\) the new permutation of features. Since we partition node features in a
sequential manner that is sensitive to the permutation of input features, feature shuffling is able to offer _hard negative_ samples for contrastive learning.

**Encoder.** As detailed in SS 4.2, our encoder is a series of peer GNNs corresponding to each group of input node features, followed by a spiking neuron to binarize the representations. Among the many variants of GNNs, GCN [17] is the most representative structure, and we adopt it as the basic unit of the encoder in this work. The number of individual GNNs in the encoder is relative to the number of time steps \(T\), which makes the model excessively complex and potentially lead to overfitting if \(T\) is large. We circumvent this problem by **parameter sharing**. Note that only the first layer needs to be different in response to diverse groups of features. For the remaining \(L-1\) layers, parameters are shared across peer GNNs to prevent excessive memory footprint and the overfitting issue.

Predictor head (decoder).A non-linear transformation named projection head is adopted to map binarized representations to continuous latent space where the contrastive loss is calculated, as advocated in [36; 43]. As a proxy for learning on discrete spikes, we employ a single-layer perceptron (MLP) to the learned representations, i.e., \(g_{\phi}(z_{u})=\text{MLP}(z_{u}),\forall u\in\mathcal{V}\). Since the hidden representations are binary, we can instead use'masked summation' [21] to enjoy the sparse characteristics and avoid expensive matrix multiplication computations during training and inference.

Contrastive objective.Contrastive objectives are widely used to measure the similarity or distance between positive and negative samples. Rather than explicitly maximizing the discrepancy between positive and negative pairs as most existing works on contrastive learning have done, the 'contrastiveness' in this approach is reflected in the diverse 'distance' naturally measured by a parameterized model \(g_{\phi}(\cdot)\). Specifically, we employ a margin ranking loss (MRL) [3] to \(g_{\phi}(\cdot)\):

\[\mathcal{J}=\frac{1}{N}\sum_{u\in\mathcal{V}}\max(0,g_{\phi}(z_{u})-g_{\phi}(z _{u}^{-})+m), \tag{9}\]

with the margin \(m\) a hyperparameter to make sure the model disregards abundant far (easy) negatives and leverages scarce nearby (hard) negatives during training. \(z_{u}\) and \(z_{u}^{-}\) is the representation of node \(u\) obtained from original graph sequence \(\tilde{\mathcal{G}}\) and its corrupted one, respectively. MRL forces the score of positive samples to be lower (towards zero) and assigns a higher score to negative samples by a margin of at least \(m\). Therefore, positive samples are separated from negative ones.

### Blockwise surrogate gradient learning

From an optimization point of view, SNNs lack straightforward gradient calculation for backward propagation, and also methods that effectively leverage their inherent advantages. Recent attempts have been made to surrogate gradient learning [18], an alternative of gradient descent that avoids the non-differentiability of spike signals. As a backpropagation-like training method, it approximates the backward gradients of the hard threshold function using a smooth activation function (such as Sigmoid) during backpropagation. Such a technique enables us to train general forms of deep SNNs directly in a fashion similar to GNNs, while preserving their binary and dynamical nature. At present, the surrogate learning technique plays an extremely important role in advanced methods to learn SNNs properly [48; 40; 21; 6; 7].

Figure 2: Overview of proposed SpikeGCL framework. (a) SpikeGCL follows the standard GCL framework, which learns binary representation by contrasting positive and negative samples by a margin ranking loss. (b) SpikeGCL first partitions node features into \(T\) non-overlapping groups, each of which is then fed to an encoder whereby spiking neurons represent nodes of graph as 1-bit spikes.
Despite the promising results, surrogate gradient learning has its own drawbacks. Typically, SNNs require relatively large time steps to approximate continuous inputs and endow the network with better expressiveness [48]. However, a large time step often leads to many problems such as high overheads and network degradation [6]. Particularly, the training of SNNs also comes with a serious vanishing gradient problem in which gradients quickly diminish with time steps. We have further discussion with respect to the phenomenon in Appendix B. These drawbacks greatly limit the performance of directly trained SNNs and prevent them from going 'deeper' with long sequence inputs.

In this paper, we explore alternatives to end-to-end backpropagation in the form of surrogate gradient learning rules, leveraging the latest advances in self-supervised learning [34] to address the above limitation in SNNs. Specifically, we propose a blockwise training strategy that separately learns each block with a local objective. We consider one or more consecutive time steps as a single block, and limit the length of the backpropagation path to each of these blocks. Parameters within each block are optimized locally with a contrastive learning objective, using stop-gradient to prevent the gradients from flowing through the blocks. A technical comparison between the existing end-to-end surrogate learning paradigm and our proposed local training strategy is shown in Figure 3. By limiting the length of the backpropagation path to a single block, it solves the problem of vanishing gradients, as long as each block has a small depth/size. Furthermore, each block can be trained either simultaneously or in a purely sequential manner, offering different ways to efficiently train deep SNNs with lower memory footprints.

## 5 Theoretical guarantees

There are several works on designing practical graph-based SNNs that achieve comparable performance with GNNs [22; 40; 48]. However, the basic principles and theoretical groundwork for their performance guarantees are lacking, and related research only shows the similarity between the IF neuron model and the ReLU activation [30]. In this section, we are motivated to bridge the gap between SNNs and GNNs. We present an overview of theoretical results regarding the approximation properties of SpikeGCL with respect to its full-precision counterparts through the following theorem:

**Theorem 1** (Informal).: _For any full-precision GNN with a hidden dimension of \(d/T\), there exists a corresponding SpikeGCL such that its approximation error, defined as the \(\ell_{2}\) distance between the firing rates of the SpikeGCL representation and the GNN representation at any single node, is of the order \(\Theta(1/T)\)._

In the order relation \(\Theta(\frac{1}{T})\), we hide factors dependent upon the underlying GNN structure, network depth as well as characteristics of the input graph, which will be stated precisely in Theorem 1 in Appendix A. Theorem 1 suggest that with a sufficiently large node feature dimension such that we may set a large simulation length \(T\), we are able to (almost) implement vanilla GNNs with a much better computational and energy efficiency. Moreover, the approximation is defined using the firing rates of SNN outputs, which measures only a restricted set of inductive biases offered by SpikeGCL. Consequently, we empirically observe that a moderate level of \(T\) might also provide satisfactory performance in our experiments. Our analysis is presented to shed insights on the connection between the computational neuroscience model (e.g., SNNs) and the machine learning neural network model (e.g., GNNs). This connection has been analytically proven under certain conditions and empirically demonstrated through our experiments (see SS 6). It can serve as the theoretical basis for potentially combining the respective merits of the two types of neural networks.

## 6 Experiments

In this section, we perform experimental evaluations to demonstrate the effectiveness of our proposed SpikeGCL framework. Due to space limitations, we present detailed experimental settings and

Figure 3: A comparison between two backpropagation learning paradigms. The backpropagation path during blockwise training is limited to a single block of networks to avoid a large memory footprint and vanishing gradient problem.
additional results in Appendix E and Appendix F accomplishing the submission. Code is available at [https://github.com/EdisonLeeeee/SpikeGCLfor](https://github.com/EdisonLeeeee/SpikeGCLfor) reproducibility.

**Datasets.** We evaluate SpikeGCL on several graph benchmarks with different scales and properties. Following prior works [43, 36, 48], we adopt 9 common benchmark graphs, including two co-purchase graphs, i.e., Amazon-Photo, Amazon-Computer [33], two co-author graphs, i.e., Coauthor-CS and Coauthor-Physics [33], three citation graphs, i.e., Cora, CiteSeer, Pubmed [32], as well as two large-scale datasets ogbn-arXiv and ogbn-MAG from Open Graph Benchmark [12]. The detailed introduction and statistics of these datasets are presented in Appendix E.

**Baselines.** We compare our proposed methods to a wide range of baselines that fall into four categories: (i) full-precision (supervised) GNNs: GCN [17] and GAT [38]; (ii) 1-bit quantization-based GNNs: Bi-GCN [39], BinaryGNN [1] and BANE [41]; (iii) contrastive methods: DGI [37], GRACE [47], CCA-SSG [43], BGRL [36], SUGRL [26], and GGD [44]; (iv) Graph SNNs: Spiking-GCN [48], SpikeNet [21], GC-SNN and GA-SNN [40]. SpikeNet is initially designed for dynamic graphs, we adapt the author's implementation to static graphs following the practice in [48, 40]. The hyperparameters of all the baselines were configured according to the experimental settings officially reported by the authors and were then carefully tuned in our experiments to achieve their best results.

**Overall performance.** The results on six graph datasets are summarized in Table 1. We defer the results on Cora, CiteSeer, and Pubmed to Appendix. Table 1 shows a significant performance gap between full-precision methods and 1-bit GNNs, particularly with the unsupervised method BANE. Even supervised methods Bi-GCN and binaryGNN struggle to match the performance of full-precision methods. In contrast, SpikeGCL competes comfortably and sometimes outperforms advanced full-precision methods. When compared to full-precision GCL methods, our model approximates about 95%\(\sim\)105% of their performance capability across all datasets. Additionally, SpikeGCL performs comparably to supervised graph SNNs and offers the possibility to scale to large graph datasets (e.g., arXiv and MAG). Surprisingly, SpikeGCL achieves a new state-of-the-art performance on the MAG dataset. Overall, the results indicate that binary representation does not necessarily lead to accuracy loss as long as it is properly trained.

**Efficiency.** We first provide a comparison of the number of parameters and theoretical energy consumption of SpikeGCL and full-precision GCL methods. The computation of theoretical energy consumption follows existing works [48, 46, 39] and is detailed in Appendix F. Table 6 summarizes the results in terms of model size and energy efficiency compared to full-precision GCL methods. SpikeGCL consistently shows better efficiency across all datasets, achieving \(\sim\)7k less energy consumption and \(\sim\)1/60 model size on MAG. As the time step \(T\) is a critical hyperparameter for SNNs

\begin{table}
\begin{tabular}{l|c c|c c c c c c} \hline \hline
**Method** & **U** & **S** & **B** & **Computers** & **Photo** & **CS** & **Physics** & **arXiv** & **MAG** \\ \hline GCN [17] & & & 86.5\(\pm\)0.5 & 92.4\(\pm\)0.2 & 92.5\(\pm\)0.4 & 95.7\(\pm\)0.5 & 70.4\(\pm\)0.3 & 30.1\(\pm\)0.3 \\ GAT [38] & & & 86.9\(\pm\)0.2 & 92.5\(\pm\)0.3 & 92.3\(\pm\)0.2 & 95.4\(\pm\)0.3 & 70.6\(\pm\)0.3 & 30.5\(\pm\)0.3 \\ \hline SpikeNet [21] & ✓ & & 87.3\(\pm\)0.6 & 92.9\(\pm\)0.1 & **93.4\(\pm\)**0.2 & **95.8\(\pm\)**0.7 & 66.8\(\pm\)0.1 & - \\ SpikingGCN [48] & ✓ & & 85.9\(\pm\)0.6 & 92.6\(\pm\)0.7 & 92.6\(\pm\)0.3 & 91.5\(\pm\)0.5 & - & - \\ GC-SNN [40] & ✓ & & 88.2\(\pm\)0.6 & 92.8\(\pm\)0.1 & 93.0\(\pm\)0.4 & 95.6\(\pm\)0.7 & - & - \\ GA-SNN [40] & ✓ & & 88.1\(\pm\)0.1 & **93.5\(\pm\)**0.6 & 92.2\(\pm\)0.1 & 95.8\(\pm\)0.5 & - & - \\ \hline Bi-GCN [39] & & ✓ & 86.4\(\pm\)0.3 & 92.1\(\pm\)0.9 & 91.0\(\pm\)0.7 & 93.3\(\pm\)1.1 & 66.0\(\pm\)0.8 & 28.2\(\pm\)0.4 \\ BinaryGNN [1] & & ✓ & 87.8\(\pm\)0.2 & 92.4\(\pm\)0.2 & 91.2\(\pm\)0.1 & 95.0\(\pm\)0.1 & 67.2\(\pm\)0.9 & - \\ BANE [41] & ✓ & ✓ & 72.7\(\pm\)0.3 & 78.2\(\pm\)0.3 & 92.8\(\pm\)0.1 & 93.4\(\pm\)0.4 & \textgreater{}3days & \textgreater{}3days \\ \hline DGI [37] & ✓ & & 84.0\(\pm\)0.5 & 91.6\(\pm\)0.2 & 92.2\(\pm\)0.6 & 94.5\(\pm\)0.5 & 65.1\(\pm\)0.4 & 31.4\(\pm\)0.3 \\ GRACE [47] & ✓ & & 86.3\(\pm\)0.3 & 92.2\(\pm\)0.2 & 92.9\(\pm\)0.0 & 95.3\(\pm\)0.0 & 68.7\(\pm\)0.4 & 31.5\(\pm\)0.3 \\ CCA-SSG [43] & ✓ & & 88.7\(\pm\)0.3 & 93.1\(\pm\)0.1 & 93.3\(\pm\)0.1 & 95.7\(\pm\)0.1 & 71.2\(\pm\)0.2 & 31.8\(\pm\)0.4 \\ BGRL [36] & ✓ & & **90.3\(\pm\)**0.2 & 93.2\(\pm\)0.3 & 93.3\(\pm\)0.1 & 95.7\(\pm\)0.0 & 71.6\(\pm\)0.1 & 31.1\(\pm\)0.1 \\ SUGRL [26] & ✓ & & 88.9\(\pm\)0.2 & 93.2\(\pm\)0.4 & 93.4\(\pm\)0.0 & 95.2\(\pm\)0.0 & 68.8\(\pm\)0.4 & 32.4\(\pm\)0.1 \\ GGD [44] & ✓ & & 88.0\(\pm\)0.1 & 92.9\(\pm\)0.2 & 93.1\(\pm\)0.1 & 95.3\(\pm\)0.0 & **71.6\(\pm\)**0.5 & 31.7\(\pm\)0.7 \\ \hline SpikeGCL & ✓ & ✓ & 88.9\(\pm\)0.3 & 93.0\(\pm\)0.1 & 92.8\(\pm\)0.1 & 95.2\(\pm\)0.6 & **70.1\(\pm\)**0.1 & **32.0\(\pm\)**0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Classification accuracy (%) on six large scale datasets. The best result for each dataset is highlighted in **red**. Darker colors indicate larger performance gaps between SpikeGCL and the best results. The missing results are due to the out-of-memory error on a GPU with 24GB memory. (**U**: unsupervised or self-supervised; **S**: spike-based; **B**: binarized)
to better approximate real-valued inputs with discrete spikes, we compare the efficiency of SpikeGCL and other graph SNNs in terms of accuracy, training time, GPU memory usage, and theoretical energy consumption with varying time steps from 5 to 30. The results on the Computers dataset are shown in Figure 4. It is observed that SpikeGCL and other graph SNNs benefit from increasing time steps, generally improving accuracy as the time step increases. However, a large time step often leads to more overheads. In our experiments, a larger time step can make SNNs inefficient and become a major bottleneck to scaling to large graphs, as demonstrated by increasing training time, memory usage, and energy consumption. Nevertheless, the efficiency of SpikeGCL is less affected by increasing time steps. The results demonstrate that SpikeGCL, coupled with compact graph sequence input and blockwise training paradigm, alleviates such a drawback of training deep SNNs. Overall, SpikeGCL is able to significantly reduces computation costs and enhance the representational learning capabilities of the model simultaneously.

**Convergence.** We compared the convergence speed of SpikeGCL with full-precision GCL baselines on Computers, which is shown in Figure 5. It is observed that negative-sample-free methods generally converged faster than negative-sample-based ones. However, they still required sufficient epochs to gradually improve their performance. In contrast, SpikeGCL that trained in a blockwise training paradigm demonstrates a significantly faster convergence speed over full-precision methods. In particular, 1-2 epochs are sufficient for SpikeGCL to learn good representations. Overall, our experiments demonstrate the effectiveness of SpikeGCL in improving the convergence speed of SNNs and provide insights into the benefits of the blockwise training paradigm.

## 7 Conclusion

In this work, we present SpikeGCL, a principled graph contrastive learning framework to learn binarized and compact representations for graphs at scale. SpikeGCL leverages the sparse and binary characteristics of SNNs, as well as contrastive learning paradigms, to meet the challenges of increasing graph scale and limited label annotations. The binarized representations learned by SpikeGCL require less memory and computations compared to traditional GCL, which leads to potential benefits for energy-efficient computing. We provide theoretical guarantees and empirical results to demonstrate that SpikeGCL is an efficient yet effective approach for learning binarized graph representations.

\begin{table}
\begin{tabular}{l|c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Computers**} & \multicolumn{2}{c}{**CS**} & \multicolumn{2}{c}{**Physics**} & \multicolumn{2}{c}{**arXiv**} & \multicolumn{2}{c}{**MAG**} \\ \cline{2-10}  & \#Param\(\downarrow\) & Energy\(\downarrow\) & \#Param\(\downarrow\) & Energy\(\downarrow\) & \#Param\(\downarrow\) & Energy\(\downarrow\) & \#Param\(\downarrow\) & Energy\(\downarrow\) & \#Param\(\downarrow\) & Energy\(\downarrow\) \\ \hline DGI & 917.5 & 0.5 & 4008.9 & 8 & 4833.3 & 6 & 590.3 & 5 & 590.3 & 568 \\ GRACE & 656.1 & 1.1 & 3747.5 & 17 & 4571.9 & 13 & 328.9 & 21 & 328.9 & 4463 \\ CCA-SSG & 262.4 & 17 & 1808.1 & 152 & 2220.2 & 352 & 98.8 & 78 & 98.8 & 340 \\ BGRL & 683.4 & 25 & 3749.8 & 163 & 4574.2 & 373 & 331.2 & 180 & 331.2 & 787 \\ SUGRL & 193.8 & 13 & 2131.2 & 147 & 2615.1 & 342 & 99.5 & 26 & 99.5 & 117 \\ GGD & 254.7 & 15 & 3747.3 & 140 & 4571.6 & 340 & 30.0 & 100 & 30.0 & 1400 \\ \hline Average & 490.4 & 11.9 & 3198.7 & 104.5 & 3906.0 & 237.6 & 246.4 & 68.3 & 246.4 & 1279.1 \\ SpikeGCL & 60.9 & 0.38 & 460.7 & 104.8 & 564.4 & 10.6 & 73.8 & 0.2 & 6.6 & 18 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The parameter size (KB) and theoretical energy consumption (mJ) of various GCL methods. The row in ‘Average’ denotes the averaged results of full-precision GCL baselines. Darker color in SpikeGCL indicates a larger improvement in efficiency over the baselines.

Figure 4: Comparison of SpikeGCL and other graph SNNs in terms of accuracy (%), training time (s), memory usage (GB), and energy consumption (mJ), respectively.

Figure 5: Convergence speed comparison among SpikeGCL and full-precision methods. Blue: negative-sample-based methods; green: negative-sample-free methods
In our extensive experimental evaluation, SpikeGCL is able to achieve performance on par with advanced baselines using full-precision or 1-bit representations while demonstrating significant efficiency advantages in terms of parameters, speed, memory usage, and energy consumption. We believe that our work is promising from a neuroscientific standpoint, and we hope it will inspire further research toward efficient graph representation learning.
