PTGB: Pre-Train Graph Neural Networks for Brain Network Analysis
The human brain is the central hub of the neurobiological system, controlling
behavior and cognition in complex ways. Recent advances in neuroscience and
neuroimaging analysis have shown a growing interest in the interactions between
brain regions of interest (ROIs) and their impact on neural development and
disorder diagnosis. As a powerful deep model for analyzing graph-structured
data, Graph Neural Networks (GNNs) have been applied for brain network
analysis. However, training deep models requires large amounts of labeled data,
which is often scarce in brain network datasets due to the complexities of data
acquisition and sharing restrictions. To make the most out of available
training data, we propose PTGB, a GNN pre-training framework that captures
intrinsic brain network structures, regardless of clinical outcomes, and is
easily adaptable to various downstream tasks. PTGB comprises two key
components: (1) an unsupervised pre-training technique designed specifically
for brain networks, which enables learning from large-scale datasets without
task-specific labels; (2) a data-driven parcellation atlas mapping pipeline
that facilitates knowledge transfer across datasets with different ROI systems.
Extensive evaluations using various GNN models have demonstrated the robust and
superior performance of PTGB compared to baseline methods.
Yi Yang, Hejie Cui, Carl Yang
# PTGB: Pre-Train Graph Neural Networks for

###### Abstract

The human brain is the central hub of the neurobiological system, controlling behavior and cognition in complex ways. Recent advances in neuroscience and neuroimaging analysis have shown a growing interest in the interactions between brain regions of interest (ROIs) and their impact on neural development and disorder diagnosis. As a powerful deep model for analyzing graph-structured data, Graph Neural Networks (GNNs) have been applied for brain network analysis. However, training deep models requires large amounts of labeled data, which is often scarce in brain network datasets due to the complexities of data acquisition and sharing restrictions. To make the most out of available training data, we propose PTGB, a GNN pre-training framework that captures intrinsic brain network structures, regardless of clinical outcomes, and is easily adaptable to various downstream tasks. PTGB comprises two key components: (1) an unsupervised pre-training technique designed specifically for brain networks, which enables learning from large-scale datasets without task-specific labels; (2) a data-driven parcellation atlas mapping pipeline that facilitates knowledge transfer across datasets with different ROI systems. Extensive evaluations using various GNN models have demonstrated the robust and superior performance of PTGB compared to baseline methods.

P TGB: Pre-Train Graph Neural Networks for Brain Network Analysis

## 1 Introduction

Brain network analysis has attracted considerable interest in neuroscience studies in recent years. A brain network is essentially a connected graph constructed from different raw imaging modalities such as Diffusion Tensor Imaging (DTI) and functional Magnetic Resonance Imaging (fMRI), where nodes are composed by the anatomical regions of interest (ROIs) given predefined parcellation atlas, and connections are usually formed with the correlations among ROIs.
Effective brain network analysis plays a pivotal role in understanding the biological structures and functions of complex neural systems, which potentially helps the early diagnosis of neurological disorders and facilitates neuroscience research (Martensson et al., 2018; Yahata et al., 2016; Lindquist, 2008; Smith, 2012).

Graph Neural Networks (GNNs) have emerged as a powerful tool for analyzing graph-structured data, delivering impressive results on a wide range of network datasets, including social networks, recommender systems, knowledge graphs, protein and gene networks, and molecules, among others (Kipf and Welling, 2017; Hamilton et al., 2017; Schlichtkrull et al., 2018; Vashishth et al., 2020; Xu et al., 2019; Ying et al., 2018; Zhang et al., 2020; Liu et al., 2022; Xiong et al., 2020; Cui et al., 2022; Xu et al., 2022). These models have proven their ability to learn powerful representations and efficiently compute complex graph structures, making them well-suited for various downstream tasks. In the field of neuroscience, GNN has been applied to brain network analysis, specifically for graph-level classification/regression (Ying et al., 2018; Xu et al., 2019; Errica et al., 2020; Luo et al., 2022; Dai et al., 2023; Xu et al., 2023a) and important vertex/edge identification (Ying et al., 2019; Luo et al., 2020; Vu and Thai, 2020; Yu et al., 2023; Kan et al., 2022c), towards tasks such as connectome-based disease prediction and multi-level neural pattern discovery. However, deep learning models, including GNNs, require large amounts of labeled data to achieve optimal performance (Hu et al., 2020; You et al., 2020; Zhu et al., 2021a). While neuroimaging datasets are available from national neuroimaging studies such as the ABCD (Casey et al., 2018), ADNI (Hinrichs et al., 2009), and PPMI (Aleksovski et al., 2018), these datasets are still relatively small compared to graph datasets from other domains, such as datasets with 41K to 452K graphs on OGB (Hu et al., 2020) and datasets with thousands to millions of graphs on NetRepo (Rossi and Ahmed, 2016)). The limited amount of data can result in overfitting when training deep models.

Transfer learning offers a solution to the challenge of limited data availability in training deep models. It allows a model pre-trained on large-scale source datasets to be adapted to smaller target datasets while maintaining robust performance. However, the success of transfer learning depends on the availability of similar supervision labels on the source and target dataset. This is not always feasible in large-scale public studies, particularly in the field of brain network analysis. Self-supervised pre-training has been shown to be effective in various domains, such as computer vision (He et al., 2020; Chen et al., 2020), natural language processing (Devlin et al., 2019; Yu et al., 2022), and graph mining (Sun et al., 2022). We aim to explore a self-supervised pre-training approach for GNNs on brain networks that is not restricted by task-specific supervision labels. Despite the promising potential, unique challenges still need to be addressed to achieve effective disease prediction. One of the major challenges is the inconsistent ROI parcellation systems in constructing different brain network datasets, which hinders the transferability of pre-trained models across datasets. The process of parcellating raw imaging data into brain networks is highly complex and usually done ad hoc by domain experts for each study, making it unrealistic to expect every institution to follow the same parcellation system. Although some institutions may release preconstructed brain network datasets (Di Martino et al., 2014), the requirement for universal adherence to a single parcellation system is infeasible.

To tackle the challenge of insufficient training data for GNNs in brain network analysis, we present **P**-**T**raining **G**raph neural networks for **B**rain networks (PTGB), a fully unsupervised pre-training approach that captures shared structures across brain network datasets. PTGB adapts the data-efficient MAML (Finn et al., 2017) with a two-level contrastive learning strategy based on the naturally aligned node systems of brain networks across individuals. Additionally, to overcome the issue of diverse parcellation systems, we introduce a novel data-driven atlas mapping technique. This technique transforms the original features into low-dimensional representations in a uniform embedding space and aligns them using variance-based projection, which incorporates regularizations that preserve spatial relationships, consider neural modules, and promote sparsity.

In summary, our contributions are three-folded:

* We present an unsupervised pre-training approach for GNNs on brain networks, addressing the issue of resource-limited training.
* We propose a two-level contrastive sampling strategy tailored for GNN pre-training on brain networks, which combines with a data-driven brain atlas mapping strategy that employs customized regularizations and variance-based sorting to enhance cross-dataset learning.
* Our experiments against shallow and deep baselines demonstrate the effectiveness of our proposed
PTGB. Further, we provide an in-depth analysis to understand the influence of each component.

## 2 Related Work

GNNs for Brain Network Analysis.GNNs are highly effective for analyzing graph-structured data and there have been some pioneering attempts to use them for predicting diseases by learning over brain networks. For example, BrainGNN (Li et al., 2021) proposes ROI-aware graph convolutional layers and ROI-selection pooling layers for predicting neurological biomarkers. BrainNetCNN (Kawahara et al., 2017) designs a CNN that includes edge-to-edge, edge-to-node, and node-to-graph convolutional filters, leveraging the topological locality of brain connectome structures. BrainNetTF (Kan et al., 2022) introduces a transformer architecture with an orthonormal clustering readout function that considers ROI similarity within functional modules. Additionally, various studies (Cui et al., 2022; Kan et al., 2022; Zhu et al., 2022; Cui et al., 2022; Yu et al., 2023) have shown that, when data is sufficient, GNNs can greatly improve performance in tasks such as disease prediction. However, in reality, the lack of training data is a common issue in neuroscience research, particularly for specific domains and clinical tasks (Xu et al., 2023). Despite this, there has been little research into the ability of GNNs to effectively train for brain network analysis when data is limited.

Unsupervised Graph Representation Learning and GNN Pre-training.Unsupervised learning is a widely used technique for training complex models when resources are limited. Recent advancements in contrastive learning (Chen et al., 2020; He et al., 2020; Yu et al., 2021; Zhu et al., 2022) have led to various techniques for graphs. For instance, GBT (Bielak et al., 2022) designs a Barlow Twins Zbontar et al. (2021) loss function based on the empirical cross-correlation of node representations learned from two different views of the graph (Zhao et al., 2021). Similarly, GraphCL (You et al., 2020) involves a comparison of graph-level representations obtained from two different augmentations of the same graph. DGI (Velickovic et al., 2019) contrasts graph and node representations learned from the original graph and its corruption.

To obtain strong models for particular downstream tasks, unsupervised training techniques can be used to pre-train a model, which is then fined tuned on the downstream tasks to reduce the dependence on labeled training data. The approach has proven highly successful in computer vision (Cao et al., 2020; Grill et al., 2020), natural language processing (Devlin et al., 2019; Radford et al., 2018, 2021; Liang et al., 2020), and multi-modality (e.g. text-image pair) learning (Li et al., 2022; Yao et al., 2022). There are various strategies for pre-training GNNs as well. GPT-GNN (Hu et al., 2020) proposes graph-oriented pretext tasks, such as masked attribute and edge reconstruction. L2P-GNN (Lu et al., 2021) introduces dual adaptation by simultaneously optimizing the encoder on a node-level link prediction objective and a graph-level self-supervision task similar to DGI. Others, such as GMPT (Hou et al., 2022) adopt an inter-graph message-passing approach to obtain context-aware node embedding and optimize the model concurrently under supervision and self-supervision. To the best of our knowledge, the effectiveness of both contrastive learning and pre-training has not been investigated in the context of the unique properties of brain networks.

## 3 Unsupervised Brain Network Pre-training

Problem Definition.The available training resource includes a collection of brain network datasets \(\mathcal{S}=\{\mathcal{D}_{1},\mathcal{D}_{2},\cdots\mathcal{D}_{s}\}\), where each dataset contains a varying number of brain networks. We consider each brain network instance with \(M\) number of defined ROIs as an undirected weighted graph \(\mathcal{G}\) with \(M\) nodes. \(\mathcal{G}\) is represented by a node-set \(\mathcal{V}=\{v_{m}\}_{m=1}^{M}\), an edge set \(\mathcal{E}=\mathcal{V}\times\mathcal{V}\), and a weighted adjacency matrix \(\mathbf{A}\in\mathbb{R}^{M\times M}\). We define a \(\theta\) parameterized GNN model \(f(\cdot)\), and our goal is to propose a pre-training schema that can effectively learn an initialization \(\theta_{0}\) for \(f(\cdot)\) on a set of source datasets \(\mathcal{S}_{\text{source}}\subset\mathcal{S}\) via self-supervision and adapt \(f_{\theta_{0}}(\cdot)\) to a local optimum \(\theta^{*}\) on a target set \(\mathcal{S}_{\text{target}}\in\mathcal{S}\).

### GNN Pre-training for Brain Networks

The goal of pre-training a GNN model for brain networks is to learn an appropriate initialization that can easily be adapted to downstream task. Note that the concept of pre-training is distinct from transfer learning since the latter expects a similarity between the source and target data as well as their learning objectives (_e.g.,_ loss functions), while this is often lacking in brain network analysis due to absence of
sufficient ground truth labels in large scale studies as well as inherent differences in their brain network parcellation methods across varying datasets. Practically, a GNN model can be pre-trained either on a singular task with a single source dataset or on a collection of tasks with multiple source datasets. The proposed PTGB framework adopts the latter option since multi-task pre-training reduces the likelihood of the model being biased towards the knowledge of data from a singular source, which could be particularly concerning if the source and target data shares limited similarity leading to poor downstream adaptation due to information loss during model transfer. However, a naive approach towards multi-task pre-training would not suffice in learning a robust model initialization. Specifically, it presents two underlying risks: (1) the model may not perform consistently well on all tasks and may also overfit to a particular task which significantly undermines model generalizability; and (2) the process could be computationally inefficient with increasing number of tasks regardless if the model is optimized sequentially or simultaneously on all tasks (Yang et al., 2022).

To this end, we adopt the popular data-efficient training techniques presented in MAML (Finn et al., 2017) with the goal of ensuring consistent performance on all tasks as well as computational efficiency. The MAML technique is characterized by an inner-loop adaptation and an outer-loop update (Raghu et al., 2019). At each training iteration, each input dataset is partitioned into an inner-loop support set and an outer-loop query set. The model is first trained on the support set without explicitly updating the parameters. Instead, the updates are temporarily stored as fast weights (Ba et al., 2016). These fast weights are then used to evaluate the query set and compute the actual gradients. This approach makes use of approximating higher-order derivatives (Tan and Lim, 2019) at each step, allowing the model to foresee its optimization trajectory a few steps ahead, which practically reduces the number of required training iterations to reach local optima. In our scenario, the joint optimization involves summing the loss over each brain network dataset, i.e., for \(n\) number of datasets and their respective temporary fast weights \(\{\theta_{i}^{\prime}\}_{i=1}^{n}\) and outer-loop queries \(\{\text{query}_{i}\}_{i=1}^{n}\), the step-wise update of the model parameter at time \(t\) is \(\theta^{t+1}=\theta^{t}-\alpha\nabla_{\theta^{t}}\sum_{i=1}^{n}\mathcal{L}_{ \text{query}_{i}}f_{\theta_{i}^{\prime}}(\cdot)\). We hereby summarize this process in Algorithm 1. In addition, we will also demonstrate the advantages of MAML-styled pre-training over vanilla multi-task pre-training as well as single task pre-training through experiments which will be discussed in Section 4.1.

### Brain Network Oriented Two-Level Contrastive Learning

Given the high cost of acquiring labeled training data for brain network analysis, our pre-training pipeline

Figure 1: Overview of the proposed framework PTGB. The initial features of the source datasets are projected to a fixed dimension through atlas transformation followed by variance-based feature alignment, which facilitates self-supervised GNN pre-training on multiple datasets via the novel two-level contrastive learning objective. The learned model can serve as the parameter initialization and be further fine-tuned on target tasks.
of PTGB adopts to the effective label-free learning strategy of contrastive learning (CL). CL aims to maximize the mutual information (MI) between an anchor point of investigation \(X\) from a data distribution \(\mathcal{H}\) and its positive samples \(X^{+}\), while minimizing MI with its negative samples \(X^{-}\). The contrastive objective function is formulated as follows:

\[\mathcal{J}_{\text{con}}=\arg\min\left[\left(-I(X;X^{+})+I(X;X^{-})\right) \right]. \tag{1}\]

In the context of graph CL, given an anchor node representation \(z_{\alpha}\), a set of positive samples \(\mathbf{S}^{+}\), and a set of negative samples \(\mathbf{S}^{-}\), the training objective is based on the Jensen-Shannon divergence (Hjelm et al., 2019),

\[\mathcal{J}_{\text{JSD}}(z_{\alpha})=\arg\min\left[\left(-I(z_{\alpha}; \mathbf{S}^{+})+I(z_{\alpha};\mathbf{S}^{-})\right)\right], \tag{2}\]

where

\[I(z_{\alpha};\mathbf{S}^{+}) =\frac{1}{|\mathbf{S}^{+}|}\sum_{z_{s^{+}}\in\mathbf{S}^{+}}\text {sp}\left(\frac{z_{\alpha}^{\top}z_{s^{+}}}{\|z_{\alpha}\|\|z_{s^{+}}\|}\right), \tag{3}\] \[I(z_{\alpha};\mathbf{S}^{-}) =\frac{1}{|\mathbf{S}^{-}|}\sum_{z_{s^{-}}\in\mathbf{S}^{-}}\text {sp}\left(\frac{z_{\alpha}^{\top}z_{s^{-}}}{\|z_{\alpha}\|\|z_{s^{-}}\|}\right), \tag{4}\]

and \(\text{sp}(\cdot)=\log(1+e^{\cdot})\) is softplus nonlinearity.

The ultimate goal of our framework is to localize effective GNN CL learning (Zhu et al., 2021) for brain networks. Given a dataset \(\mathcal{D}\) and an anchor node \(i\) from graph \(\mathcal{G}_{p}\in\mathcal{D}\) with the learned representation \(z_{i,p}\), we propose to categorize the possible sample selections into three fundamental types (a visualization is shown in Figure 2):

* \(\mathbf{\underline{S_{1}}}\): \(\{z_{j,p}\,:\,j\in\mathcal{N}_{k}(i,p)\}\) refers to the node representation set within the the \(k\)-hop neighborhood of the anchor in graph \(\mathcal{G}_{p}\).
* \(\mathbf{\underline{S_{2}}}\): \(\{z_{j,p}\,:\,j\notin\mathcal{N}_{k}(i,p)\}\) refers to the remaining node representation set in graph \(\mathcal{G}_{p}\) that are not in the the \(k\)-hop neighborhood of the anchor.
* \(\mathbf{\underline{S_{3}}}\): \(\{z_{j,q}\,:\,\mathcal{G}_{q}\in\mathcal{D},\,j\in\mathcal{G}_{q},\,q\neq p\}\) refers to the node representation set of nodes in all the other graphs of dataset \(\mathcal{D}\).

Notice that our framework leverages the \(k\)-hop substructure around the anchor node to further differentiate \(\mathbf{S_{1}}\) and \(\mathbf{S_{2}}\) for contrastive optimization. This design is driven by two considerations: **(1) Regarding GNN learning.** Given that node representations are learned from the information aggregation of its \(k\)-hop neighborhood, maximizing the MI of an anchor to its \(k\)-hop neighbors naturally enhances lossless message passing of GNN convolutions. **(2) Regarding the uniqueness of brain networks.** Brain networks can be anatomically segmented into smaller neural system modules (Cui et al., 2022), thus capturing subgraph-level knowledge can provide valuable signals for brain-related analysis.

Building on these three fundamental types of samples, we take advantage of the property of brain networks that ROI identities and orders are fixed across samples to introduce an additional sample type. This encourages the GNN to extract shared substructure knowledge by evaluating the MI of an anchor against its presence in other graphs. Given an anchor representation \(z_{i,p}\) of node \(i\) from graph \(\mathcal{G}_{p}\in\mathcal{D}\), the novel inter-graph sample type is defined as:

Figure 2: Visual demonstration of the sample types where \(X_{i,p}\) is the anchor and \(\mathbf{S_{1}}/\mathbf{S_{4}}\) are sampled as 1-hop neighbors.
* \(\mathbf{S_{4}}\):\(\{z_{j,q}\,:\,j\in\mathcal{N}_{k}(i,q)\cap\mathcal{N}_{k}(i,p),\,\mathcal{G}_{q} \in\mathcal{D},\,q\neq p\}\), refers to the node representation set within the \(k\)-hop neighborhood of node \(i\) in all other graphs in \(\mathcal{D}\). Conceptually, \(\mathbf{S_{4}}\) is a special subset of \(\mathbf{S_{3}}\).

It is important to note that for an anchor node \(i\), its \(k\)-hop neighborhood structures might not be identical among different graphs. As a result, we only consider shared neighborhoods when evaluating the mutual information across multiple graphs. To encourage the learning of unique neighborhood knowledge within a single brain network instance and shared substructure knowledge across the entire dataset, we configure \(\mathbf{S_{1}}\) and \(\mathbf{S_{4}}\) as positive samples while \(\mathbf{S_{2}}\) and the set \(\mathbf{S_{3}}-\mathbf{S_{4}}\) as negative samples, as illustrated in Figure 3. Strictly speaking, \(\mathbf{S_{1}}\) does not include the anchor itself, but the anchor is always a positive sample to itself by default. Furthermore, our sampling categorization can also help understand the objective formulations in various state-of-the-art graph CL frameworks (Velickovic et al., 2019; Qiu et al., 2020; Xia et al., 2022; Sun et al., 2019; Zhu et al., 2021). We summarize our findings in Table 1. Specifically, "+" denotes positive sampling; "-" denotes negative sampling; and "/" means that the sample type is not considered. It can be observed that DGI and InfoGraph (InfoG) use graph representation pooled from node representations as a special sample, which is essentially equivalent to jointly considering \(\mathbf{S_{1}}\) and \(\mathbf{S_{2}}\) without explicit differentiation. On the other hand, GCC and EGI, which are more closely related to our framework, leverage neighborhood mutual information maximization on a single graph, but fail to extend this to a multi-graph setting like ours.

### Data-driven Brain Atlas Mapping

MotivationWhen fine-tuning a pre-trained model on a new data domain, the misalignment between source and target signals can negatively impact its adaptation. This issue is particularly relevant in brain networks, where it is hard, if not impossible, to require every brain network data provider to stick to the same brain atlas template, and each template can use a unique system of ROIs. For instance, the HIV dataset we obtained is parcellated from the AAL90 template (Tzourio-Mazoyer et al., 2002), leading to 90 defined ROIs; while the PPMI dataset uses the Desikan-Killiany84 template (Desikan et al., 2006), resulting in 84 defined ROIs. As a result, brain networks in the two datasets will have different ROI semantics and graph structures. Although GNNs can handle graphs without fixed numbers and orders of nodes, constructing the most informative ROI (_i.e.,_ node) features as the connection profiles (_i.e.,_ adjacency) (Cui et al., 2022, 2022) can result in different feature dimensions and physical meanings. While manual conversion can be performed to translate between templates, it is a costly process that requires domain expertise to perform even coarse cross-atlas mappings.

To address this issue, we aim to provide a data-driven atlas mapping solution that is easily accessible and eliminates the strong dependency on network construction. The data-driven atlas mapping solution, which transforms the original node features into lower-dimensional representations that preserve the original connectivity information and align features across datasets, is learned independently on each dataset prior to GNN pre-training.

#### 3.3.1 Autoencoder with Brain Network Oriented Regularizers

PTGB adopts a one-layer linear autoencoder (AE) as the base structure. The AE consists of a linear projection encoder \(\mathbf{W}\) and a transposed decoder \(\mathbf{W}^{\top}\)

\begin{table}
\begin{tabular}{|l|c|c|c|c|} \hline  & \(\mathbf{S_{1}}\) & \(\mathbf{S_{2}}\) & \(\mathbf{S_{3}}\) & \(\mathbf{S_{4}}\) \\ \hline DGI & + & + & / & / \\ \hline InfoG & + & + & – & / \\ \hline GCC & + & – & – & / \\ \hline EGI & + & – & – & / \\ \hline Ours & + & – & – & + \\ \hline \end{tabular}
\end{table}
Table 1: The sampling configuration of some existing graph contrastive learning methods. “+” denotes positive sampling, “-” for negative, and “/” for no consideration.

Figure 3: The sampling configuration of the proposed PTGB framework. \(\mathbf{S_{1}}\) and \(\mathbf{S_{4}}\) are positive samples, \(\mathbf{S_{2}}\) and the set \(\mathbf{S_{3}}-\mathbf{S_{4}}\) are negative samples.
with the goal of learning a low-dimensional projection that can easily reconstruct the original presentation. The loss function is defined as minimizing the reconstruction error \(\mathcal{L}_{\text{rec}}=(1/M)\|\mathbf{X}-\mathbf{X}\mathbf{W}\mathbf{W}^{\top} \|_{2}^{2}\), where \(\mathbf{X}\in\mathbb{R}^{M\times M}\) is the input and \(\mathbf{W}\in\mathbb{R}^{M\times D}\) is the learnable projection (Hinton and Zemel, 1993). To further enhance the feature compression and to guide the overall AE optimization, we propose to incorporate several regularizers that take into account the unique characteristics of brain networks.:

**Locality-Preserving Regularizer (LR).** We aim to ensure that the compressed features preserve the spatial relationships of the original brain surface. To achieve this, we incorporate a locality preserving regularizer (He et al., 2005) to the AE objective. The regularizer is formulated as \(\mathcal{L}_{\text{loc}}=(1/M)\|\mathbf{Y}-\mathbf{T}\mathbf{Y}\|^{2}\), where \(\mathbf{Y}\in\mathbb{R}^{M\times D}\) represents the projected features from the AE and \(\mathbf{T}\in\mathbb{R}^{M\times M}\) is a transition matrix constructed from the \(k\)-NN graph of the 3D coordinates of ROIs.

**Modularity-Aware Regularizer (CR).** Brain networks can be segmented into various neural system modules that characterize functional subsets of ROIs. In graph terminology, they are community structures. The projected feature should also capture information about neural system membership. However, obtaining ground-truth segmentations is a difficult task that requires expert knowledge. To overcome this challenge, we resort to community detection methods on graphs, specifically based on modularity maximization. The regularizer (Salha-Galvan et al., 2022) is defined as minimizing

\[\mathcal{L}_{\text{com}}=-\frac{1}{2D}\sum_{i,j=1}^{M}\left[\mathbf{A}_{ij}- \frac{k_{i}k_{j}}{2D}\right]\exp(-\|y_{i}-y_{j}\|_{2}^{2}), \tag{5}\]

where \(\mathbf{A}\in\mathbb{R}^{M\times M}\) is the graph adjacency matrix, \(k_{i}\) denotes degree of node \(i\), and \(y_{i}\) is the AE projected features. Essentially, this optimization minimizes the \(L_{2}\) distance between representations of nodes within the same communities, as measured by the modularity score, and maximizes the distance between representations of nodes in different communities.

**Sparsity-Oriented Regularizer (SC).** Sparse networks have proven to be effective in learning robust representations from noisy data (Jeong et al., 2017; Shi et al., 2019; Makhzani and Frey, 2014). In brain connectome analysis, sparsity has also been shown to improve the interpretation of task-specific ROI connections in generation and classification tasks (Kan et al., 2022). To this end, we implement the popular KL-divergence smoothing to enforce sparsity in the parameters of the linear projection encoder, \(\mathbf{W}\)). This is formulated as:

\[\mathcal{L}_{\text{KL}}=\sum_{i=1}^{M}\sum_{j=1}^{D}\left[\rho\log\left(\frac {\rho}{\hat{\rho}_{ij}}\right)+(1-\rho)\log\left(\frac{1-\rho}{1-\hat{\rho}_ {ij}}\right)\right], \tag{6}\]

where \(\rho\) is a small positive float set as the target sparsity value, and \(\hat{\rho}_{ij}\) represents the element-wise activation of the encoder projection matrix \(\mathbf{W}\in\mathbb{R}^{M\times D}\).

#### 3.3.2 Variance-based Dimension Sorting

In addition to transforming dataset-specific features, cross-dataset alignment of feature signals is also crucial for improving model adaptation. The one-layer AE transforms the original feature vectors into weighted combinations of multiple dimensions, creating new feature dimensions which we name as _virtual ROIs_. In the context of brain networks, this process helps to group ROIs and their signals. This idea is inspired by the well-studied functional brain modules (Philipson, 2002; Anderson et al., 2004; Hilger et al., 2020; Brodmann, 1999; Zhou et al., 2020), which provide a higher-level and generic organization of the brain surface, as opposed to fine-grained ROI systems. Since the variations in ROI parcellations are due to differences in clinical conventions, it is reasonable to assume that there exists a shared virtual ROI system underlying different parcellation systems, similar to the discretization of functional brain modules. The community learning and neighborhood preserving regularizers, introduced in Section 3.3, allow us to capture these shared virtual ROIs in a data-driven manner. Our ultimate goal is to align the discovered virtual ROIs across datasets, so that each virtual ROI characterizes the same functional module in the human brain, regardless of its origin. This cross-dataset alignment of virtual ROIs ensures that the model can effectively adapt to new datasets and provide meaningful insights into the different downstream analyses.

The objective of the one-layer linear AE is similar to PCA, as discussed in more detail in Appendix A.1, with the added benefit of incorporating additional regularizers. PCA orders dimensions based on decreasing levels of sample variance (Hotelling, 1933). PTGB leverage this approach by utilizing the learned parameters of the AE projection to estimate the variance of each virtual ROI (_i.e._, projected feature di
mension). The sample variance of each virtual ROI indicates its representativeness of the original data variations. Given the shared patterns across different parcellation systems, we expect that similar virtual ROIs in datasets with different atlas templates will have similar variance scores, especially in terms of their order. By sorting the same number of virtual ROIs based on their sample variance in each dataset, we aim to align virtual ROI cross datasets, so that each virtual ROI represents the same functional unit in the human brain. The procedure is explained in detail in Algorithm 2 in Appendix A.2.

## 4 Experiments

We evaluate the effectiveness of PTGB through extensive experiments on real brain network datasets, with a focus on the following research questions:

* **RQ1**: How does PTGB compare with other unsupervised GNN pre-training frameworks adapted to the scenario of brain networks?
* **RQ2**: What is the contribution of each major component in PTGB to the overall performance?
* **RQ3**: How does the choice of sampling method affect model convergence and performance?
* **RQ4**: How effective is the variance-based sorting in aligning virtual ROIs among different parcellation systems?

Datasets, Configurations, and Metrics.Our experiments are conducted on three real-world brain network datasets: PPMI, BP, and HIV. The PPMI dataset is parcellated using the Desikan-Killiany84 atlas template and includes brain networks from 718 subjects, 569 of whom are Parkinson's Disease (PD) patients and 149 are Healthy Control (HC). The networks are constructed using three tractography algorithms: Probabilistic Index of Connectivity (PICo), Hough voting (Hough), and FSL. The BP dataset is parcellated using the Brodmann82 template and includes resting-state fMRI and DTI modalities from 97 subjects, 52 of whom have Bipolar I disorder and 45 are HCs. The HIV dataset is parcellated using the AAL90 template and includes fMRI and DTI modalities from 70 subjects, with 35 early HIV patients and 35 HCs. We pre-train the model on the PPMI dataset and evaluate the downstream performance on BP and HIV. Further details about the datasets can be found in Appendix B.

PTGB employs GCN as the backbone for the GNN (Kipf and Welling, 2017) encoder. We also benchmark PTGB with GAT (Velickovic et al., 2018) and GIN (Xu et al., 2019), and the results are provided in Appendix D.1. The hyperparameter settings are described in detail in Appendix C. The hyperparameter tuning follows the standard designs in related studies such as in (Yang et al., 2021; Wein et al., 2021; Hu et al., 2021). The downstream evaluation is binary graph classification for disease prediction. To assess the performance, we use the two widely used metrics in the medical field (Li et al., 2021; Cui et al., 2022): accuracy score (ACC) and the area under the receiver operating characteristic curve (AUC).

### Overall Performance Comparison (RQ1)

We present a comprehensive comparison of the target performance between the proposed PTGB and popular unsupervised learning strategies in Table 2. To fairly compare the methods, we apply atlas mapping pre-processing and the multi-dataset learning backbone discussed in section 3.1 to all methods. The purpose of this comparison is to effectively highlight the impact of the proposed two-level contrastive pre-training and we will further analyze the effect of atlas mapping in subsequent subsections. In addition, for a clearer presentation, we group the selected baselines according to their optimization strategies:

* No pre-training (NPT): the backbone with randomly initialized parameters for target evaluation.
* Non-CL-based (NCL): methods with cost functions regularized by co-occurrence agreement or link reconstruction, including Node2Vec (Grover and Leskovec, 2016), DeepWalk (Perozzi et al., 2014), and VGAE (Kipf and Welling, 2016).
* Single-scale CL (SCL): methods utilizing either node- or graph-level representations in the CL optimization, including GBT (Bielak et al., 2022), ProGCL (Xia et al., 2022), and GraphCL (You et al., 2020).
* Multi-scale CL (MCL): methods whose CL optimization utilizes both nodes- and graph-level representations, including DGI (Velickovic et al., 2019) and InfoG (Sun et al., 2019).
* Ego-graph sampling (EGS): methods whose contrastive samplings consider \(k\)-hop ego-networks as discriminative instances, which are the most similar to the proposed PTGB, including GCC (Qiu et al., 2020) and EGI (Zhu et al., 2021).
* Our proposed two-level contrastive optimization (Ours): methods include single task pre-training (STP) in which we select the PICo modality of
the PPMI study to be the only source task; multi-task pre-trainig (MTP) which does not utilize the MAML technique; and the full implementation of the PTGB framework. The experiments reveal the following insights:

* The proposed PTGB consistently outperforms all the baselines, achieving a relative improvement of 7.34%-13.30% over the best-performing baselines and 31.80%-38.26% over the NPT setting. The results of PTGB have been statistically compared against baselines using paired \(t\)-tests. With a significance level set to 0.05, the largest two-tailed \(p\) value is reported at 0.042, indicating that PTGB demonstrates a statistically significant performance increase over other selected methods.
* Compared with the transductive methods of Node2Vec and DeepWalk, the GNN pre-trained by VGAE learns structure-preserving representations and achieves the best results in the NCL-type methods. This indicates the potential benefit of the locality-preserving regularizer design in PTGB.
* Maximizing mutual information between augmented instances may hinder GNNs from learning a shared understanding of the entire dataset. For baselines belonging to the categories of SCL, MCL, and EGS, pre-training with non-augmented CL (InfoG, EGI) generally results in a 4.36% relative improvement across both metrics and a 7.63% relative decrease in performance variance compared to their augmentation-based counterparts (GBT, GraphCL, ProGCL, DGI, GCC). This explains why PTGB does not employ data augmentation.
* Multi-scale MI promotes the capture of effective local (_i.e.,_ node-level) representations that can summarize the global (_i.e.,_ graph-level) information of the entire network. The MCL-type methods typically outperform the SCL-type ones by a relative gain of 2.68% in ACC and 3.27% in AUC.
* The group of baselines considering \(k\)-hop neighborhoods (EGS) presents the strongest performance, indicating the importance of local neighborhoods in brain network analysis. The proposed PTGB, which captures this aspect through both node- and graph-level CL, is the only one that comprehensively captures the local neighborhoods of nodes.
* Learning from multiple tasks (MTP) brings significant improvement over STP, reporting a relative increase of 8.47% in accuracy and 6.90% in AUC. Furthermore, the full PTGB framework with MAML-styled training achieves a relative improvement of 11.29% in accuracy, 14.75% in AUC, and a reduced variance over MTP, demonstrating its advantages in enhancing model generalizability.

### Ablation Studies (RQ2)

We examine two key components of PTGB- (1) the two-level contrastive sampling and (2) the atlas mapping regularizers. The best contrastive sampling configuration is fixed when examining the atlas regularizers, and all regularizers are equipped when examin

\begin{table}
\begin{tabular}{l l|c c c c c c c} \hline \hline \multirow{2}{*}{Type} & \multirow{2}{*}{Method} & \multicolumn{2}{c}{BP-fMRI} & \multicolumn{2}{c}{BP-DTI} & \multicolumn{2}{c}{HIV-fMRI} & \multicolumn{2}{c}{HIV-DTI} \\ \cline{3-10}  & & ACC & AUC & ACC & AUC & ACC & AUC & ACC & AUC \\ \hline NPT & GCN & 50.07\(\pm\)0.70 & 50.11\(\pm\)5.80 & 49.51\(\pm\)0.68 & 51.83\(\pm\)0.80 & 56.27\(\pm\)1.84 & 57.16\(\pm\)5.14 & 51.30\(\pm\)0.42 & 53.82\(\pm\)1.94 \\ \hline \multirow{3}{*}{NCL} & Node2Vec & 48.51\(\pm\)0.30 & 49.68\(\pm\)7.23 & 50.83\(\pm\)1.44 & 46.70\(\pm\)10.30 & 52.61\(\pm\)10.38 & 50.75\(\pm\)10.94 & 49.65\(\pm\)0.30 & 51.22\(\pm\)10.79 \\  & DeepWalk & 50.28\(\pm\)0.33 & 51.59\(\pm\)0.60 & 51.72\(\pm\)5.74 & 38.46\(\pm\)9.37 & 54.81\(\pm\)11.20 & 55.55\(\pm\)11.93 & 52.67\(\pm\)11.20 & 50.88\(\pm\)10.39 \\  & VGAE & 56.71\(\pm\)1.48 & 55.24\(\pm\)11.40 & 54.63\(\pm\)11.20 & 54.11\(\pm\)11.82 & 62.76\(\pm\)4.77 & 61.25\(\pm\)11.54 & 56.90\(\pm\)0.42 & 55.35\(\pm\)0.44 \\ \hline \multirow{3}{*}{SCL} & GBT & 57.21\(\pm\)0.68 & 57.32\(\pm\)10.00 & 56.29\(\pm\)0.53 & 55.27\(\pm\)10.54 & 65.73\(\pm\)10.00 & 66.08\(\pm\)10.63 & 59.80\(\pm\)7.76 & 57.37\(\pm\)0.40 \\  & GraphCL & 59.79\(\pm\)30 & 59.10\(\pm\)0.78 & 57.57\(\pm\)10.63 & 57.35\(\pm\)0.67 & 67.08\(\pm\)7.76 & 69.17\(\pm\)6.68 & 60.43\(\pm\)3.90 & 60.03\(\pm\)10.48 \\  & ProGCL & 62.36\(\pm\)0.50 & 62.61\(\pm\)0.34 & 61.26\(\pm\)37 & 62.67\(\pm\)8.46 & 71.52\(\pm\)10.39 & 72.16\(\pm\)8.55 & 62.48\(\pm\)10.38 & 61.94\(\pm\)10.37 \\ \hline \multirow{3}{*}{MCL} & DGI & 62.44\(\pm\)10.13 & 60.75\(\pm\)10.97 & 58.15\(\pm\)0.60 & 58.95\(\pm\)0.60 & 70.22\(\pm\)11.40 & 70.12\(\pm\)12.16 & 60.83\(\pm\)10.48 & 62.06\(\pm\)10.16 \\  & InfoG & 62.87\(\pm\)0.52 & 62.37\(\pm\)0.67 & 60.88\(\pm\)0.67 & 60.44\(\pm\)0.61 & 72.46\(\pm\)8.71 & 72.94\(\pm\)5.66 & 61.75\(\pm\)76 & 61.37\(\pm\)0.45 \\ \hline \multirow{3}{*}{EGS} & GCC & 63.45\(\pm\)0.62 & 62.39\(\pm\)0.60 & 60.44\(\pm\)0.54 & 60.29\(\pm\)0.10 & 70.97\(\pm\)0.13 & 72.48\(\pm\)10.13 & 61.27\(\pm\)10.66 & 61.38\(\pm\)10.79 \\  & EGI & 63.38\(\pm\)0.63 & 63.58\(\pm\)0.62 & 61.82\(\pm\)0.63 & 61.57\(\pm\)8.27 & 37.46\(\pm\)0.42 & 32.85\(\pm\)0.48 & 60.98\(\pm\)0.42 & 62.41\(\pm\)10.50 \\ \hline \multirow{3}{*}{Ours} & STP & 53.92\(\pm\)12.2\(\pm\)27 & 54.61\(\pm\)11.28 & 55.51\(\pm\)11.28 & 56.73\(\pm\)10.20 & 61.18\(\pm\)14.57 & 62.88\(\pm\)11.55 & 55.29\(\pm\)12.38 & 57.31\(\pm\)11.27 \\  & MTP & 60.37\(\pm\)11.47 & 61.44\(\pm\)11.28 & 59.41\(\pm\)11.26 & 59.92\(\pm\)13.37 & 67.65\(\pm\)12.30 & 68.38\(\pm\)12.36 & 60.54\(\pm\)13.37 & 59.46\(\pm\)12.39 \\  & PTGB & **68.84\(\pm\)0.84** & **68.45\(\pm\)0.86** & **66.57\(\pm\)0.87** & **68.31\(\pm\)0.88** & **77.80\(\pm\)0.98** & **77.22\(\pm\)0.74** & **67.51\(\pm\)0.87** & **67.74\(\pm\)0.88** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Disease prediction performance comparison. All results are averaged from 5-fold cross-validation along with standard deviations. The best result is highlighted in bold and runner-up is underlined. * denotes a significant improvement according to paired \(t\)-test with \(\alpha=0.05\) compared with baselines.
ing the contrastive samplings. The results, shown in Figure 4 (with additional DTI version in Appendix D.2), are analyzed based on the four possible variants of contrastive sampling listed in Table 3. Our analyses yield the following observations: **(1)** leveraging \(k\)-hop neighborhood (_i.e.,_ positive **S2**) MI maximization brings visible performance gain, confirming its benefit in brain structure learning; **(2)** The extension to multi-graph CL (_i.e.,_ consideration of **S3**) facilitates the extraction of unique ROI knowledge, leading to improved results in Var. 3/4; **(3)** Var. 4 outperforms Var. 3 as it effectively summarizes of global (_i.e.,_ graph-level) information in local node representations; **(4)** The full implementation of PTGB brings a relative gain of 4.27% in both metrics on top of Var. 4, highlighting the significance of considering shared substructure knowledge across multiple graphs (_i.e.,_ through the inclusion of **S4**).

The right-side sub-figures examine the impact of the atlas mapping regularizers by comparing the results of the full framework to those without the sparsity regularizer (w/o SR), the locality regularizer (w/o LR), and the community regularizer (w/o CR). Two key observations are made: **(1)** The removal of SR leads to the greatest performance drop, emphasizing its crucial role in learning robust projections that can effectively handle noise and prevent over-fitting; **(2)** The inferior results when LR and CR are absent emphasize the importance of spatial sensitivity and blockwise feature information in brain network analysis. This supports our intuition to consider the relative positioning of ROIs in the 3D coordinate as

Figure 4: Ablation comparisons on contrastive sampling choices (left two) and atlas mapping regularizers (right two). The \(y\)-axis refers to the numeric values of evaluated metrics (in %). The setup of Var. 1 - 4 is described in Table 3. “SC”, “LR”, and “CR” are abbreviations for “sparsity constraints”, “locality regularizer”, and “community (modularity-aware) regularizer” respectively.

Figure 5: In-depth comparison among the four variants and the full model. The \(x\)-axis is epochs. Fig. (a) evaluates the trajectory of pre-training loss, Fig. (b) evaluates their respective testing accuracy on the fMRI view of the HIV dataset, and Fig. (c) reports the pre-training runtime in seconds.

\begin{table}
\begin{tabular}{|l|l|l|l|l|} \hline  & \multicolumn{1}{|c|}{**S1**} & \multicolumn{1}{|c|}{**S2**} & \multicolumn{1}{|c|}{**S3**} & \multicolumn{1}{|c|}{**S4**} \\ \hline Var. 1 & – & – & / & / \\ \hline Var. 2 & + & – & / & / \\ \hline Var. 3 & + & – & – & / \\ \hline Var. 4 & + & + & – & / \\ \hline \end{tabular}
\end{table}
Table 3: The four variants of sampling strategies.
well as knowledge on community belongings based on modularity measures.

### Analysis of Two-level Contrastive Sampling (RQ3)

Figure 5 offers insight into the pre-training convergence, target adaptation progression, and pre-training runtime consumption of the four sampling variants and the full framework. Key observations include: **(1)** As seen in Figure 5(a), all variants demonstrate efficient pre-training convergence due to the multi-dataset joint optimization inspired by MAML. The full model demonstrates the most optimal convergence, highlighting the advantage of learning shared neighborhood information in brain network data through two-level node contrastive sampling. **(2)** Figure 5(b) shows the superiority of our design in terms of downstream adaptation performance compared to other variants. **(3)** Figure 5(c) reveals that the more sophisticated the sampling considerations result in greater computational complexity for mutual information evaluation, leading to longer runtime for each pre-training epoch. However, the total time consumptions are all on the same scale.

### Analysis of ROI Alignment (RQ4)

To further validate the variance-based virtual ROI sorting, we select the top 2 virtual ROIs with the highest sample variances for each atlas template (_i.e.,_ dataset) and backtrack to locate their corresponding projected ROIs. The results are illustrated in Figure 6, which shows a 3D brain surface visualization highlighting the original ROIs. From this, we draw two main conclusions: **(1)** There exists multiple regional overlaps between pairs of two atlas templates, reflecting some working effectiveness of our proposed solution as well as confirming the feasibility of converting between atlas templates. **(2)** It is relatively harder to find regions that overlap across all three atlas templates which shows a limitation of the proposed unsupervised ROI alignment scheme, suggesting a need to modify against the current variance-based heuristic which may inspire further study and research opportunity.

## 5 Conclusion

Brain network analysis for task-specific disease prediction has been a challenging task for conventional GNN frameworks due to the limited availability of labeled training data and the absence of a unifying brain atlas definition, which hinders efficient knowledge transfer across different datasets. To address these challenges, we propose PTGB, a novel unsupervised multi-dataset GNN pre-training that leverages a two-level node contrastive sampling to overcome data scarcity. Additionally, PTGB incorporates atlas mapping through brain-network-oriented regularizers and variance-based sorting to address the issue of incompatible ROI parcellation systems in cross-dataset model adaptation in a data-driven way. Extensive experiments on real-world brain connectome datasets demonstrate the superiority and robustness of PTGB in disease prediction and its clear advantage over various state-of-the-art baselines. As more brain network datasets become available, it will be intriguing to further validate its generalizability.

Figure 6: The virtual ROI mapping across the three investigated datasets. We highlight pairs of overlapping regions with colored boxes. In particular, we use gold boxes for the PPMI and BP mapping; blue boxes for the BP and HIV mapping; and purple boxes for the PPMI and HIV mapping.