Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level
  Tasks
Recently, graph neural networks (GNNs) have shown its unprecedented success
in many graph-related tasks. However, GNNs face the label scarcity issue as
other neural networks do. Thus, recent efforts try to pre-train GNNs on a
large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to
the target downstream task. The adaptation is generally achieved by fine-tuning
the pre-trained GNNs with a limited number of labeled data. Despite the
importance of fine-tuning, current GNNs pre-training works often ignore
designing a good fine-tuning strategy to better leverage transferred knowledge
and improve the performance on downstream tasks. Only few works start to
investigate a better fine-tuning strategy for pre-trained GNNs. But their
designs either have strong assumptions or overlook the data-aware issue for
various downstream datasets. Therefore, we aim to design a better fine-tuning
strategy for pre-trained GNNs to improve the model performance in this paper.
Given a pre-trained GNN, we propose to search to fine-tune pre-trained graph
neural networks for graph-level tasks (S2PGNN), which adaptively design a
suitable fine-tuning framework for the given labeled data on the downstream
task. To ensure the improvement brought by searching fine-tuning strategy, we
carefully summarize a proper search space of fine-tuning framework that is
suitable for GNNs. The empirical studies show that S2PGNN can be implemented on
the top of 10 famous pre-trained GNNs and consistently improve their
performance. Besides, S2PGNN achieves better performance than existing
fine-tuning strategies within and outside the GNN area. Our code is publicly
available at \url{https://anonymous.4open.science/r/code_icde2024-A9CB/}.
Zhili Wang, Shimin Di, Lei Chen, Xiaofang Zhou
# Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks

###### Abstract

Recently, graph neural networks (GNNs) have shown its unprecedented success in many graph-related tasks. However, GNNs face the label scarcity issue as other neural networks do. Thus, recent efforts try to pre-train GNNs on a large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to the target downstream task. The adaptation is generally achieved by fine-tuning the pre-trained GNNs with a limited number of labeled data. Despite the importance of fine-tuning, current GNNs pre-training works often ignore designing a good fine-tuning strategy to better leverage transferred knowledge and improve the performance on downstream tasks. Only few works start to investigate a better fine-tuning strategy for pre-trained GNNs. But their designs either have strong assumptions or overlook the data-aware issue for various downstream datasets. Therefore, we aim to design a better fine-tuning strategy for pre-trained GNNs to improve the model performance in this paper. Given a pre-trained GNN, we propose to search to fine-tune pre-trained graph neural networks for graph-level tasks (S2PGNN), which adaptively design a suitable fine-tuning framework for the given labeled data on the downstream task. To ensure the improvement brought by searching fine-tuning strategy, we carefully summarize a proper search space of fine-tuning framework that is suitable for GNNs. The empirical studies show that S2PGNN can be implemented on the top of 10 famous pre-trained GNNs and consistently improve their performance. Besides, S2PGNN achieves better performance than existing fine-tuning strategies within and outside the GNN area. Our code is publicly available at [https://anonymous.4open.science/r/code_jcdc2024-A9CB/](https://anonymous.4open.science/r/code_jcdc2024-A9CB/).

 graph neural network, fine-tuning, pre-training graph neural networks

## I Introduction

As one of the most ubiquitous data structures, graph is a powerful way to represent diverse and complex real-world systems, e.g., social networks [1], protein interactions [2], and molecules [3, 4, 5]. Graph representation learning [6] maps the original graph into the low-dimensional vector space to handle various graph scenarios. Recently, Graph Neural Networks (GNNs) [7, 8, 9, 10, 11, 12, 13, 14], which follow the message-passing schema [15] to learn representations via iteratively neighboring message aggregation, have become the leading approaches towards powerful graph representation learning. GNNs have demonstrated state-of-the-art performances in a variety of graph tasks. e.g., node classification [7, 8, 9, 13], link prediction [16, 17, 18, 12], and graph classification [3, 4, 5, 10, 15, 19].

Despite the revolutionary success of GNNs on graph data, they are mainly trained in an end-to-end manner with task-specific supervision, which generally requires abundant labeled data. However, in many realistic graph scenarios, high-quality and task-specific labels can be scarce, which seriously impedes the application of GNNs on many graph scenarios [20]. The label scarcity issue is mainly caused by requiring extensive and laborious expert knowledge for adequate annotation, which can be even more exacerbated in some scientific fields, e.g., medicine, chemistry, and biology. Therefore, some recent efforts [20, 21, 22, 23, 24, 25, 26] investigate the pre-training [27] in GNNs so as to tackle this challenge and improve the generalization performances of GNNs. These works mainly follow the self-supervised way to pre-train GNNs on large-scale unlabeled graph data by exploiting various self-supervised learning (SSL) [28] strategies, including Autoregressive Modeling (AM) [29, 23], Masked Component Modeling (MCM) [20, 30], and Contrastive Learning (CL) [21, 22, 25, 21]. Then, the invariant knowledge captured by the pre-trained GNNs can be transferred to the downstream graph tasks. However, due to domain discrepancy [32], the _fine-tuning_ strategy [33, 34] is required to adapt the transferred knowledge from pre-trained GNNs to the downstream task by training the model with smaller-scale and limited labels.

Although tremendous efforts have been put into the designs of various SSL strategies to attain more powerful pre-trained GNNs, how to leverage these models and conduct fine-tuning on downstream datasets is still under exploration. Currently, the most common strategy is vanilla fine-tuning [35], where the downstream model is initialized by the pre-trained parameters and continually trained on the target dataset. Unfortunately, the vanilla fine-tuning for pre-trained GNNs may be inadequate and problematic. Generally, the success of vanilla fine-tuning heavily relies on the consistency between the pre-training and downstream data structures and properties [20, 36], which however may not always hold. For example, in the context of molecular graphs, downstream datasets usually encompass novel substructures (a.k.a., _scaffold_[37]), which have not been encountered during pre-training. Besides, downstream graph scenarios in real world are often complex and diverse. Blindly utilizing the vanilla strategy to fine-tune the pre-trained GNNs under various downstream scenarios may be inflexible and insufficient.

A few recent efforts [38, 39, 32] dedicate to designing novel GNN fine-tuning strategies so as to improve the vanilla solution and mitigate potential issues. AUX-TS [32] augments
downstream objectives with SSL objectives in an adaptive manner to improve the flexibility and effectiveness. WordReg [38] develops smoothness-inducing regularization built on dropout [40] to constrain the representation distance induced by pre-trained and fine-tuned models. GTOT-Tuning [39] considers graph structures and proposes optimal transport-based regularization to preserve invariant knowledge. However, there are still some obvious problems exist among existing solutions. Firstly, some improved solutions may be infeasible in practice, e.g., AUX-TS, which takes the auxiliary SSL tasks as prerequisite but they are often inaccessible for downstream graph tasks. Secondly, WordReg and GTOT-Tuning focus on sophisticated regularization to fine-tune GNN parameters. But they neglect the fundamental fact that variations in best-performing architectures may generally exist between pre-training and downstream dataset. Such a data-aware issue [41] has been demonstrated crucial in GNN architecture design literatures [42, 43, 44, 45]. Therefore, how to leverage pre-trained GNNs to achieve better adaption for downstream graph tasks seems still a challenging problem that need to be resolved.

To fully unleash the potential of pre-trained GNNs on various graph tasks, we propose a novel idea that adaptively designs a suitable fine-tuning framework for the given pre-trained GNN and downstream dataset, namely searching to fine-tune pre-trained graph neural networks for graph classification (S2PGNN). Specifically, we first investigate the existing fine-tuning works and carefully summarize a search space of fine-tuning frameworks that is suitable for GNNs, which incorporate influential design dimensions and powerful candidates of the existing fine-tuning works. Then, we leverage an efficient search algorithm to search a concrete fine-tuning strategy from the proposed search space. The main contributions of this work are summarized as follows:

* In this paper, we present a novel view that systemically exploring GNN fine-tuning strategies, an important yet seriously under-investigated problem, to improve the utilization of pre-trained GNNs.
* To further improve pre-trained GNNs, we present a new work S2PGNN that adaptively designs a suitable fine-tuning strategy for the given pre-trained GNN and downstream dataset, which broadens the perspective of the GNN fine-tuning works.
* We propose a novel search space of fine-tuning strategies in S2PGNN, which identifies key factors that affect GNN fine-tuning results and presents improved strategies. The S2PGNN framework is model-agnostic and can be readily plugged into existing GNN backbone models and pre-trained GNNs for better downstream performances.
* The empirical studies demonstrate that S2PGNN can be implemented on the top of 10 famous pre-trained GNNs and consistently improve their performance. Besides, S2PGNN achieves better performance than existing fine-tuning strategies within and outside the GNN area.

## II Related Work

A graph typically can be represented as \(G=(V,E,\mathbf{A},\mathbf{X}^{V},\mathbf{X}^{E})\), where \(V=\{v_{1},\ldots,v_{n}\}\) is the node-set, \(E=\{(v_{i},v_{j})|v_{i},v_{j}\in V\}\) is the edge-set, \(\mathbf{A}\in\{0,1\}^{|V|\times|V|}\) is the adjacency matrix to define graph connectivity and \(\mathbf{A}_{ij}=1\) iff \((v_{i},v_{j})\in E\), \(\mathbf{X}^{V}=[X_{v_{1}}^{\top},X_{v_{2}}^{\top},...]\in\mathbb{R}^{|V| \times d}\) (\(\mathbf{X}_{v}\in\mathbb{R}^{d}\) for the node \(v\)) is the node attribute matrix with feature dimension \(d\), and \(\mathbf{X}^{E}\in\mathbb{R}^{|E|\times d}\) (\(\mathbf{X}_{uv}\in\mathbb{R}^{d}\) for edge \((u,v)\)) is the edge attribute matrix.

In general, the learning on graph data first requires a graph encoder \(f_{\mathbf{\theta}}(\cdot)\) parameterized by \(\mathbf{\theta}\) to map the original graph \(G\) into the low-dimensional vector space \(\mathbb{R}^{d}\): \(\mathbf{H}=f_{\mathbf{\theta}}(G)\), where \(\mathbf{H}\in\mathbb{R}^{d}\) can be the learned representation of node, edge, or graph, depending on the prediction level of downstream tasks. Then, \(\mathbf{H}\) can be fed into an additional prediction head \(g_{\mathbf{\omega}}(\cdot)\) parameterized by \(\mathbf{\omega}\) (e.g., linear classifier, multi-layer perceptron) to generate predicted labels: \(y_{pred}=g_{\mathbf{\omega}}(\mathbf{H})\). After that, the entire model \(g_{\mathbf{\omega}}(f_{\mathbf{\theta}}(\cdot))\) can be trained in an end-to-end manner under the guidance of task-specific supervision:

\[\mathbf{\theta}^{*},\mathbf{\omega}^{*}=\arg\min_{\mathbf{\theta},\mathbf{\omega}}\mathcal{L} _{sup}(g_{\mathbf{\omega}}(f_{\mathbf{\theta}}(\cdot));\mathcal{D}^{tra}), \tag{1}\]

where \(\mathcal{L}_{sup}(\cdot)\) is the supervised loss function (e.g., cross entropy). As for the training data \(\mathcal{D}^{tra}\), it represents the graph data with labels and also depends on the down-streaming task, e.g., \(\mathcal{D}^{tra}=\{(G,y)\}\) for the graph classification.

### _Graph Neural Networks (GNNs)_

Recent years have witnessed the unprecedented success of GNNs for modeling graph data and dealing with various graph tasks. As one of powerful graph encoders \(f_{\mathbf{\theta}}(\cdot)\), GNNs typically leverage the graph topology \(\mathbf{A}\) as well as features \(\mathbf{X}^{V}\)

\begin{table}
\begin{tabular}{l|l} \hline
**Notation** & **Definition** \\ \hline \(\mathbb{R}^{d}\) & The \(d\)-dimension real space. \\ \hline \(G\!=\!(V,E,\mathbf{A},\mathbf{X}^{V},\mathbf{X}^{E})\) & An attributed graph with node-set \(V\), edge-set \(E\), adjacency matrix \(\mathbf{A}\), and attribute matrices \(\mathbf{X}^{V}\) and \(\mathbf{X}^{E}\). \\ \hline \(V\), \(E\) & \(V=\{v_{1},\ldots,v_{n}\}\), \\  & \(E=\{(v_{i},v_{j})|v_{i},v_{j}\in V\}\). \\ \hline \(\mathbf{X}^{V}\in\mathbb{R}^{|V|\times d},\mathbf{X}_{v}\in\mathbb{R}^{d}\) & Node attribute matrix and vector. \\ \hline \(\mathbf{X}^{E}\in\mathbb{R}^{|E|\times d},\mathbf{X}_{uv}\in\mathbb{R}^{d}\) & Edge attribute matrix and vector. \\ \hline \(k\), \(K\) & The current and maximum GNN layer index, \(1\leq k\leq K\). \\ \hline \(\mathbf{H}_{k}^{(k)}\!\in\!\mathbb{R}^{d}\) & The node representation at \(k\)-th layer. \\ \hline \(\mathbf{H}_{G}\!\in\!\mathbb{R}^{d}\) & The graph representation. \\ \hline \(f_{\mathbf{\theta}}(\cdot)\) & The GNN encoder with parameters \(\mathbf{\theta}\). \\ \hline \(g_{\mathbf{\omega}}(\cdot)\) & The prediction head with parameters \(\mathbf{\omega}\). \\ \hline \(\pi_{\mathbf{\alpha}}(\cdot)\) & The GNN fine-tuning controller with parameters \(\mathbf{\alpha}\). \\ \hline \(\mathbf{\Phi}_{ft}\) & The GNN fine-tuning strategy. \\ \hline \(f_{aug}(\cdot)\), \(f_{fuse}(\cdot)\), \(f_{read}(\cdot)\) & The GNN fine-tuning dimension of identity augmentation, multi-scale fusion, graph-level readout. \\ \hline \(\mathcal{O}_{aug}\), \(\mathcal{O}_{fuse}\), \(\mathcal{O}_{read}\) & The candidate-set of \\  & \(f_{aug}(\cdot)\), \(f_{fuse}(\cdot)\), \(f_{read}(\cdot)\). \\ \hline \(\mathcal{D}_{ssl}\), \(\mathcal{D}_{sup}\) & The pre-training and downstream dataset. \\ \hline \(\mathcal{L}_{ssl}(\cdot)\), \(\mathcal{L}_{sup}(\cdot)\) & The pre-training and downstream loss. \\ \hline \end{tabular}
\end{table} TABLE I: A summary of common notations.
and \(\mathbf{X}^{E}\) to achieve the graph representation \(\mathbf{H}\). The majority of GNNs [7, 8, 9, 10] follow the message passing paradigm [15] to learn the representation of node \(v\) by iteratively aggregating messages from \(v\)'s neighbors \(N(v)\). Formally, the intra-layer message passing for given node \(v\) can be formulated as:

\[\mathbf{M}_{v}^{(k)}=AGG^{(k)}(\{(\mathbf{H}_{u}^{(k-1)},\mathbf{ H}_{v}^{(k-1)},\mathbf{X}_{uv})|u\in N(v)\}), \tag{2}\] \[\mathbf{H}_{v}^{(k)}=COMB^{(k)}(\mathbf{H}_{v}^{(k-1)},\mathbf{M }_{v}^{(k)}), \tag{3}\]

where \(\mathbf{H}_{v}^{(k)}\) is the representation of node \(v\) at \(k\)-th iteration (usually \(\mathbf{H}_{v}^{(0)}=\mathbf{X}_{v}\)), \(\mathbf{X}_{uv}\) is the attribute of edge \((u,v)\), \(\mathbf{M}_{v}^{(k)}\) is the intermediate representation collected from neighbors via aggregation function \(AGG(\cdot)\), \(COMB(\cdot)\) function combines information from neighbors and center node itself to update representation from \(\mathbf{H}_{v}^{(k-1)}\) to \(\mathbf{H}_{v}^{(k)}\). Obviously, after \(k\) (\(1\leq k\leq K\)) iterations/layers of aggregation, each node captures information from their \(k\)-hop neighbors. Note that \(N(v)\), \(AGG(\cdot)\), and \(COMB(\cdot)\) are crucial functions to determine the message passing process. They mainly differentiate various GNNs and may affect model expressiveness substantially. We next present several classic GNNs with specific instantiations of Eq. (2) and (3), which are adopted in later experiments.

* Graph Convolutional Network (GCN) [7] adopts \(MEAN(\cdot)\) as its mean aggregation function and the non-linear activation function \(\sigma(\cdot)\) e.g., \(ReLU(\cdot)\). It proposes to transform intermediate embeddings into representation by the trainable matrix \(\mathbf{W}\): \[\mathbf{M}_{v}^{(k)}=MEAN(\{\mathbf{H}_{u}^{(k-1)}|u\in N(v)\cup\{ v\}),\] (4) \[\mathbf{H}_{v}^{(k)}=\sigma(\mathbf{W}^{(k)}\mathbf{M}_{v}^{(k)}).\] (5)
* GraphSAGE (SAGE) [8] concatenates the intermediate embedding \(\mathbf{M}_{v}^{(k)}\) with the representation of last layer \(\mathbf{H}_{v}^{(k-1)}\) before the transformation: \[\mathbf{M}_{v}^{(k)}=MEAN(\{\mathbf{H}_{u}^{(k-1)}|u\in N(v)\}),\] (6) \[\mathbf{H}_{v}^{(k)}=\sigma(\mathbf{W}^{(k)}[\mathbf{M}_{v}^{(k)} |\mathbf{H}_{v}^{(k-1)}]).\] (7)
* Graph Isomorphism Network (GIN) [10], as one of the most expressive GNN architectures, adopts \(SUM(\cdot)\) as the sum aggregation function and the multi-layer perceptron \(MLP(\cdot)\) to transform the combined messages. Besides, it adds the scalar \(\epsilon\) to balance the messages weights from center node itself and neighbors: \[\mathbf{M}_{v}^{(k)}=SUM(\{\mathbf{H}_{u}^{(k-1)}|u\in N(v)\}),\] (8) \[\mathbf{H}_{v}^{(k)}=MLP^{(k)}((1+\epsilon^{(k)})\mathbf{H}_{v}^{ (k-1)}+\mathbf{M}_{v}^{(k)}).\] (9)
* Graph Attention Network (GAT) [9] introduce the attentive function \(ATT(\cdot)\)[46] as its aggregation function: \[\mathbf{M}_{v}^{(k)}=ATT(\{\mathbf{H}_{u}^{(k-1)}|u\in N(v)\},\] (10) \[\mathbf{H}_{v}^{(k)}=\sigma(\mathbf{W}^{(k)}\mathbf{M}_{v}^{(k)}).\] (11)

Furthermore, for the graph-level task, a permutation-invariant readout function \(READOUT(\cdot)\) is required to obtain the graph-level representation \(\mathbf{H}_{G}\) of the entire graph \(G\):

\[\mathbf{H}_{G}=READOUT(\{\mathbf{H}_{v}^{(K)}|v\in V\}), \tag{12}\]

where \(READOUT(\cdot)\) can be simple non-parameterized function, e.g., sum pooling and mean pooling, or other more advanced methods [47, 48, 49].

### _Pre-training and fine-tuning (P&F) in GNNs_

As introduced in Sec. I, labeling graph data can be time-consuming, expensive, and sometimes even infeasible in many realistic graph scenarios (e.g., molecule graphs). The lack of label information severely impedes GNN capacities, thereby posing challenges to graph learning in those fields. To alleviate the data scarcity issue, recent efforts [20, 21, 23] try to generalize the self-supervised pre-training and fine-tuning (P&F) paradigm to GNNs. Generally, they first follow the self-supervised learning (SSL) way to train a pre-trained GNN model on the large scale of unlabeled data \(\mathcal{D}_{ssl}\):

\[\boldsymbol{\theta}^{init}=\arg\min_{\boldsymbol{\theta}}\mathcal{L}_{ssl}(f_{ \boldsymbol{\theta}}(\cdot);\mathcal{D}_{ssl}) \tag{13}\]

where \(\mathcal{L}_{ssl}(\cdot)\) is the SSL loss Then, to achieve the knowledge transfer from \(\mathcal{D}_{ssl}\), the down-streaming model is initialized by the pre-trained parameter \(\boldsymbol{\theta}^{init}\) and fine-tuned with a small fine-tuning data \(\mathcal{D}_{ft}\):

\[\boldsymbol{\theta}^{*},\boldsymbol{\omega}^{*}=\arg\min_{\boldsymbol{\theta}, \boldsymbol{\omega}}\mathcal{L}_{ft}((g_{\boldsymbol{\omega}}(f_{\boldsymbol{ \theta}}(\cdot));\mathcal{D}_{ft}), \tag{14}\]

where the initialization model \(f_{\boldsymbol{\theta}^{init}}(\cdot)\) is optimized by the fine-tuning objective \(\mathcal{L}_{ft}(\cdot)\). We next introduce several ways to instantiate the strategies of self-supervised pre-training in Eq. (13) and fine-tuning in Eq. (14), respectively.

#### Iii-B1 Self-supervised pre-training strategies

The literature design novel self-supervised pre-training objectives \(\mathcal{L}_{ssl}(\cdot)\) to empower the pre-trained GNNs. Due to the space limit, we here briefly introduce several well-known and representative methods (please refer to [28, 54, 55, 56] for more details), which can be roughly categorized into generative and contrastive based. The former pre-train GNNs to predict a graph or subgraph that is similar to the original input graph, while the latter aims to distinguish between pairs of similar and dissimilar graphs.

* AutoEncoding (AE): Given the partial access to graph, AE methods (e.g., EdgePred [8] and GraphMAE [26]) propose to reconstruct the input graph via autoencoder architecture

Fig. 1: The illustration of overall GNN P&F framework.
[57]. Let \(\tilde{G}\) be reconstructed graph, they formulate the objective \(\mathcal{L}_{ssl}(\cdot)\) as: \[\mathcal{L}_{ssl}(\cdot)=-\sum_{G\in\mathcal{D}_{ssl}}\log p(\tilde{G}|G).\] (15)
* Autoregressive Modeling (AM): AM methods (e.g., MGSSL [23]) factorize the input graph \(G\) as a sequence of components \(\mathcal{C}=\{C_{1},C_{2},\dots\}\) (e.g., nodes, edges, and subgraphs) with some preset ordering and perform graph reconstruction in an autoregressive manner: \[\mathcal{L}_{ssl}(\cdot)=-\sum_{G\in\mathcal{D}_{ssl}}\sum_{i=1}^{|\mathcal{C }|}\log p(C_{i}|C_{<i}).\] (16)
* Masked Component Modeling (MCM): MCM works (e.g., AttrMasking [20] and Mole-BERT [30]) masks out some components of the input graphs (e.g., nodes, edges, and subgraphs), then recover those masked ones \(m(G)\) through the remaining ones \(G\backslash m(G)\): \[\mathcal{L}_{ssl}(\cdot)=-\sum_{G\in\mathcal{D}_{scl}}\sum_{\hat{G}\in m(G)} \log p(\hat{G}|G\backslash m(G)).\] (17)
* Context Prediction (CP): CP works explore graph structures and utilize contextual information to design pre-training objectives. Let \(t=1\) if subgraph \(C_{1}\) and surrounding context \(C_{2}\) share the same center node, otherwise \(t=0\). ContextPred [20] leverages subgraphs to predict their surrounding context structures: \[\mathcal{L}_{ssl}(\cdot)=-\sum_{G\in\mathcal{D}_{ssl}}\log p(t|C_{1},C_{2}).\] (18)
* Contrastive Learning (CL): CL approaches perform pre-training via maximizing the agreement between a pair of similar inputs, including Cross-Scale Contrastive Learning and Same-Scale Contrastive Learning. The former one (e.g., Infomax [58]) contrasts a pair of graph and its local sub-structure \((G,C)\) against negative pairs \((G,C^{-})\): \[\mathcal{L}_{ssl}(\cdot)\!=\!-\!\sum_{G\in\mathcal{D}_{scl}}[\log s(G,C)\!-\! \sum_{C^{-}}\log s(G,C^{-})]],\] (19) where \(s(\cdot,\cdot)\) is the similarity function. The latter ones (e.g., GraphCL [21], SimGRACE [25], and GraphLoG [22]) maximize the agreement between the augmented graph and its anchor graph \((G,G^{+})\) and repel negative pairs \((G,G^{-})\): \[\mathcal{L}_{ssl}(\cdot)\!=\!-\!\sum_{G\in\mathcal{D}_{scl}}[\log s(G,G^{+}) \!-\!\!\sum_{C^{-}}\log s(G,G^{-})]].\] (20)

#### Iii-B2 Fine-tuning strategies

Despite the development of self-supervised GNN pre-training strategies (see Sec. II-B1), the design of GNN fine-tuning strategies, i.e., how to instantiate Eq. (14), is still under exploration. So far, there are only a few fine-tuning methods proposed specifically for pre-trained GNNs. As summarized in Tab. II, here we first review 2 mainstream strategies in GNN domains:

* Standard Tuning (ST): ST is the most prevalent fine-tuning strategy among existing GNN P&F literatures [20, 21, 23, 25]. Under this schema, all parameters \((\boldsymbol{\theta}_{init},\boldsymbol{\omega})\) of the GNNs are further fine-tuned under the supervision \(\mathcal{L}_{sup}(\cdot)\) for downstream tasks. \[\boldsymbol{\theta}^{*},\boldsymbol{\omega}^{*}=\arg\min_{\boldsymbol{\theta} _{init},\boldsymbol{\omega}}\mathcal{L}_{ssp}((g_{\boldsymbol{\omega}}(f_{ \boldsymbol{\theta}_{init}}(\cdot));\mathcal{D}_{ft}).\] (21)
* Regularized Tuning (RT): RT methods generally introduce a regularizer loss \(\mathcal{L}_{reg}(\cdot)\) into \(\mathcal{L}_{ft}(\cdot)\): \[\mathcal{L}_{ft}(\cdot)=\mathcal{L}_{ssl}(\cdot)+\mathcal{L}_{reg}(\cdot),\] (22) where different models varies in the way to instantiate \(\mathcal{L}_{reg}(\cdot)\). In literature, GTOT-Tuning [39] is specifically designed for fine-tuning GNNs, which considers the topology information in graph and presents an optimal transport-based feature regularizer. In Sec. IV-C, we also discussed several RT methods that are initially developed for fine-tuning CNNs, including \(L^{2}\)-SP [50], DELTA [51], BSS [52], and StochNorm [53].

Except for above ST and RT methods, we investigate the performance of more fine-tuning strategies from other domains on the pre-training GNN tasks (see Sec. IV-C for more details).

\begin{table}
\begin{tabular}{l|l|c c|c c c|c} \hline \hline \multirow{2}{*}{**Strategy Name**} & \multicolumn{3}{c|}{**Fine-tuning Scenario**} & \multicolumn{3}{c|}{**Fine-tuning Dimension**} & \multirow{2}{*}{**Automated**} \\ \cline{3-4} \cline{6-7}  & & GNN/Other & Graph Task & Architecture & Identity & Fusion & Readout \\ \hline Standard Tuning (ST) & \(\surd\surd\) & \(\surd\)/ & Node/EdgeGraph & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline \multirow{6}{*}{Regularized Tuning (RT)} & \(L^{2}\)-SP [50] & \(\times\)/ & - & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & DELTA [51] & \(\times\)/ & - & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & BSS [52] & \(\times\)/ & - & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & StochNorm [53] & \(\times\)/ & - & \(\surd\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & GTOT-Tuning [39] & \(\surd\)/ & Graph & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & AUX-TS [32] & \(\surd\)/ & Node/Edge & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \cline{2-7}  & WordReg [38] & \(\surd\)/ & Graph & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline Parameter-Efficient Tuning (PET) & \(\times\)/ & - & \(\surd\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline Feature Extractor (FE) & \(\times\)/ & - & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline Last-\(k\) Tuning (LKT) & \(\times\)/ & - & \(\times\) & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\ \hline S2PGNN & \(\surd\)/ & Graph & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) \\ \hline \hline \end{tabular}
\end{table} TABLE II: Overview of common fine-tuning strategies in GNN and other domains. Fine-tuning Scenario presents the application scenario of the fine-tuning strategies, including whether is designed for GNN or other neural networks and which graph task the strategy can be applied to. Fine-tuning Dimension presents what aspects they focus on during fine-tuning.
### _Automated GNNs (AutoGNNs)_

Despite the great success, many existing GNNs rely heavily on expert knowledge to manually design GNN model architectures. In most cases, the well-performing GNNs on some graphs may not be suitable on other graphs due to the divergence in graph structures or properties as well as task-specific requirements. And real-world graph datasets and tasks usually exhibit complex and diverse patterns. Besides, out-of-distribution predictions may also prevalent. Therefore, the conventional way to simply apply the fixed GNN architectures to all graph scenarios may be suboptimal, Instead, data/task-specific GNN architecture designs are much demanded. Recent works argue that such hand-crafted GNNs are prone to suffer from the well-known data-aware issue [41], i.e., the optimal GNN architecture for different graphs may be quite different.

To alleviate the data-aware issue and better handle versatile graph applications, Automated GNNs (AutoGNNs) [42, 43, 44, 45] have been recently developed and demonstrate promising results. AutoGNNs leverage GNN controllers to achieve automated GNN architecture designs for the specific given graph data and task. Typically, AutoGNNs first design a unified and comprehensive GNN search space, which include various key design dimensions (functions) in the message passing process. Each design dimension may contain multiple candidate choices (operations) that can be selected by the controller to constitute the optimal model. Then, various search algorithms, e.g., Reinforcement Learning (RL)-based methods [59], Evolutionary Algorithm (EA)-based methods [60], and differentiable methods [61], can be leveraged to instantiate the controller and achieve the search objective. Novel GNN architectures identified by AutoGNN approaches have demonstrated superior results than their hand-crafted counterparts on a wide range of graph scenarios, e.g, node classification [42, 43, 44, 62], link prediction [43], and graph classification [43, 64, 63]. However, existing AutoGNNs neglect important design dimensions for fine-tuning pre-trained GNNs, which makes them incapable to handle various downstream fine-tuning scenarios.

## III Methodology

As introduced in Sec. I, recent developed pre-trained GNNs have demonstrated promising to tackle the data scarcity issue and facilitate downstream learning. Despite the importance of fine-tuning, current pre-trained GNN works often ignore designing a good fine-tuning strategy to better leverage transferred knowledge and the improve downstream performance. Only few works start to investigate this problem, but their designs either have strong assumptions or overlook the data-aware issue for various downstream datasets.

Therefore, in this work, we aim to design a better fine-tuning strategy S2PGNN for pre-trained GNNs to improve the model performance. S2PGNN adaptively designs a suitable fine-tuning framework for the given labeled data on the downstream task. To ensure the improvement brought by searching fine-tuning strategy, we carefully design a novel search space of fine-tuning that is suitable for GNNs, which identifies key factors that affect GNN fine-tuning results and presents improved strategies. The S2PGNN framework is model-agnostic and can be readily plugged into existing GNN backbone models and pre-trained GNNs. In Sec. III-A, we formulate our search problem and provide the overall optimization objective. In Sec. III-B, we present the proposed GNN fine-tuning search space. In Sec. III-C, we demonstrate how we leverage a controller to search the optimal fine-tuning strategy from the defined search space. In Sec. III-D, we provide the overall optimization.

### _Problem Formulation_

Given a pre-trained GNN and downstream dataset \(\mathcal{D}_{ft}\), the problem of S2PGNN is to design a GNN fine-tuning strategy on top of the given pre-trained GNN that can achieve good performance on \(\mathcal{D}_{ft}\). Formally, it can be formulated as:

\[\boldsymbol{\alpha^{*}},\boldsymbol{\theta^{*}},\boldsymbol{\omega^{*}}\!=\! \arg\min_{\boldsymbol{\alpha},\boldsymbol{\theta},\boldsymbol{\omega}}\!\! \mathbb{E}_{\boldsymbol{\Phi}_{ft}\sim\pi_{\boldsymbol{\alpha}}(\cdot)}\mathcal{ L}_{ft}[g_{\boldsymbol{\omega}}(f_{\boldsymbol{\theta}}(\cdot));\mathcal{D}_{ft}], \tag{23}\]

where \(\pi_{\boldsymbol{\alpha}}(\cdot)\) parameterized by \(\boldsymbol{\alpha}\) (light-weight) is the controller that leveraged to search the optimal strategy \(\boldsymbol{\Phi}_{ft}\) (i.e., \(\boldsymbol{\Phi}_{ft}\sim\pi_{\boldsymbol{\alpha}}(\cdot)\)) from proposed search space (see Sec. III-B) to fine-tune the downstream model \(g_{\boldsymbol{\omega}}(f_{\boldsymbol{\theta}}(\cdot))\) with parameters \((\boldsymbol{\theta},\boldsymbol{\omega})\), and \(\mathbb{E}(\cdot)\) is the expectation function.

### _GNN Fine-tuning Search Space_

To ensure the improvement brought by searching fine-tuning strategy, we identify that identity augmentation \(f_{aug}(\cdot)\), multi-scale fusion \(f_{fuse}(\cdot)\), and graph-level readout \(f_{read}(\cdot)\) operations within the message passing procedure are key

Fig. 2: Illustration of GNN fine-tuning strategies (refer to Fig. 3 for the legend).
factors that affect GNN fine-tuning results and summarize a proper search space of fine-tuning framework that is suitable for GNNs. As illustrated in Fig. 3, the improved message functions in S2PGNN built on top of GIN can be represented as:

\[\begin{split}\mathbf{H}_{v}^{(k)}\!=\!f_{g\_conv}^{(k)}(\{\mathbf{H }_{u}^{(k-1)},\mathbf{H}_{v}^{(k-1)},\mathbf{X}_{uv}|u\in N(v)\}),\\ \mathbf{Z}_{v}^{(k)}\!=\!f_{aug}^{(k)}(\mathbf{H}_{v}^{(k-1)}, \mathbf{H}_{v}^{(k)}),\quad 1\leq k\leq K,\end{split} \tag{24}\]

\[\mathbf{H}_{v}=f_{fuse}(\{\mathbf{Z}_{v}^{(1)},\mathbf{Z}_{v}^{(2)},\dots, \mathbf{Z}_{v}^{(K)}\}), \tag{25}\] \[\mathbf{H}_{G}=f_{read}(\{\mathbf{H}_{v}|v\in V\}), \tag{26}\]

where \(f_{g\_conv}(\cdot)\) (denoted as _GINConv_ in Fig. 3) is the intra-layer message aggregator. Note that \(f_{g\_conv}(\cdot)\) is the most basic building block in both pre-trained and fine-tuned model, which can be replaced by other GNN backbone models or pre-trained framework. We here utilize GIN to demonstrate an example. Besides, \(f_{aug}(\cdot)\) is the identity augmentation operation that enhance the information from node itself before the next iteration of aggregation, \(f_{fuse}(\cdot)\) is the multi-scale fusion operation that comprehensively combines the multi-scale informations from different GNN layers, and \(f_{read}(\cdot)\) is the graph-level readout operation that adaptively aggregate node representations for graph representation. Next, we provide detailed illustrations towards key dimensions in GNN fine-tuning \(\{f_{aug}(\cdot),f_{fuse}(\cdot),f_{read}(\cdot)\}\) and according candidate-set \(\{\mathcal{O}_{aug},\mathcal{O}_{fuse},\mathcal{O}_{read}\}\) that allows the optimal strategy \(\mathbf{\Phi}_{ft}=\{f_{aug}(\cdot)=?,f_{fuse}(\cdot)=?,f_{read}(\cdot)=?\}\) to be adaptively designed (marked with orange ovals in Fig. 3). Note that in S2PGNN \(\mathbf{\theta}=\{\mathbf{\theta}_{g\_conv}\}\cup\{\mathbf{\theta}_{aug}, \mathbf{\theta}_{fuse},\mathbf{\theta}_{read}\}\) (marked with blue \(\cup\) orange in Fig. 3).

#### Iii-B1 Identity Augmentation \(\mathcal{O}_{aug}\)

Augmenting the identity information \(\mathbf{H}_{v}^{(k-1)}\) from center node itself can be indispensable for GNN fine-tuning due to several reasons. Firstly, in some downstream cases, node-specific information may be more important than messages from neighbors, since the latter sometimes can be missing, noisy, or unreliable in real-world datasets. Besides, in some GNN backbone architectures, node-specific information may be easy to lose and representations may be prone to become indistinguishable (a.k.a., over-smoothing [65]) with the increase of GNN layers. This can happen if the adopted aggregation function in pre-trained backbones only considers the information of neighboring nodes (e.g., GCN). Unfortunately, existing GNN fine-tuning works tend to ignore the importance of preserving node-specific identity information and lack of corresponding designs as shown in Tab. II. Therefore, we propose to incorporate identity augmentation into the GNN fine-tuning design space and allows suitable augmentation to be adaptively searched from the following candidates in Eq. (24):

* **No augmentation.** We do not perform additional identity augmentation and keep consistent as in pre-trained backbone with \(zero\_aug\) operation: \(\mathbf{Z}_{v}^{(k)}=\mathbf{H}_{v}^{(k)}\).
* **Additive augmentation.** We allow direct skip-connection as in [] with \(identity\_aug\) operation: \(\mathbf{Z}_{v}^{(k)}=\mathbf{H}_{v}^{(k-1)}+\mathbf{H}_{v}^{(k)}\). We also allow transformed augmentation with \(trans\_aug\) operation: \(\mathbf{Z}_{v}^{(k)}=\phi^{(k)}(\mathbf{H}_{v}^{(k-1)})+\mathbf{H}_{v}^{(k)}\), where \(\phi^{(k)}(\cdot)\) is a parameterized neural network with bottleneck architecture \(\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}\rightarrow\mathbb{R}^{d}\) (we let \(m\ll f\) for parameter-efficient similar as Adapter [34]).

#### Iii-B2 Multi-scale Fusion \(\mathcal{O}_{fuse}\)

Immediate information from different pre-trained GNN layers \(\{\mathbf{Z}_{v}^{(1)},\mathbf{Z}_{v}^{(2)},\dots,\mathbf{Z}_{v}^{(K)}\}\) may be better suited for capturing graph information at different scales, e.g., local or global [11]. Besides, different downstream graph datasets and tasks may also have their specific and variant requirements in terms of the most suitable number of GNN layers [13, 66]. However, existing strategies to fine-tune pre-trained GNNs simply leverage the last-layer output to induce final representations, i.e., \(\mathbf{H}_{v}=0\cdot\mathbf{Z}_{v}^{(1)}\!+\!0\cdot\!\mathbf{Z}_{v}^{(2)}\!+ \!\dots\!+\!1\!\cdot\!\mathbf{Z}_{v}^{(K)}\), which may be suboptimal and cause unsatisfactory fine-tuning performances. Therefore, we incorporate the multi-scale fusion as novel dimension into the GNN fine-tuning space, which allows flexibly information fusion and utilization for better downstream performances. Fusion process can be roughly represented as \(\mathbf{H}_{v}=w_{v}^{(1)}\cdot\mathbf{Z}_{v}^{(1)}+w_{v}^{(2)}\cdot\mathbf{Z}_ {v}^{(2)}+\dots+w_{v}^{(K)}\cdot\mathbf{Z}_{v}^{(K)}\), where \(w_{v}^{(k)}\) is importance weights of information at \(k\)-th layer for node \(v\), which is to be assigned or adaptively learned depending on specific fusion choices. We include candidate fusion operations from diverse categories:

* **Non-parametric fusion.** Non-parametric methods to attain \(w_{v}^{(k)}\) are simple and computational efficient. Among candidates listed in Tab. III, \(\{last(\cdot),concat(\cdot),max(\cdot),mean(\cdot),ppr(\cdot)\}\) are from this category, where \(last(\cdot)\)

Fig. 3: Illustration to the framework of S2PGNN built on top of pre-trained 5-layer GIN. The orange part indicates the search dimensions in S2PGNN. PT., FT., and Id.Aug. are abbreviations for pre-training, fine-tuning, and identity augmentation.
disables fusion and directly takes the last-layer output as learned representations, \(concat(\cdot)\) conducts multi-scale concatenation as \(\mathbf{H}_{v}=[\mathbf{Z}_{v}^{(1)}]|\ldots||\mathbf{Z}_{v}^{(K)}]\), \(max(\cdot)\) takes the maximum value from each channel to induce fused representations, \(mean(\cdot)\) assigns equal importance weights for each layer, and \(ppr(\cdot)\) assigns decayed importance weights with Personalized PageRank method as in [67].
* **Attentive fusion.** Attentive fusion methods adaptively learn importance weights \(w_{v}^{(k)}\) via the attention mechanism, which generally have \(w_{v}^{(k)}\in[0,1]\) and \(\sum_{k}w_{v}^{(k)}=1\). We adopt the powerful \(lstm\) fusion that is similar as in [11].
* **Gated fusion.** Gated fusion methods use gating functions to selectively filter information at different layers. We use \(gpr\) method for this category similar as in [13], which allows the adaptive scale as well as the sign of information, i.e., \(w_{v}^{(k)}\in[-1,1]\).

#### Iii-B3 Graph-level Readout \(\mathcal{O}_{read}\)

Graph-level readout aggregates representations across all nodes to attain the graph-level representation, which is a compulsory function for graph-level downstream tasks, e.g., graph classification or regression. Different readout methods may focus on capturing different aspects of node features or graph topology [68]. Besides, they may also be more or less suitable for handling downstream graphs with variant structures and properties. Thus, it is important to carefully consider the data characteristics and task-specific requirements to adaptively determine the most suitable readout function for the specific downstream task. However, existing GNN fine-tuning works largely ignore the importance of adaptive readout methods and lack of according designs, which may hinder their performances on diverse downstream scenarios. Therefore, we integrate the graph-level readout into S2PGNN's fine-tuning space so that the optimal readout method can be adaptively selected. We include multiple candidates from various categories:

* **Simple readout.** Simple readout methods are parameter-free and computationally efficient, which may be effective for scenarios where the overall structure of the graph is less important than the individual node features. We include \(sum\_pooling(\cdot)\), \(mean\_pooling(\cdot)\), and \(max\_pooling(\cdot)\) readouts for this type.
* **Adaptive readout.** Other readout methods, such as adaptive readout, focus on identifying and capturing the most informative nodes or substructures via more sophisticated designs. They may be more effective for scenarios where specific nodes or substructures in the graph are more crucial. We carefully review existing readout literatures and cover powerful candidates \(set2set\)[69], \(sort\_pooling\)[70], \(multiset\_pooling\)[49], and \(neural\_pooling\)[71] for this category.

### _GNN Fine-tuning Controller_

Based on proposed search space \(\{\mathcal{O}_{aug},\mathcal{O}_{fuse},\mathcal{O}_{read}\}\) for dimensions \(\{f_{aug}(\cdot),f_{fuse}(\cdot),f_{read}(\cdot)\}\), we then leverage the light-weighted controller \(\pi_{\boldsymbol{\alpha}}(\cdot)\) to allow the selection of most suitable GNN fine-tuning strategy \(\boldsymbol{\Phi}_{ft}=\{f_{aug}(\cdot)=?,f_{fuse}(\cdot)=?,f_{read}(\cdot)=?\}\) for each downstream task.

We decompose the fine-tuning strategy search procedure into sub-steps and decisions are to be made for each proposed design dimension via their respective controllers. For the given design dimension with candidate-set \(\mathcal{O}\) (e.g., \(\mathcal{O}_{fuse}\)), we let the one-hot vector \(\boldsymbol{\Phi}_{ft}^{\mathcal{O}}\in\{0,1\}^{|\mathcal{O}|}\) record the optimal strategy for \(\mathcal{O}\). We let \(\pi_{\boldsymbol{\alpha}^{\mathcal{O}}}(\cdot)\) denote its controller parameterized by \(\boldsymbol{\alpha}^{\mathcal{O}}\), which records the importance weights of each candidate in \(\mathcal{O}\) that to be selected:

\[\boldsymbol{\Phi}_{ft}^{\mathcal{O}}\sim\pi_{\boldsymbol{\alpha}^{\mathcal{O} }}(\cdot). \tag{27}\]

Then, given some intermediate representation \(\mathbf{Z}_{in}\) as input, the output from this dimension can be calculated as:

\[\mathbf{Z}_{out}=\sum_{i=1}^{|\mathcal{O}|}\boldsymbol{\Phi}_{ft}^{\mathcal{O} }[i]\cdot\mathcal{O}[i](\mathbf{Z}_{in}). \tag{28}\]

\(\boldsymbol{\alpha}^{\mathcal{O}}\) need to be optimized such that the controller can yield improved fine-tuning strategy. However, the controller selection step in Eq. (27) is intrinsically discrete, which makes the entire model non-differentiable and thereby infeasible be optimized via back-propagation.

To tackle this, we leverage the re-parameterization technique in [72], [73] to make the discrete process to be differentiable so that the entire S2PGNN model can be optimized via general gradient descend. Specifically, during forward-propagation stage, the selection (sampling) step in Eq. (27) is achieved via:

\[\boldsymbol{\Phi}_{ft}^{\mathcal{O}}=onehot(\arg\max_{i}[\log \boldsymbol{\alpha}^{\mathcal{O}}[i]-\log(-\log(\boldsymbol{U}[i]))]), \tag{29}\]

where \(\boldsymbol{U}\in\mathbb{R}^{|\mathcal{O}|}\) has same size as \(\boldsymbol{\alpha}^{\mathcal{O}}\) and each entry of it is sampled from the uniform distribution, i.e., \(\boldsymbol{U}[i]\sim Uniform(0,1)\). The induced \(-\log(-\log(\boldsymbol{U}[i]))\) is known as _Gumbel_ random variable [73]. Then, to approximate and back-propagate gradient \(\frac{\partial\boldsymbol{\Phi}_{ft}^{\mathcal{O}}}{\partial\boldsymbol{ \alpha}^{\mathcal{O}}}\), we follow [73] to unitize softmax as continuous relaxation towards \(\arg\max\):

\[\boldsymbol{\Phi}_{ft}^{\mathcal{O}}=\frac{\exp((\log\boldsymbol{\alpha}^{ \mathcal{O}}[i]-\log(-\log(\boldsymbol{U}[i])))/\tau)}{\sum_{j=1}^{|\mathcal{O }|}\exp((\log\boldsymbol{\alpha}^{\mathcal{O}}[j]-\log(-\log(\boldsymbol{U}[ j])))/\tau)}, \tag{30}\]

where \(\tau\) is the softmax temperature that controls discreteness. Note that \(\tau\to 0\) makes the continuous output from softmax Eq. (30) almost indistinguishable with the one-hot vector from \(\arg\max\) Eq. (29), which thereby ensures the continuous relaxation unbiased once converged. In this way, the controller can be jointly optimized with GNN model weights to solve the optimization objective in Eq. (23).

\begin{table}
\begin{tabular}{c|c|c|c} \hline \hline Type & Design Dimension & Candidate-set \(\mathcal{O}\) \\ \hline \hline \multirow{2}{*}{Inter-layer} & Identity Augmentation & \(f_{aug}(\cdot)\) & \(\{zero\_aug,identity\_aug,trans\_aug\}\) \\ \cline{2-4}  & Multi-scale Fusion & \(f_{fuse}(\cdot)\) & \(\{last,concat,max,mean,ppr,lstm,gpr\}\) \\ \hline Graph-level & Graph-level Readout & \(f_{read}(\cdot)\) & \(\{sum\_pooling,mean\_pooling,max\_pooling,set2set,sort\_pooling,neural\_pooling\}\) \\ \hline \hline \end{tabular}
\end{table} TABLE III: The GNN fine-tuning design space in S2PGNN: design dimensions and candidate-sets.
### _Optimization_

Empirically, we focus on the graph-level classification and regression tasks for the downstream molecular property prediction MPP (see more details for MPP task in Sec. IV-A3). To achieve the optimization goal in Eq. (23), we follow common practices to choose proper loss functions \(\mathcal{L}_{ft}(\cdot)\) for downstream graph dataset with labels \(\mathcal{D}_{ft}=\{(G,y)\}\).

For the graph classification task, we leverage cross-entropy loss as Eq. (31), where \(C\) denotes the total number of graph classes in downstream dataset \(\mathcal{D}_{ft}\), \(\mathbf{Z}_{i}\in\mathbb{R}^{C}\) is the prediction results for \(i\)-th graph instance, and \(y_{i}\in\mathbb{R}\) is its true label.

\[\mathcal{L}_{ft}(\cdot)=\frac{1}{|\mathcal{D}_{ft}|}\sum_{i=1}^{|\mathcal{D}_{ft }|}-log(\frac{\exp(\mathbf{Z}_{i}[y_{i}])}{\sum_{j=0}^{C-1}\exp(\mathbf{Z}_{i} [j])}). \tag{31}\]

For the graph regression task, we utilize MSE loss as Eq. (32), where \(\hat{y}_{i}\in\mathbb{R}\) is the predicted value for \(i\)-th graph instance.

\[\mathcal{L}_{ft}(\cdot)=\frac{1}{|\mathcal{D}_{ft}|}\sum_{i=1}^{|\mathcal{D}_ {ft}|}(y_{i}-\hat{y}_{i})^{2}. \tag{32}\]

## IV Experiments

As presented in Sec. II, the GNN pre-training approach generally contains a GNN backbone model (e.g., GCN [7], SAGE [8]), a GNN pre-training strategy (e.g., MCM [20, 30], CL [58]) to transfer knowledge from a large scale of unlabeled graph data \(\mathcal{D}_{ssl}\), a GNN fine-tuning strategy (e.g., ST [20], RT [39]) to adapt the pre-trained GNNs to the domain-specific data with labels \(\mathcal{D}_{ft}\). In this paper, we mainly investigate the improvement from the perspective of automatically designing a suitable fine-tuning strategy for the given data \(\mathcal{D}_{ft}\). Thus, to validate the effectiveness of the proposed S2PGNN, we need to answer the following questions:

* **Q1**: Can S2PGNN be built on the top of different GNN backbone and pre-training methods and consistently improve their performance? (see Sec. IV-B and Sec. IV-E)
* **Q2**: On the same configuration of GNN backbone model and pre-training strategy, can S2PGNN be more effective than other GNN fine-tuning strategies? (see Sec. IV-C)
* **Q3**: As discussed in Sec. II-B2, there are several classic fine-tuning strategies in other domains that are not included in our search space. Will these models perform well on the GNN area? (see Sec. IV-C)
* **Q4**: What are the effect of each design dimensions in S2PGNN? (see Sec. IV-D)
* **Q5**: Is there a risk that the fine-tuning search method will consume more computing resources to achieve improved performance? (see Sec. IV-F)

### _Experimental Settings_

S2PGNN 1 is implemented based on PyTorch [74] and PyTorch Geometric [75] libraries. All experiments are conducted with one single NVIDIA Tesla V100 GPU.

Footnote 1: Code and data are available at [https://anonymous.4open.science/rcode_icde2024-A9CB](https://anonymous.4open.science/rcode_icde2024-A9CB).

#### Iv-A1 GNN backbone models

For GNN backbone architectures, we mainly adopt classic and promising GNNs in recent years, including (5-layer) GCN [7], SAGE [8], Graph Isomorphism Network (GIN) [10], and GAT [9] (see more details in Sec. II-A). But due to the limited space, the experiments in sections Sec. IV-B, Sec. IV-C, Sec. IV-D, and Sec. IV-F are conducted on GIN. And we report the performance on other backbone models in Sec. IV-E.

#### Iv-A2 GNN pre-training methods and datasets \(\mathcal{D}_{ssl}\)

To demonstrate the effectiveness of S2PGNN, we implement S2PGNN on top of 10 well-known and publicly available pre-training methods, including Infomax [58], EdgePred [8], ContextPred [20], AttrMasking [20], GraphCL [21], GraphLoG [22], MGSSL [23], SimGRACE [25], GraphMAE [26], and Mole-BERT [30]. As summarized in Tab. V, they cover a wide spectrum of various SSL strategies \(\mathcal{L}_{ssl}(\cdot)\) (see more details in Sec. II-B1). Due to the space limit, the fine-tuning experiments in Sec. IV-B are conducted on top of all 10 pre-trained models, and experiments in Sec. IV-C, Sec. IV-D, Sec. IV-F, and Sec. IV-F are built on top of the pioneering pre-training work ContextPred [20].

\begin{table}
\begin{tabular}{l|l|l} \hline \hline
**Method** & **SSL Strategy \(\mathcal{L}_{ssl}(\cdot)\)** & **SSL Data \(\mathcal{D}_{ssl}\)** \\ \hline \hline Infomax & Contrastive Learning (CL) & ZINC15 (2M) \\ \hline EdgePred & Autoencoding (AE) & ZINC15 (2M) \\ \hline ContextPred & Context Prediction (CP) & ZINC15 (2M) \\ \hline AttMasking & Masked Component Modeling (MCM) & ZINC15 (2M) \\ \hline GraphCL & Contrastive Learning (CL) & ZINC15 (2M) \\ \hline GraphLoG & Contrastive Learning (CL) & ZINC15 (2M) \\ \hline MGSSL & Autoregressive Modeling (AM) & ZINC15 (150K) \\ \hline SimGRACE & Contrastive Learning (CL) & ZINC15 (2M) \\ \hline GraphMAE & Autoencoding (AE) & ZINC15 (2M) \\ \hline Mole-BERT & Masked Component Modeling (MCM) & ZINC15 (2M) \\ \hline \hline \end{tabular}
\end{table} TABLE V: Summary of base GNN pre-training methods.

\begin{table}
\begin{tabular}{l|l l l l l} \hline \hline
**Dataset** & **\#Molecules** & **\#Tasks** & **Task Type** & **Metric** & **Domain** \\ \hline \hline BBBP & 2039 & 1 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Pharmacology \\ \hline Tox21 & 7831 & 12 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Pharmacology \\ \hline ToxCast & 8575 & 617 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Pharmacology \\ \hline SIDER & 1427 & 27 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Pharmacology \\ \hline ClinTox & 1478 & 2 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Pharmacology \\ \hline BACE & 1513 & 1 & Classification & ROC-AUC (\%) (\(\uparrow\)) & Biophysics \\ \hline ESOL & 1128 & 1 & Regression & RMSE (\(\downarrow\)) & Physical Chemistry \\ \hline Lipophilicity (Lipo) & 4200 & 1 & Regression & RMSE (\(\downarrow\)) & Physical Chemistry \\ \hline \hline \end{tabular}
\end{table} TABLE IV: Summary of downstream GNN fine-tuning datasets \(\mathcal{D}_{ft}\).
In this paper, we follow the literature to adopt ZINC15 (150K) for MGSSL [23], which contains 150K unlabeled molecules collected from the ZINC15 database [76]. We use the larger version ZINC15 with 2 million molecules for methods other than MGSSL [23].

#### V-A3 GNN fine-tuning tasks and datasets \(\mathcal{D}_{ft}\)

In this paper, we mainly conduct graph-level experiments on the downstream molecular property prediction (MPP), which is an important task for a variety of domains (e.g., physics, chemistry, and materials science). In MPP, a molecule is represented as a graph, where nodes and edges denote atoms and bonds, and labels are related to molecular toxicity or enzyme binding properties. The aim of MPP is to predict the properties for unlabeled molecule. We employ ROC-AUC and RMSE for evaluating the classifier and regressive MPP tasks, respectively. For datasets with multiple prediction tasks (see Tab. IV), we report average results over all their tasks.

As shown in Tab. VI, we follow the related literature [20, 77] to adopt 8 popular benchmark datasets: BBBP [78], Tox21 [79], ToxCast [80], SIDER [81], ClinTox [82], BACE [83], ESOL [84], and Lipophilicity (Lipo) [85] that are provided in MoleculeNet [86]. The selected datasets are from several domains, including pharmacology, biophysics, and physical chemistry. Among them, ESOL and Lipo are used for graph regression, while the rest are for graph classification. As for data split, we utilize _scaffold-split_[37] to split molecular datasets according to their substructures as suggested by [20, 77]. It provides a more challenging yet more realistic split to deal with real-world applications (where out-of-distribution predictions are often required) compared with random-split.

#### V-A4 Implementation details

For GNN pre-training methods, we employ their officially released pre-trained models to conduct the next stage fine-tuning. Readers may refer to the original papers for their detailed pre-training settings.

S2PGNN and other fine-tuning baselines are evaluated with the same protocol for rigorously fair comparisons. We follow pioneering literature [20] to setup fine-tuning configurations. Specifically, we leverage the simple linear classifier as downstream prediction head. To fine-tune the whole model, we use Adam optimizer with a learning rate of 1e-3. We set batch size as 32 and dropout rate as 50%. We perform fine-tuning for 100 epochs with early stopping based on validation set. The ratio to split train/validation/test set is 80%/10%/10%. We run all experiments for 10 times with different random seeds and report the mean results (standard deviations).

### _The implementation S2PGNN with Pre-trained GNNs and comparison with standard fine-tuning_

Despite the blooming development of GNN pre-training methods, the specific strategies for GNN fine-tuning (Eq. (14)) is still scarce. As discussed in Sec. II-B2, standard fine-tuning is probably still the most prevalent strategy to leverage pre-trained GNNs among existing literatures. Thus, we first implement and compare the gains of S2PGNN over standard fine-tuning on top of 10 GNN pre-training methods.

The main results on 8 downstream datasets are reported in Tab. VI. When equipped with S2PGNN, the pre-trained GNNs consistently demonstrate better fine-tuning performances on the graph classification and graph regression tasks than the standard fine-tuning. The gains are consistent on different datasets with diverse characteristics (see Tab. IV). Moreover, we also observe the superiority of S2PGNN is agnostic to GNN pre-training configurations, such as SSL strategy, SSL data, and attained pre-trained models (see Sec. IV-A2 and Tab. V). To summarize, observations from Tab. VI validate that

\begin{table}
\begin{tabular}{c|c c c c c c c c|c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%)) \(\uparrow\)**} & \multicolumn{3}{c|}{**Regression (RMSE) \(\downarrow\)**} & **Avg.** \\ \hline
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinBox & BACE & ESOL & Lipo & **Gain** \\ \hline Infomax [58] & 68.4 \(\pm\) 1.7 & 75.6 \(\pm\) 0.5 & 62.5 \(\pm\) 0.8 & 58.3 \(\pm\) 0.7 & 71.3 \(\pm\) 2.6 & 75.5 \(\pm\) 2.3 & 2.6 \(\pm\) 0.1 & 1.0 \(\pm\) 0.1 & \multirow{2}{*}{\(+17.7\%\)} \\ Infomax + S2PGNN & **69.9 \(\pm\) 1.4** & **76.7 \(\pm\) 0.5** & **65.8 \(\pm\) 0.4** & **62.3 \(\pm\) 1.3** & **74.8 \(\pm\) 3.8** & **82.3 \(\pm\) 1.3** & **1.5 \(\pm\) 0.3** & **0.8 \(\pm\) 0.0** & \\ \hline EdgePred [8] & 67.2 \(\pm\) 2.9 & 75.8 \(\pm\) 0.9 & 63.9 \(\pm\) 0.4 & 60.5 \(\pm\) 0.8 & 65.7 \(\pm\) 4.1 & 79.4 \(\pm\) 1.4 & 2.8 \(\pm\) 0.0 & 1.0 \(\pm\) 0.1 & \\ EdgePred + S2PGNN & **69.1 \(\pm\) 0.8** & **77.1 \(\pm\) 0.8** & **66.2 \(\pm\) 0.3** & **62.3 \(\pm\) 0.5** & **71.9 \(\pm\) 1.1** & **82.2 \(\pm\) 1.1** & **1.7 \(\pm\) 0.2** & **0.9 \(\pm\) 0.0** & \\ \hline ContextPred [20] & 69.0 \(\pm\) 0.9 & 76.0 \(\pm\) 0.4 & 63.5 \(\pm\) 0.4 & 60.7 \(\pm\) 0.6 & 69.7 \(\pm\) 1.4 & 80.6 \(\pm\) 0.8 & 2.8 \(\pm\) 0.4 & 1.1 \(\pm\) 0.0 & \multirow{2}{*}{\(+15.1\%\)} \\ ContextPred + S2PGNN & **70.9 \(\pm\) 1.3** & **76.3 \(\pm\) 0.4** & **67.0 \(\pm\) 0.5** & **62.8 \(\pm\) 0.3** & **75.9 \(\pm\) 2.2** & **82.6 \(\pm\) 0.7** & **1.7 \(\pm\) 0.2** & **0.9 \(\pm\) 0.0** & \\ \hline AttrMasking [20] & 65.1 \(\pm\) 2.3 & 76.7 \(\pm\) 0.6 & 64.4 \(\pm\) 0.3 & 60.6 \(\pm\) 0.9 & 72.0 \(\pm\) 4.2 & 79.5 \(\pm\) 0.7 & 2.8 \(\pm\) 0.1 & 1.1 \(\pm\) 0.0 & \multirow{2}{*}{\(+15.6\%\)} \\ AttrMasking + S2PGNN & **71.9 \(\pm\) 1.1** & **77.3 \(\pm\) 0.4** & **66.8 \(\pm\) 0.5** & **62.9 \(\pm\) 0.4** & **74.8 \(\pm\) 3.1** & **82.7 \(\pm\) 0.8** & **1.7 \(\pm\) 0.1** & **0.9 \(\pm\) 0.0** & \\ \hline GraphCL [21] & 68.3 \(\pm\) 1.6 & 74.1 \(\pm\) 0.8 & 62.6 \(\pm\) 0.5 & 59.7 \(\pm\) 1.1 & 71.4 \(\pm\) 6.2 & 75.8 \(\pm\) 2.7 & 2.5 \(\pm\) 0.1 & 1.0 \(\pm\) 0.0 & \multirow{2}{*}{\(+10.1\%\)} \\ GraphCL + S2PGNN & **70.8 \(\pm\) 1.1** & **76.8 \(\pm\) 0.5** & **66.6 \(\pm\) 0.3** & **62.4 \(\pm\) 1.2** & **75.2 \(\pm\) 3.4** & **82.6 \(\pm\) 2.3** & **1.9 \(\pm\) 0.1** & **0.9 \(\pm\) 0.0** & \\ \hline GraphCLoG [22] & 66.5 \(\pm\) 2.3 & 75.3 \(\pm\) 0.3 & 63.3 \(\pm\) 0.5 & 57.8 \(\pm\) 1.2 & 69.6 \(\pm\) 5.8 & 80.1 \(\pm\) 2.4 & 2.5 \(\pm\) 0.1 & 1.0 \(\pm\) 0.0 & \multirow{2}{*}{\(+9.1\%\)} \\ GraphLod + S2PGNN & **69.9 \(\pm\) 1.5** & **76.8 \(\pm\) 0.3** & **66.1 \(\pm\) 0.2** & **62.0 \(\pm\) 1.0** & **75.8 \(\pm\) 2.1** & **85.1 \(\pm\) 1.3** & **2.0 \(\pm\) 0.1** & **0.9 \(\pm\) 0.0** & \\ \hline MGSSL [23] & 67.4 \(\pm\) 1.8 & 75.0 \(\pm\) 0.6 & 63.1 \(\pm\) 0.5 & 58.0 \(\pm\) 1.2 & 68.1 \(\pm\) 5.4 & 81.2 \(\pm\) 3.3 & 2.4 \(\pm\) 0.1 & 1.0 \(\pm\) 0.0 & \multirow{2}{*}{\(+9.6\%\)} \\ MGSSL + S2PGNN & **69.4 \(\pm\) 1.8** & **77.0 \(\pm\) 0.6** & **66.3 \(\pm\) 0.4** & **62.8 \(\pm\) 1.2** & **76.7 \(\pm\) 2.4** & **85.2 \(\pm\) 1.2** & **1.9 \(\pm\) 0.2** & **0.9 \(\pm\) 0.0** & \\ \hline SimGRACE [25] & 67.9 \(\pm\) 0.6 & 73.9 \(\pm\
S2PGNN fine-tuning provides a promisingly better solution than the standard strategy to achieve the better utilization of various pre-trained GNNs.

### _Comparison with other fine-tuning strategies_

As discussed in Sec. II-B2, we first investigate the standard fine-tuning and regularized fine-tuning strategies since they have already been adapted to the GNN area. Then, we explore more fine-tuning methods on other domains (e.g., computer vision) but have not been discussed in the GNN community.

#### Iv-C1 GNN Fine-tuning Baselines

Apart from the standard strategy (see main results in Sec. IV-B), we also compare S2PGNN with another GNN fine-tuning work GTOT-Tuning [39], which belongs to regularized fine-tuning. We exclude AUX-TS [32] and WordReg [38] for baseline comparisons because: AUX-TS targets on different downstream tasks, and WordReg uses different data split with [20] but its code is not publicly available for reproducing results. Besides, we additionally report the results of several regularized fine-tuning baselines tailored from the computer vision domain (that are originally designed to fine-tune CNNs), including \(L^{2}\)-_SP_[50], DELTA [51], BSS [52], and StochNorm [53]. \(L^{2}\)-_SP_ regularizes on model parameters to induce the fine-tuned weights to be close to pre-trained weights. DELTA imposes regularization on representations via the attention mechanism. BSS penalizes small eigenvalues of learned representations to suppress untransferable components. StochNorm regularizes on the encoder architecture in a dropout-like way. Please refer to Sec. II-B2 and Tab. II for more technical discussions.

The comparisons on 6 classic MPP datasets have been summarized in Tab. VII. As shown in Tab. VII, we first observe the non-negligible improvement brought by S2PGNN compared with all baselines. Among baselines, regularized techniques (DELTA, BSS) from computer vision domain sometimes yield slightly better results than the standard fine-tuning strategy. However, in several cases (\(L^{2}\)-_SP_, StochNorm), the performance gains are not significant. This is probably due to their ignorance of characteristics in graph data (e.g., graph topology) and specific requirements in fine-tuning GNNs. By taking the graph topology into consideration, the method GTOT-Tuning, which is designed specifically to fine-tune GNNs, demonstrates higher performances than regularized variants extended from other domains. However, even the most competitive baseline GTOT-Tuning is still inferior to S2PGNN in 5 out of 6 datasets, indicating that the design dimensions proposed in S2PGNN (see Sec. III-B and Tab. II) may be indispensable and crucial designs towards more effective GNN fine-tuning. Furthermore, we note that various regularized strategies, such as GTOT-Tuning, is orthogonal to the proposed S2PGNN. Therefore, it may be promising to combine S2PGNN with other advanced regularized methods, i.e., combine an additional regularization term in S2PGNN's loss function Eq. (31) and (32) to further boost its performances, which we leave as future works.

#### Iv-C2 Exploration of Other Fine-tuning Strategies

As discussed in Sec. II-B2, fine-tuning has been well explored in the domains beyond GNNs. Therefore, we try to investigate the performance of more fine-tuning strategies that are promising in other domains. However, we conclude that they may not be suitable for fine-tuning GNNs, thereby we discard them from our search space. Overall, we explore following fine-tuning strategies:

* Feature Extractor (FE): FE [87] proposes to reuse pre-trained model parameters to achieve better knowledge preservation. Once the pre-training is finished, all pre-trained layers are frozen and the pre-trained model works as a pure feature extractor for downstream data. Only a small amount of additional parameters in the task-specific prediction head are further tuned to perform downstream predictions.
* Last-\(k\) Tuning (LKT): With LKT [88], only parameters in last-\(k\) layers of the pre-trained models are further tuned, while the other initial layers are frozen and keep unchanged. LKT resides between ST and FE, and is popular in the computer version area [88]. In our experiments, we consider \(k\in\{1,2,3\}\), which means we use around \(20\%\sim 60\%\) tunable parameters of the original model (where \(k=5\))
* Adapter-Tuning (AT): AT [34] proposes to fine-tune only a small number of extra model parameters to attain the competitive performance, which is promising to achieve faster tuning and alleviate the over-fitting issue in natural language process and computer vision areas. More specifically, it adds small and task-specific neural network modules, called adapters, to pre-trained models. These adapters involve feature transformation \(\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}\rightarrow\mathbb{R}^{d}\) via the bottleneck architecture, where \(m\ll d\) is to ensure parameter-efficient. Adapters are inserted between pre-trained layers and trained on the specific task, while the pre-trained layers remain

\begin{table}
\begin{tabular}{c|c c c c c c|c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%))**} & \multirow{2}{*}{**Avg.**} \\ \cline{2-2} \cline{5-7}
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinTox & BACE \\ \hline Vanilla tuning & 68.0 \(\pm\) 2.0 & 75.7 \(\pm\) 0.7 & 63.9 \(\pm\) 0.6 & 60.9 \(\pm\) 0.6 & 65.9 \(\pm\) 3.8 & 79.6 \(\pm\) 1.2 & 69.0 \\ \hline \(L^{2}\)-_SP_[50] & 68.2 \(\pm\) 0.7 & 73.6 \(\pm\) 0.8 & 62.4 \(\pm\) 0.3 & 61.1 \(\pm\) 0.7 & 68.1 \(\pm\) 3.7 & 82.2 \(\pm\) 2.4 & 69.3 \\ DELTA [51] & 67.8 \(\pm\) 0.8 & 75.2 \(\pm\) 0.5 & 63.3 \(\pm\) 0.5 & 62.2 \(\pm\) 0.4 & 73.4 \(\pm\) 3.0 & 81.8 \(\pm\) 1.1 & 70.6 \\ BSS [52] & 68.1 \(\pm\) 1.4 & 75.9 \(\pm\) 0.8 & 63.9 \(\pm\) 0.4 & 60.9 \(\pm\) 0.8 & 70.9 \(\pm\) 5.1 & 82.4 \(\pm\) 1.8 & 70.4 \\ StochNorm [53] & 69.3 \(\pm\) 1.6 & 74.9 \(\pm\) 0.6 & 63.4 \(\pm\) 0.5 & 61.0 \(\pm\) 1.1 & 65.5 \(\pm\) 4.2 & 80.5 \(\pm\) 2.7 & 69.1 \\ \hline GTOT-tuning [39] & 70.0 \(\pm\) 1.7 & 75.2 \(\pm\) 0.9 & 63.0 \(\pm\) 0.5 & **63.1 \(\pm\) 0.6** & 71.8 \(\pm\) 5.4 & 82.6 \(\pm\) 2.0 & 71.0 \\ \hline S2PGNN & **70.9 \(\pm\) 1.3** & **76.3 \(\pm\) 0.4** & **67.0 \(\pm\) 0.5** & 62.8 \(\pm\) 0.3 & **75.9 \(\pm\) 2.2** & **82.6 \(\pm\) 0.7** & **72.6** \\ \hline \hline \end{tabular}
\end{table} TABLE VII: The performance comparison between proposed S2PGNN fine-tuning and other fine-tuning strategies with fixed ContextPred pre-training objective and GIN backbone architecture.
fixed and unchanged to preserve the knowledge learned during pre-training. In our empirical explorations, we tailor the adapter design in [34] and consider adapter size \(m\in\{2,4,8\}\) to use only around \(1.3\%\sim 5.2\%\) tunable parameters of the original model (where \(d=300\)).

The results on classic MPP tasks are summarized in Tab. VIII. Clearly, S2PGNN demonstrates consistent superior results than other approaches. Directly using the pre-trained GNN as pure feature extractor (equivalent to \(k=0\)) can reduce the total number of tunable parameters during fine-tuning, but it leads to the severe performance degradation on all 6 datasets. This indicates that the fixed model may impede sufficient adaption when dealing with various downstream datasets. By gradually increasing the tunable layers \(k\) (\(1\to 3\)), the performance drop caused by insufficient adaption is mitigated. However, by tuning only partial model parameters, Last-\(k\) still yield inferior results than the standard strategy (equivalent to \(k=5\)). Adapter method, although has demonstrated competitive fine-tuning capacity with the standard strategy when fine-tuning language models in natural language processing domain, fails to yield satisfactory results when fine-tuning GNNs for graph data.

To summarize, by investigating more fine-tuning strategies that are promising in other domains, we conclude that they may not be suitable for fine-tuning GNNs, which is probably due to the divergence among different domains. Instead, specific and innovative designs to fine-tune GNNs in graph domain is much more demanded. The performance comparison in Tab. VIII also indicates that the design dimensions in S2PGNN's search space may be more validated for searching the suitable fine-tuning strategy in pre-trained GNNs.

### _Ablation study on S2PGNN's design dimensions_

To investigate S2PGNN's important design dimensions (see Sec. III-B) regarding GNN fine-tuning strategy: identity augmentation, multi-scale fusion, and adaptive graph-level readout, we further propose S2PGNN variants with degraded space and conduct ablation studies: S2PGNN-\(\backslash id\_aug\) disables identity augmentation when aggregating messages from neighbors; S2PGNN-\(\backslash fuse\) discards the multi-scale fusion and directly uses the last-layer output as learned node representations as most existing works does; S2PGNN-\(\backslash read\) uses the simple and fixed mean pooling as [20] and follow-up works.

The man results of S2PGNN's variants are shown in Tab. IX. Significant performances drop are observed in each S2PGNN variant with degraded space, which provide empirical vali

Fig. 4: Illustration to other fine-tuning strategies (refer to Fig. 3 for the legend).

\begin{table}
\begin{tabular}{c|c c c c c c|c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%)) \(\uparrow\)**} & **Avg.** \\ \hline
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinTox & BACE \\ \hline Vanilla tuning & 68.0 \(\pm\) 2.0 & 75.7 \(\pm\) 0.7 & 63.9 \(\pm\) 0.6 & 60.9 \(\pm\) 0.6 & 65.9 \(\pm\) 3.8 & 79.6 \(\pm\) 1.2 & 69.0 \\ \hline Feature extractor & 58.9 \(\pm\) 0.4 & 68.7 \(\pm\) 0.4 & 59.3 \(\pm\) 0.2 & 59.9 \(\pm\) 0.3 & 40.5 \(\pm\) 2.7 & 61.6 \(\pm\) 4.5 & 58.2 \\ \hline Last-\(k\) (\(k=3\)) & 68.1 \(\pm\) 0.9 & 75.1 \(\pm\) 0.4 & 64.0 \(\pm\) 0.5 & 60.7 \(\pm\) 0.5 & 63.9 \(\pm\) 4.5 & 79.1 \(\pm\) 1.2 & 68.5 \\ Last-\(k\) (\(k=2\)) & 65.3 \(\pm\) 1.0 & 74.5 \(\pm\) 0.5 & 63.0 \(\pm\) 0.7 & 61.6 \(\pm\) 0.6 & 64.0 \(\pm\) 3.5 & 80.6 \(\pm\) 1.2 & 68.2 \\ Last-\(k\) (\(k=1\)) & 64.6 \(\pm\) 1.3 & 73.0 \(\pm\) 0.5 & 61.4 \(\pm\) 0.5 & 60.8 \(\pm\) 0.5 & 66.1 \(\pm\) 2.4 & 76.6 \(\pm\) 0.8 & 67.1 \\ \hline Adapter (\(m=2\)) & 61.2 \(\pm\) 0.4 & 71.4 \(\pm\) 0.3 & 60.6 \(\pm\) 0.1 & 59.6 \(\pm\) 0.3 & 45.3 \(\pm\) 3.1 & 71.1 \(\pm\) 0.9 & 61.6 \\ Adapter (\(m=4\)) & 62.5 \(\pm\) 0.7 & 71.7 \(\pm\) 0.3 & 60.6 \(\pm\) 0.2 & 59.6 \(\pm\) 0.3 & 47.4 \(\pm\) 2.2 & 74.4 \(\pm\) 1.8 & 62.7 \\ Adapter (\(m=8\)) & 63.9 \(\pm\) 0.7 & 71.8 \(\pm\) 0.3 & 60.5 \(\pm\) 0.4 & 59.9 \(\pm\) 0.3 & 50.0 \(\pm\) 1.0 & 76.8 \(\pm\) 0.9 & 63.8 \\ \hline S2PGNN & **70.9 \(\pm\) 1.3** & **76.3 \(\pm\) 0.4** & **67.0 \(\pm\) 0.5** & **62.8 \(\pm\) 0.3** & **75.9 \(\pm\) 2.2** & **82.6 \(\pm\) 0.7** & **72.6** \\ \hline \hline \end{tabular}
\end{table} TABLE VIII: The performance of more fine-tuning strategies that are excluded from S2PGNNX’s design space with fixed ContextPred pre-training objective and GIN backbone architecture. Second best results are marked with underline.
dation that the proposed design dimensions in S2PGNN are key factors that affect GNN fine-tuning results and should be incorporated during fine-tuning to achieve the optimal downstream results.

### _Effect of GNN backbone architectures_

Recent works [89] have identified that GNN backbone architectures also play a crucial role in GNN pre-training. Therefore, here we further provide additional results of S2PGNN when built on top of other classic GNN backbone architectures other than the GIN. Tab. X summarizes the performance of S2PGNN based on several pre-trained GNNs, including GCN, SAGE, and GAT models via the SSL strategy ContextPred [20]. We observe that pre-trained GNNs with all these backbone architectures can benefit from S2PGNN fine-tuning and achieve better performances than the vanilla strategy. This verifies that S2PGNN is agnostic to the base GNN architectures and is capable to achieve the consistent improvement.

### _Efficiency_

Apart from the effectiveness results provided in previous subsections, here we further report the concrete running time comparisons among several fine-tuning baselines. As shown in Tab. XI, we observe that the running time of S2PGNN is comparable with fine-tuning baselines, which eliminates the concern that the fine-tuning search method in S2PGNN may consume more computing resources to achieve improved performance.

## V Conclusion

In this paper, to bridge the missing gap between better fine-tuning strategies with pre-trained GNNs, we propose to search to fine-tune pre-trained graph neural networks for graph classification, named S2PGNN. To improve the utilization of pre-trained GNNs, S2PGNN adaptively designs a suitable fine-tuning strategy for the given pre-trained GNN and downstream dataset. S2PGNN presents a novel search space for fine-tuning strategies, which identifies key factors that affect GNN fine-tuning results. S2PGNN is model-agnostic and can be plugged into existing GNN backbone models and pre-trained GNNs. Empirical studies demonstrate that S2PGNN can consistently improve 10 classic pre-trained GNNs and achieve better performance than other fine-tuning works. Therefore, we expect S2PGNN may shed light on potential future directions towards better GNN fine-tuning innovation.

\begin{table}
\begin{tabular}{c|c c c c c c|c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%)) \(\uparrow\)**} & \multicolumn{2}{c|}{**Regression (RMSE) \(\downarrow\)**} & **Avg.** \\ \hline
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinTox & BACE & ESOL & Lipo & **Gain** \\ \hline ContextPred (GCN) & 64.6 \(\pm\) 2.2 & 73.0 \(\pm\) 0.5 & 61.9 \(\pm\) 1.1 & 56.8 \(\pm\) 0.6 & 69.0 \(\pm\) 1.3 & 79.9 \(\pm\) 1.7 & 2.4 \(\pm\) 0.1 & **1.0 \(\pm\) 0.0** & \(+\)4.6\(\%\) \\ ContextPred (GCN) + S2PGNN & **68.3 \(\pm\) 1.0** & **75.7 \(\pm\) 0.5** & **66.5 \(\pm\) 0.3** & **62.3 \(\pm\) 0.4** & **71.6 \(\pm\) 1.3** & **81.5 \(\pm\) 0.5** & **2.3 \(\pm\) 0.1** & **1.0 \(\pm\) 0.0** & \(+\)4.6\(\%\) \\ \hline ContextPred (SAGE) & 6.0 \(\pm\) 3.0 & 74.7 \(\pm\) 0.5 & 63.4 \(\pm\) 0.2 & **62.0 \(\pm\) 0.6** & 61.1 \(\pm\) 3.1 & 78.8 \(\pm\) 1.4 & 2.5 \(\pm\) 0.1 & **1.0 \(\pm\) 0.0** & \(+\)6.0\(\%\) \\ ContextPred (SAGE) + S2PGNN & **69.0 \(\pm\) 1.4** & **75.1 \(\pm\) 0.4** & **66.4 \(\pm\) 0.6** & 61.6 \(\pm\) 0.4 & **67.1 \(\pm\) 1.6** & **79.1 \(\pm\) 0.7** & **2.0 \(\pm\) 0.2** & **1.0 \(\pm\) 0.0** & \(+\)6.0\(\%\) \\ \hline ContextPred (GAT) & 64.9 \(\pm\) 1.2 & 69.6 \(\pm\) 0.7 & 59.5 \(\pm\) 0.8 & 52.5 \(\pm\) 3.4 & 58.2 \(\pm\) 6.9 & 60.5 \(\pm\) 3.5 & 3.2 \(\pm\) 0.1 & 1.1 \(\pm\) 0.0 & \(+\)19.7\(\%\) \\ ContextPred (GAT) + S2PGNN & **69.6 \(\pm\) 0.9** & **75.0 \(\pm\) 0.4** & **65.3 \(\pm\) 0.3** & **61.8 \(\pm\) 1.1** & **66.7 \(\pm\) 3.2** & **80.4 \(\pm\) 1.3** & **1.8 \(\pm\) 0.1** & **1.0 \(\pm\) 0.0** & \(+\)19.7\(\%\) \\ \hline \hline \end{tabular}
\end{table} TABLE X: The performance comparison between proposed S2PGNN fine-tuning and standard fine-tuning strategies with fixed ContextPred pre-training objective and other popular GNN backbone architectures.

\begin{table}
\begin{tabular}{c|c c c c c c|c c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%)) \(\uparrow\)**} & \multicolumn{2}{c|}{**Regression (RMSE) \(\downarrow\)**} & **Avg.** \\ \hline
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinTox & BACE & ESOL & Lipo & **Drop** \\ \hline S2PGNN\(\shortdownarrow\)_aug_ & 69.5 \(\pm\) 2.0 & 75.9 \(\pm\) 0.3 & 66.3 \(\pm\) 0.4 & 61.3 \(\pm\) 0.7 & 69.4 \(\pm\) 6.3 & 79.5 \(\pm\) 1.6 & 2.0 \(\pm\) 0.3 & 0.9 \(\pm\) 0.0 & \(-\)5.2\(\%\) \\ S2PGNN\(\shortuparrow\)_base_ & 69.0 \(\pm\) 1.5 & 75.7 \(\pm\) 0.6 & 65.7 \(\pm\) 0.4 & 61.6 \(\pm\) 0.7 & 61.6 \(\pm\) 4.4 & 82.0 \(\pm\) 1.0 & 2.5 \(\pm\) 0.1 & 1.0 \(\pm\) 0.0 & \(-\)12.1\(\%\) \\ S2PGNN\(\shortmid\)_read_ & 70.3 \(\pm\) 1.6 & 75.2 \(\pm\) 0.3 & 63.9 \(\pm\) 0.3 & 62.2 \(\pm\) 0.7 & 73.7 \(\pm\) 4.4 & 80.3 \(\pm\) 1.7 & 2.7 \(\pm\) 0.0 & 1.0 \(\pm\) 0.0 & \(-\)12.3\(\%\) \\ S2PGNN & **70.9 \(\pm\) 1.3** & **76.3 \(\pm\) 0.4** & **67.0 \(\pm\) 0.5** & **62.8 \(\pm\) 0.3** & **75.9 \(\pm\) 2.2** & **82.6 \(\pm\) 0.7** & **1.7 \(\pm\) 0.2** & **0.9 \(\pm\) 0.0** & - \\ \hline \hline \end{tabular}
\end{table} TABLE IX: The performance comparison among S2PGNN’s variants with degraded space.

\begin{table}
\begin{tabular}{c|c c c c c|c} \hline \hline  & \multicolumn{6}{c|}{**Classification (ROC-AUC (\%)) \(\uparrow\)**} & \multicolumn{2}{c|}{**Regression (RMSE) \(\downarrow\)**} & **Avg.** \\ \hline
**Dataset** & BBBP & Tox21 & ToxCast & SIDER & ClinTox & BACE & ESOL & Lipo & **Gain** \\ \hline
**ContextPred (GCN)** & 64.6 \(\pm\) 2.2 & 73.0 \(\pm\) 0.5 & 61.9 \(\pm\) 1.1 & 56.8 \(\pm\) 0.6 & 69.0 \(\pm\) 1.3 & 79.9 \(\pm\) 1.7 & 2.4 \(\pm\) 0.1 & **1.0 \(\pm\) 0.0** & \(+\)4.0\(\%\) \\ ContextPred (GCN) + S2PGNN & **68.3 \(\pm\) 1.0** & **75.7 \(\pm\) 0.8** & **66.5 \(\pm\) 0.3** & **62.3 \(\pm\) 0.4** & **71.6 \(\pm\) 1.3** & **81.5 \(\pm\) 0.5** & **2.3 \(\pm\) 0.1** & **1.0 \(\pm\) 0.0** & \(+\)4.6\(\%\) \\ \hline ContextPred (SAGE) & 6.0 \(\pm\) 3.0 & 74.7 \(\pm\) 0.5 & 63.4 \(\pm\) 0.2 & **62.0 \(\pm\) 0.6** & 61.1 \(\pm\) 3.1 & 78.8 \(\pm\) 1.4 & 2.5 \(\pm\) 0.1 & **1.0 \(\pm\) 0.0** & \(+\)6.0\(\%\) \\ ContextPred (SAGE) + S2PGNN & **69.0 \(\pm\) 1.4** & **75.1 \(\pm\) 0.4** & **66.4 \(\pm\) 0.6** & 61.6 \(\pm\) 0.4 & **67.1 \(\pm\) 1.6** & **79.1 \(\pm\) 0.7** & **2.0 \(\pm\) 0.2** & **1.0 \(\pm\) 0.0** & \(+\)6.0\(\%\) \\ \hline Context