# Subgraph-level Universal Prompt Tuning

Junhyun Lee<EMAIL_ADDRESS>Korea UniversitySeoulSouth Korea02841 ,
Wooseong Yang<EMAIL_ADDRESS>University of Illinois at
ChicagoChicagoIllinoisUSA60607 and  Jaewoo Kang<EMAIL_ADDRESS>Korea
UniversitySeoulSouth Korea02841

(2018)

###### Abstract.

In the evolving landscape of machine learning, the adaptation of pre-trained
models through prompt tuning has become increasingly prominent. This trend is
particularly observable in the graph domain, where diverse pre-training
strategies present unique challenges in developing effective prompt-based
tuning methods for graph neural networks. Previous approaches have been
limited, focusing on specialized prompting functions tailored to models with
edge prediction pre-training tasks. These methods, however, suffer from a lack
of generalizability across different pre-training strategies. Recently, a
simple prompt tuning method has been designed for any pre-training strategy,
functioning within the input graph’s feature space. This allows it to
theoretically emulate any type of prompting function, thereby significantly
increasing its versatility for a range of downstream applications.
Nevertheless, the capacity of such simple prompts to fully grasp the complex
contexts found in graphs remains an open question, necessitating further
investigation. Addressing this challenge, our work introduces the Subgraph-
level Universal Prompt Tuning (SUPT) approach, focusing on the detailed
context within subgraphs. In SUPT, prompt features are assigned at the
subgraph-level, preserving the method’s universal capability. This requires
extremely fewer tuning parameters than fine-tuning-based methods,
outperforming them in 42 out of 45 full-shot scenario experiments with an
average improvement of over 2.5%. In few-shot scenarios, it excels in 41 out
of 45 experiments, achieving an average performance increase of more than
6.6%.

Graph Prompt Tuning, Universal Graph Prompt, Graph Pooling

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Computing methodologies Neural networks††ccs: Computing methodologies Learning
latent representations††ccs: Computing methodologies Transfer learning

##  1\. Introduction

Pre-trained models have significantly revolutionized the fields of Natural
Language Processing (NLP) and Computer Vision, primarily addressing two
critical challenges: the scarcity of labeled data and the need for strong out-
of-distribution generalization ability. In NLP, models like BERT and GPT have
demonstrated remarkable proficiency in understanding and generating human
language by leveraging vast, unlabeled datasets (Devlin et al., 2018; Brown et
al., 2020). Similarly, in Computer Vision, models like ResNet and EfficientNet
have shown exceptional performance in image recognition tasks by training on
large-scale, diverse image datasets (He et al., 2016; Tan and Le, 2019).
Inspired by these successes, the concept of pre-trained models has recently
gained traction in the domain of graph representation learning (Xia et al.,
2022c). The adoption of pre-trained models in this area is particularly
promising, considering the complex and often sparse nature of graph data. By
pre-training on extensive graph datasets, these models can capture intricate
patterns and relationships inherent in graph structures, thereby enhancing
their performance on various downstream tasks in the graph domain.

Figure 1.  Illustration of the concepts: (a) pixel-level visual prompts (Bahng
et al., 2022; Xing et al., 2023; Chen et al., 2023; Liu et al., 2023a; Li et
al., 2023; Oh et al., 2023; Tsao et al., 2024) and (b) node-level graph
prompts (Fang et al., 2023).

Figure 2.  Illustration of Different Tuning Approaches: Comparing Fine-Tuning,
GPF, GPF-Plus, and SUPT. GPF assigns a uniform prompt feature across all
nodes, irrespective of their contextual disparities. GPF-Plus advances by
allocating prompt features according to node type with category-specific
uniformity but not individual graph contextuality. In contrast, SUPT innovates
by assigning prompt features at the subgraph level, capturing the intricate
contextual nuances of each subgraph. This distinction allows SUPT to apply
varied prompt features to nodes of the same type, contingent on their subgraph
context, thus introducing a layer of context-specific differentiation even
among nodes sharing the same type.

In recent years, the primary method to leverage pre-trained models in various
domains, including graphs, has been the ”pre-train, fine-tune” framework (Hu
et al., 2020b). This approach entails initial training of the model on a large
and diverse dataset to capture general features, which is subsequently
followed by fine-tuning on a smaller, task-specific dataset. Although this
methodology has demonstrated success, it is not devoid of critical issues. One
of the most significant challenges is the misalignment between the objectives
of pre-training tasks and those of downstream tasks. For instance, a model
pre-trained on generic graph structures may struggle to adapt to specific
graph classification tasks due to differing objectives. Furthermore, ensuring
that the model retains its generalization ability during the fine-tuning
process is a non-trivial task. As models are fine-tuned on specific datasets,
they risk losing their ability to generalize to new, unseen data, a phenomenon
known as catastrophic forgetting (Liu et al., 2021; Zhou and Cao, 2021).

To address these challenges, the concept of prompt tuning, ”pre-train, prompt,
fine-tune”, has emerged as a promising solution (Li and Liang, 2021; Lester et
al., 2021; Liu et al., 2023c). Inspired by the success of prompt tuning in
NLP, recent studies in computer vision are actively exploring the adaptation
of these methods to incorporate visual prompts at the pixel-level, as depicted
in Figure 1 (Bahng et al., 2022; Xing et al., 2023; Chen et al., 2023; Liu et
al., 2023a; Li et al., 2023; Oh et al., 2023; Tsao et al., 2024). Unlike the
”pre-train, fine-tune” approach, prompt tuning involves modifying the input
data to better align with the downstream tasks while keeping the pre-trained
model parameters fixed. This method has shown potential in maintaining the
generalization abilities of the model and ensuring better alignment with
specific task objectives.

However, when it comes to Graph Neural Networks (GNNs), the landscape of
prompt tuning methods remains largely unexplored, particularly in terms of
universal applicability (Fang et al., 2023). Existing prompt tuning methods in
GNNs are often tightly coupled with specific pre-training strategies, limiting
their broader application (Sun et al., 2022; Liu et al., 2023b). For example,
a prompt tuning method designed for a GNN pre-trained on node classification
tasks may not be effective for a GNN pre-trained on link prediction tasks.
This lack of universal prompt tuning methods for GNNs stems from the diverse
and complex nature of graph data and the corresponding variety of pre-training
strategies employed. Each graph dataset and task can have unique
characteristics and requirements, making the development of a one-size-fits-
all prompt tuning approach particularly challenging. As a result, there is a
pressing need for innovative methods in prompt tuning that can adapt to the
wide range of pre-training strategies employed in graph neural networks,
thereby unlocking their full potential across various graph-based
applications. Recently, Fang et al. introduced the concepts of Graph Prompt
Feature (GPF) and its extension (GPF-plus), designed to be theoretically
equivalent to any prompting function, thus enhancing their adaptability across
various downstream tasks. However, the effectiveness of these relatively
straightforward prompting mechanisms in capturing the intricate contexts
prevalent within graph structures is yet to be fully determined. Further
investigation is required in this domain, given that GPF utilizes uniform
prompting and GPF-plus adopts a node-specific approach, as illustrated in
Figure 2.

In this study, we refine the concept of universal graph prompt tuning by
introducing an approach that emphasizes intricacy at the subgraph-level, which
we term Subgraph-level Universal Prompt Tuning (SUPT). This advancement builds
upon existing frameworks like GPF but distinguishes itself by applying prompts
at the subgraph-level, enabling more refined control over input features. Such
granularity improves the method’s flexibility and utility across a variety of
graph-based applications, addressing the complex and diverse nature of graph
data. SUPT not only delivers improved performance across a variety of
downstream tasks but also does so while maintaining a small number of
parameters comparable to the GPF and GPF-plus. Through an extensive evaluation
involving 45 experiments across nine datasets and five pre-training
methodologies, SUPT stands out by requiring far fewer tuning parameters
compared to fine-tuning approaches. In full-shot scenarios, it surpasses these
conventional methods in 42 out of 45 cases, registering an average performance
gain of over 2.5%. The efficacy of SUPT extends into few-shot scenarios as
well, where it excels in 41 out of 45 experiments, marking an average
performance increment of more than 6.6%. The implementation is available at:
https://anonymous.4open.science/r/SUPT-F7B1/

##  2\. Related works

#### Graph Pre-training.

Influenced by advancements in NLP (Qiu et al., 2020b) and Computer Vision (Du
et al., 2022), the graph domain increasingly adopts pre-trained models (Xia et
al., 2022c). These models, trained on extensive graph datasets, learn to
generalize features across a broad range of applications. Various methods for
graph pre-training have emerged, showing substantial progress. Early efforts
included GAE (Kipf and Welling, 2016) that introduces a method to train graphs
using edge prediction as a training objective utilizing variational auto-
encoder. Deep Graph Infomax (Veličković et al., 2019) and InfoGraph (Sun et
al., 2019) focus on maximizing mutual information between different levels of
representations to obtain pre-trained graph model. Subsequent studies (Hu et
al., 2020a; You et al., 2020) diversify the pre-training tasks to self-
supervised attributed graph generation task (Hu et al., 2020a) and contrastive
learning with diverse graph augmentation techniques (You et al., 2020). Hu et
al. propose an effective strategy to pre-train through attribute masking and
context prediction, followed by fine-tuning for classification tasks. Follow-
up studies confirm the effectiveness of ”pre-train, fine-tune” framework in
adapting pre-trained models to individual downstream tasks (Han et al., 2021;
Xia et al., 2022b). AUX-TS (Han et al., 2021) employs a meta-learning based
weighting model for adaptive auxiliary task selection during fine-tuning. Xia
et al. introduce data augmentation and regularization methods to optimize the
fine-tuning of pre-trained molecular graphs. Nevertheless, challenges arise
with the strategy, particularly in aligning various pre-training objectives
with specific downstream tasks, which often results in catastrophic forgetting
and limited task-specific adaptability (Liu et al., 2021). Additionally, as
the number of pre-trained parameters increases, fine-tuning for individual
downstream tasks becomes more resource-intensive (Liu et al., 2023c).

#### Graph Prompt Tuning.

Prompt tuning (Liu et al., 2023c), initially conceptualized in NLP, has been
widely applied to enhance the adaptability of pre-trained models to downstream
tasks. The primary advantage of the method lies in bridging the gap between
pre-training and downstream task objectives, requiring fewer parameter updates
compared to traditional fine-tuning (Liu et al., 2023c). Early approaches
utilize hard prompts (Brown et al., 2020), which are in the form of discrete
natural language phrases. Subsequent research explores soft prompts (Lester et
al., 2021; Liu et al., 2023d; Li and Liang, 2021), employing continuous
vectors to surpass the limitations of natural language embeddings. In graph
prompt tuning, studies have actively investigated the advantages (Sun et al.,
2022; Liu et al., 2023b; Fang et al., 2023), with a focus on soft prompts due
to the unique nature of graph data. GPPT (Sun et al., 2022) introduces a ”pre-
train, prompt, fine-tune” framework to mitigate the objective gap using soft
prompt, but it only deals with link prediction for pre-training and node
classification for downstream task. GraphPrompt (Liu et al., 2023b) expands
its application to node and graph classifications, but remains restricted in
pre-training objectives. Sun et al. proposes a multi-task prompting framework
that bridges the gap between pre-training and multiple downstream tasks
inserting learnable prompt graph into the original graph. PRODIGY introduces a
framework that introduces in-context learning on graph, making the model to be
pre-trained to solve tasks across a wide range of tasks with prompt graph
(Huang et al., 2023). GPF and GPF-plus (Fang et al., 2023) add soft prompts to
all node features of the input graph, enabling the method to be applied to any
pre-training strategy. Despite these advancements, most methods face
limitations in handling diverse pre-training strategies and downstream tasks.
Fang et al. effectively expand the scope of application but face difficulties
in embedding unique graph context, restricting soft prompts from capturing
context-specific details.

##  3\. Method

In this section, we introduce an advanced method for universal graph prompt
tuning that is applied to the features of input nodes at the subgraph-level,
referred to as Subgraph-level Universal Prompt Tuning (SUPT). Inspired by the
concept of pixel-level prompts for images (Bahng et al., 2022; Xing et al.,
2023; Chen et al., 2023; Liu et al., 2023a; Li et al., 2023; Oh et al., 2023;
Tsao et al., 2024) and node-level prompts for graphs (Fang et al., 2023), SUPT
similarly utilizes additional prompt features into the input space of graphs.

###  3.1. Preliminaries

Graph-structured data can be represented as
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ denotes the set
of nodes and $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$ represents the
set of edges, respectively. The graph includes a node feature matrix
$X\in\mathbb{R}^{|\mathcal{V}|\times d}$ and an adjacency matrix
$A\in\\{0,1\\}^{|\mathcal{V}|\times|\mathcal{V}|}$ with feature dimension $d$.
Denoted as $f_{\theta}$, the pre-trained model is optimized according to the
pre-training method’s objective with a parameter set $\theta$.

#### Fine-Tuning.

In general, fine-tuning methods optimize the parameters of the pre-trained
model $f_{\theta}$ along with a projection head $h_{\phi}$ parameterized by
$\phi$, using a downstream task dataset $\mathcal{D}$ that consists of tuples
of graphs and labels $(\mathcal{G},y)$. The optimization objective is to
maximize the likelihood of correctly predicting labels for graphs in the
downstream task, with parameters $\theta$ and $\phi$:

(1) $\max_{\theta,\phi}P_{f_{\theta},h_{\phi}}(y|\mathcal{G})$

#### Prompt Tuning.

In prompt tuning, the input graph $\mathcal{G}$ is modified to prompted graph
$g_{\psi}(\mathcal{G})$ through a parameterized prompt function $g_{\psi}$,
instead of updating the pre-trained model $f_{\theta}$. The objective of
prompt tuning is to find an optimal task-specific prompted graph
$g_{\psi}(\mathcal{G})$ by maximizing the likelihood with parameters $\psi$
and $\phi$:

(2) $\max_{\psi,\phi}P_{f_{\theta},h_{\phi}}(y|g_{\psi}(\mathcal{G}))$

Note that the number of parameters $\psi$ is significantly smaller than the
number of parameters $\theta$ in the pre-trained model. This is demonstrated
in Table 1, highlighting the parameter efficiency in the training process of
prompt tuning methods compared to fine-tuning methods.

| #Params | #Params Ratio(%) |  | Full Shot |  | 50-Shot
---|---|---|---|---|---|---
| Chem | Bio |  | Outperformance | Gain |  | Outperformance | Gain
FT |  $\sim$ 1.8M-2.7M | 100 | 100 |  | - | 0 |  | - | 0
GPF |  $\sim$ 0.3K | 0.02 | 0.01 |  | 27 out of 45 | +0.91 |  | 33 out of 45 | +1.89
GPF-plus |  $\sim$ 3-12K | 0.17-0.68 | 0.11-0.44 |  | 27 out of 45 | +1.06 |  | 35 out of 45 | +2.00
SUPT${}_{\text{hard}}$ |  $\sim$ 0.6-3K | 0.03-0.17 | 0.02-0.11 |  | 42 out of 45 | +1.85 |  | 41 out of 45 | +3.73
SUPT${}_{\text{soft}}$ |  $\sim$ 0.6-3K | 0.03-0.17 | 0.02-0.11 |  | 43 out of 45 | +2.05 |  | 41 out of 45 | +3.67
Table 1. In an overall comparison of the fine-tuning (FT), GPF, and SUPT
approaches, it is observed that while FT involves adjusting the model’s entire
parameter set, prompt tuning strategies like GPF and SUPT employ far fewer
additional training parameters. In particular, SUPT stands out by maintaining
a small number of prompt parameters yet consistently delivers improved
performance, outperforming FT across most experiments.

#### Graph Prompt Feature.

Recently, Fang et al. introduced a parameterized prompt function known as GPF
and GPF-plus. GPF-plus incorporates the learnable prompt feature vector
$p\in\mathbb{R}^{d}$ into the node feature $x\in\mathbb{R}^{d}$ as:

(3) $X=\\{x_{1},...,x_{N}\\},\quad X^{*}=\\{x_{1}+p_{1},...,x_{N}+p_{N}\\}$

where $N$ is the number of nodes and $p$ is calculated as weighted sum of
learnable basis vectors. The weights for the attentive aggregation of the
basis vector $b$ are determined by the softmax values of the learnable linear
projection $a$ applied to the node feature $x$. This can be formulated as:

(4)
$p_{i}=\sum_{j}^{k}\alpha_{i,j}b_{j},\quad\alpha_{i,j}=\frac{\exp(a_{j}^{\top}x_{i})}{\sum_{l}^{k}\exp(a_{l}^{\top}x_{i})}$

where $p_{i}$ denotes the final prompt feature for the $i$-th node,
$\alpha_{i,j}$ represents the attention weight of the $j$-th basis for the
$i$-th node, and the hyperparameter $k$ is the number of basis vectors and
projections. The weight $\alpha_{i,j}$ varies according to the type of node
feature, that is, $x_{i}$, and therefore, the prompt feature $p_{i}$ becomes a
category-specific prompt for the node. GPF, applying a uniform prompt feature
to all nodes, can be considered a special case of GPF-plus with $k=1$.

###  3.2. Subgraph-level Universal Prompt Tuning

In subgraph-level prompt tuning, we adhere to Equation (3), which operates in
the input space. However, instead of utilizing the node feature $x$, we
allocate prompt feature vectors based on scores derived from simple GNNs, such
as GCN (Kipf and Welling, 2017) or SGC (Wu et al., 2019) for $m$-hop context.
Consequently, the prompt calculation is as follows:

(5)
$p_{i}=\sum_{j}^{k}\sigma(\alpha)_{i,j}b_{j},\quad\alpha=(\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}})^{m}(X\oplus\sum_{j}^{k}b_{j})W$

where $\tilde{A}$ and $\tilde{D}$ denote the adjacency matrix and degree
matrix with self-loops, respectively, $\oplus$ represents a broadcasted
addition, $W\in\mathbb{R}^{d\times k}$ is a weight matrix, $k$ is a
hyperparameter defining the number of subgraphs, and $\sigma$ indicates a non-
linear operation such as a softmax or an activation function with top-rank
selection. In this context, $W$ and $b$ are learnable parameters of SUPT and
correspond to elements of $\psi$ in Equation (2). Based on the definition of
$\sigma$, we have developed two variants: SUPT${}_{\text{soft}}$, which
utilizes a softmax function, and SUPT${}_{\text{hard}}$, which employs a top-
rank selection mechanism.

#### SUPT${}_{\text{soft}}$

Inspired by cluster-based graph pooling methods (Ying et al., 2018; Ma et al.,
2019; Bianchi et al., 2020; Yuan and Ji, 2020; Khasahmadi et al., 2020), we
view the basis feature $b$ as a prompt assigned to a cluster, which
constitutes a subset of nodes. The non-linear operation $\sigma$ in Equation
(5) is defined as a column-wise softmax function that is defined as a column-
wise softmax function, which ensures that each node is associated with a
specific basis. This setup allows $\alpha\in\mathbb{R}^{N\times k}$ to
function similarly to the assignment matrix in soft-cluster graph pooling
methods. Additional techniques, such as entropy regularization (Ying et al.,
2018), can be employed to induce sparsity in $\alpha$, making the basis
assignment for each node (e.g. $\alpha_{i,:}$ for $i$-th node) approximate a
one-hot vector. However, in this work, we achieved a sparse implementation by
leveraging node-selection graph pooling methods as described below.

Dataset | PPI | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE
---|---|---|---|---|---|---|---|---|---
# Proteins / Molecules | 88K | 2039 | 7831 | 8575 | 1427 | 1478 | 93087 | 41127 | 1513
# Binary prediction tasks | 40 | 1 | 12 | 617 | 27 | 2 | 17 | 1 | 1
Table 2. Statistics of downstream task datasets.

#### SUPT${}_{\text{hard}}$

We exploit node-selection graph pooling methods (Gao and Ji, 2019; Lee et al.,
2019; Zhang et al., 2020; Ma et al., 2020) to implement the sparse version of
SUPT. Unlike SUPT${}_{\text{soft}}$, SUPT${}_{\text{hard}}$ determines the
nodes to be assigned to each basis through top-rank selection. By performing a
row-wise top-rank selection on $\alpha$, we obtain the indices of the subset
of nodes corresponding to each basis. Nodes that do not correspond to a
particular basis do not receive any addition of the prompt. For instance,
considering the $k$-th basis:

idx $\displaystyle=\operatorname{top-rank}(\alpha_{:,k},\lceil rN\rceil)$ (6)
$\displaystyle X^{*}_{\text{idx},:}$
$\displaystyle=X_{\text{idx},:}+\delta(\alpha_{\text{idx},k})\odot b_{k}$

where $r$, a hyperparameter in the range $(0,1]$, specifies the pooling ratio,
$\delta$ represents an activation function such as sigmoid, and $\odot$
denotes element-wise multiplication with broadcasting. The $\operatorname{top-
rank}$ function selects the indices of the top $\lceil rN\rceil$ values in
$\alpha_{:,k}\in\mathbb{R}^{N}$, choosing the nodes for the $k$-th basis. In
SUPT${}_{\text{hard}}$, it is possible for some nodes to not be selected by
any basis, whereas others could be influenced by prompts from all bases. To
address these disparities, an additional normalization term, such as division
by the number of corresponding bases, can be incorporated.

#### Methodological Characteristics of Two Variants

The primary distinction between the two variants, SUPT${}_{\text{soft}}$ and
SUPT${}_{\text{hard}}$, lies in their respective approaches to subgraph
definition.

SUPT${}_{\text{soft}}$ employs a learning approach akin to defining a subset
of nodes belonging to the same cluster when determining which nodes will be
influenced by the same prompt basis, essentially training an assignment
matrix. Consequently, every node is influenced by at least one prompt basis,
with the extent of influence from any given cluster determined by a softmax
function. While similar to GPF-plus in that a prompt is added to all nodes,
SUPT${}_{\text{soft}}$ differentiates itself by determining the assignment
score for each basis through a GNN, rather than simply relying on the input
node features $x$.

Conversely, SUPT${}_{\text{hard}}$ exclusively selects only the top-ranked
nodes when determining which ones will be influenced by the same prompt basis.
Nodes not selected by any basis are input into the pre-trained model without
any added prompt, maintaining their original input features. This constitutes
the technical distinction from GPF, GPF-plus, and SUPT${}_{\text{soft}}$.

Both methodologies are inspired by graph pooling methods, yet they do not
inherit the associated limitations. For instance, cluster-based graph pooling,
while beneficial for aggregating node features to create new features, has
faced criticism for leading to dense output graphs. SUPT${}_{\text{soft}}$, on
the other hand, merely adds an extra prompt but preserves the sparsity of the
graph structure, addressing this concern. Similarly, node selection-based
pooling methods have been criticized for the potential loss of information
from nodes that are not selected. However, this concern does not apply to
SUPT${}_{\text{hard}}$, which merely refrains from adding prompts to
unselected nodes and preserves their original features. In summary, SUPT is
capable of integrating subgraph-level information beneficial for downstream
tasks, leveraging an advantage of graph pooling methods without inheriting
their significant limitations.

#### Theoretical Analysis

In this part, we present a theoretical analysis for SUPT, establishing its
universality. We theoretically show that our methodologies are capable of
attaining outcomes equivalent to any prompt function. This establishes the
broad applicability and versatility of our techniques, making them suitable
for a variety of pre-training strategies. Unlike GPF, a notable feature of
SUPT is that the prompt is applied only to a subgraph (i.e., a subset of nodes
$\mathcal{S}\subseteq\mathcal{V}$), rather than to the entire node set of a
given graph. A key aspect of our work is to theoretically demonstrate that
SUPT maintains universality under these conditions. Consequently, this implies
that SUPT can also achieve the theoretical performance upper bound for any
prompting function, as shown in the GPF paper (Fang et al., 2023). In the
context of a specified pre-training task $t$ from a set of tasks $\mathbb{T}$,
and given an input graph $\mathcal{G}$ characterized by its nodes $X$ and
adjacency matrix $A$, we introduce a prompting function $\Psi_{t}(\cdot)$.
This function generates a graph template $\mathcal{G}^{*}=(X^{*},A^{*})$,
where $X^{*}$ is an element of the candidate space $\mathbb{X}$, and $A^{*}$
is an element of the candidate space $\mathbb{A}$, with
$\mathcal{G}^{*}=\Psi_{t}(\mathcal{G})$.

######  Theorem 1 (Universal Capability of SUPT).

Let $f$ be a pre-trained GNN model and $\mathcal{G}:(X,A)$ be an input graph
with sets of nodes and edges, $\mathcal{V}$ and $\mathcal{E}$. Given any
prompting function $\psi_{t}(\cdot)$, if a prompted graph
$\hat{\mathcal{G}}:(\hat{X}\in\mathbb{X},\hat{A}\in\mathbb{A})$ is in the
candidate space $\mathcal{G}^{*}=\Psi_{t}(\mathcal{G})$, then there exists a
vector $\hat{p}$ such that for any non-empty subset of nodes $\mathcal{S}$
(i.e., $\mathcal{S}\subseteq\mathcal{V}$ and $|\mathcal{S}|\geq 1$), the
updated features can be defined as:

$X^{\prime}=\begin{cases}X_{s}+\hat{p}&\text{if }s\in\mathcal{S}\\\
X_{s}&\text{if }s\notin\mathcal{S}\end{cases}$

such that there exists $X^{\prime}$ satisfying:

$f(X^{\prime},A)=f(\hat{X},\hat{A})$

The proof of Theorem 1 is detailed in the appendix. This theorem establishes
that applying a prompt to a subgraph can meet the theoretical performance
upper bound for any prompting function, as delineated in Equation (2).
Essentially, if the optimization of a graph template $\mathcal{G}^{*}$,
formulated by a given prompting function $\Psi(\cdot)$, leads to effective
graph representations, then theoretically, the optimization of the prompt
vector $p$ within the SUPT framework can also achieve exact graph
representations.

##  4\. Experiments

###  4.1. Experiment Setup

For a fair comparison, we strictly adhere to the experimental setup outlined
in the GPF paper (Fang et al., 2023), which is itself based on the
implementation developed by Hu et al..

#### Datasets

In the pre-training phase, Fang et al. and Hu et al. leveraged datasets from
both the biological and chemical domains, referred to as the Biology and
Chemistry datasets, respectively. The Biology dataset is composed of
approximately 395k unlabeled protein ego-networks, derived from Protein-
Protein Interaction (PPI) networks across 50 distinct species. These networks
were employed for node-level self-supervised pre-training. Additionally, the
dataset includes 88k labeled protein ego-networks, utilized for the prediction
of 5,000 biological functions, serving the purpose of graph-level multi-task
supervised pre-training. On the chemical side, the Chemistry dataset
incorporates around 2 million unlabeled molecules, which were sampled from the
ZINC15 database (Sterling and Irwin, 2015) for the purpose of node-level self-
supervised pre-training. For graph-level multi-task supervised pre-training, a
preprocessed ChEMBL dataset (Gaulton et al., 2012; Mayr et al., 2018) was
used. This subset comprises 456k molecules with 1,310 biochemical assays,
providing a diverse chemical space for comprehensive model training.

For downstream tasks, they utilized the PPI networks dataset, which consists
of 88K proteins from 8 different species, to predict 40 biological functions
(Zitnik et al., 2019), as well as 8 binary classification datasets for
molecular property prediction (Wu et al., 2018). The statistics of datasets
for downstream tasks are summarized in Table 2.

| Tuning | PPI | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
Infomax | FT | 71.29 | 67.55 | 78.57 | 65.16 | 63.34 | 70.06 | 81.42 | 77.71 | 81.32 | 72.94
GPF | 78.87 | 66.58 | 78.13 | 65.96 | 65.64 | 74.14 | 80.39 | 76.67 | 83.24 | 74.40
GPF-plus | 78.85 | 67.03 | 78.50 | 65.83 | 65.89 | 73.61 | 79.94 | 75.60 | 82.96 | 74.25
SUPT${}_{\text{hard}}$ | 79.64 | 67.29 | 79.14 | 66.48 | 65.78 | 75.65 | 81.51 | 77.96 | 84.05 | 75.28
SUPT${}_{\text{soft}}$ | 79.45 | 67.63 | 78.98 | 66.53 | 65.88 | 76.08 | 81.59 | 77.94 | 84.01 | 75.34
AttrMasking | FT | 73.93 | 66.33 | 78.28 | 65.34 | 66.77 | 74.46 | 81.78 | 77.90 | 80.94 | 73.97
GPF | 81.47 | 67.49 | 78.66 | 66.09 | 68.19 | 72.72 | 81.44 | 77.09 | 84.77 | 75.32
GPF-plus | 81.43 | 67.32 | 78.83 | 66.53 | 68.53 | 74.17 | 81.27 | 77.64 | 85.01 | 75.64
SUPT${}_{\text{hard}}$ | 81.90 | 68.39 | 79.50 | 66.79 | 68.82 | 75.25 | 81.81 | 79.10 | 84.98 | 76.28
SUPT${}_{\text{soft}}$ | 81.91 | 68.37 | 79.73 | 67.02 | 68.68 | 77.36 | 81.93 | 79.15 | 85.22 | 76.60
ContextPred | FT | 72.10 | 69.65 | 78.29 | 66.39 | 64.45 | 73.71 | 82.36 | 79.20 | 84.66 | 74.53
GPF | 80.21 | 69.04 | 79.90 | 67.68 | 65.31 | 73.15 | 84.22 | 77.77 | 85.03 | 75.81
GPF-plus | 80.28 | 68.59 | 79.95 | 67.72 | 66.32 | 72.73 | 84.40 | 78.10 | 84.50 | 75.84
SUPT${}_{\text{hard}}$ | 80.85 | 70.18 | 80.00 | 68.40 | 66.37 | 74.90 | 84.54 | 79.42 | 85.21 | 76.65
SUPT${}_{\text{soft}}$ | 80.86 | 70.06 | 80.12 | 68.50 | 66.48 | 75.55 | 84.67 | 79.39 | 85.27 | 76.77
GCL | FT | 67.76 | 69.49 | 73.35 | 62.54 | 60.63 | 75.17 | 69.78 | 78.26 | 75.51 | 70.27
GPF | 67.91 | 68.11 | 72.15 | 60.73 | 61.00 | 69.48 | 70.81 | 75.12 | 76.67 | 69.11
GPF-plus | 66.71 | 67.77 | 73.06 | 61.47 | 61.57 | 68.89 | 70.54 | 75.90 | 77.84 | 69.31
SUPT${}_{\text{hard}}$ | 68.59 | 69.68 | 73.44 | 63.35 | 62.51 | 69.47 | 70.64 | 76.22 | 79.17 | 70.34
SUPT${}_{\text{soft}}$ | 69.57 | 69.79 | 73.40 | 63.42 | 62.84 | 71.21 | 70.92 | 76.44 | 78.92 | 70.72
EdgePred | FT | 71.54 | 66.56 | 78.67 | 66.29 | 64.35 | 69.07 | 79.67 | 77.44 | 80.90 | 72.72
GPF | 79.96 | 69.36 | 79.69 | 65.70 | 66.71 | 67.92 | 82.61 | 77.53 | 79.76 | 74.36
GPF-plus | 79.88 | 69.31 | 79.91 | 65.66 | 66.61 | 69.58 | 82.74 | 77.77 | 80.91 | 74.71
SUPT${}_{\text{hard}}$ | 80.20 | 69.51 | 80.76 | 66.44 | 66.85 | 69.97 | 82.94 | 77.79 | 81.72 | 75.13
SUPT${}_{\text{soft}}$ | 80.19 | 69.93 | 80.90 | 66.65 | 66.95 | 70.41 | 82.95 | 77.95 | 81.44 | 75.26
Table 3. Comparison of test ROC-AUC performance in full-shot scenarios across
protein function prediction and molecular prediction benchmarks using various
pre-training strategies, highlighted in blue for performances superior to
fine-tuning (FT) and in red otherwise. The reported values are the averages
obtained from experiments repeated with 20 random seeds. Full results
including comparisons with other tuning methods such as GPPT and GraphPrompt,
are provided in the Appendix.

#### Pre-training Methods

We employ the model weights (checkpoints) of the pre-trained models provided
by Fang et al. specifically to assess the effectiveness of various prompt
tuning methods, while controlling for other variables. In their study, the
evaluation of GPF across five distinct pre-training strategies was conducted
to establish its universal applicability. The first of these, Deep Graph
Infomax (referred to as Infomax), aims to refine node or graph representations
by optimizing the mutual information between the representations of entire
graphs and their substructures across various levels (Veličković et al.,
2019). Attribute Masking (AttrMasking) involves masking attributes of nodes or
edges and tasking the GNN with inferring these attributes using the adjacent
structural information (Hu et al., 2020b). Similarly, Context Prediction
(ContextPred), also introduced by Hu et al., uses subgraphs to forecast their
surrounding graph contexts, aiming to align nodes that appear in similar
structural contexts closely within the embedding space. Graph Contrastive
Learning (GCL), embeds augmented versions of an anchor graph in close
proximity (as positive samples) while distancing the embeddings of negative
samples (You et al., 2020). Lastly, Edge Prediction (EdgePred), commonly
employed in graph reconstruction studies by the Graph Autoencoder (GAE),
focuses on predicting the presence or absence of edges between pairs of nodes
(Kipf and Welling, 2016).

#### Model Architecture and Training Scheme

Consistent with the approach adopted by Fang et al., our research utilizes the
predominant 5-layer GIN architecture (Xu et al., 2019), which is widely
acknowledged in pre-trained GNN studies (Qiu et al., 2020a; You et al., 2020;
Hu et al., 2020b; Suresh et al., 2021; Xu et al., 2021; Zhang et al., 2021;
You et al., 2022; Xia et al., 2022a). While constructing larger GNN models by
addressing the challenges such as over-smoothing or over-squashing is a
significant research endeavor in itself (Xu et al., 2018; Liu et al., 2020;
Zhou et al., 2021; Li et al., 2021; Wu et al., 2023), the primary focus of
this study is to explore more informative prompt tuning methods when models
are pre-trained through various strategies.

| Tuning | PPI | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
Infomax | FT | 48.79 | 53.81 | 61.42 | 53.93 | 50.77 | 58.60 | 66.12 | 65.09 | 52.64 | 56.80
GPF | 52.39 | 56.18 | 65.42 | 56.80 | 50.19 | 62.40 | 68.00 | 66.97 | 51.13 | 58.83
GPF-plus | 52.44 | 56.25 | 65.19 | 56.30 | 50.20 | 62.34 | 67.99 | 66.71 | 53.53 | 58.99
SUPT${}_{\text{hard}}$ | 56.16 | 57.12 | 65.64 | 57.16 | 51.56 | 64.55 | 67.97 | 67.03 | 57.03 | 60.47
SUPT${}_{\text{soft}}$ | 56.29 | 57.06 | 65.92 | 57.16 | 51.51 | 64.44 | 68.27 | 67.04 | 56.59 | 60.48
AttrMasking | FT | 48.17 | 51.26 | 60.28 | 53.47 | 50.11 | 61.51 | 59.35 | 67.18 | 55.62 | 56.33
GPF | 52.05 | 52.60 | 64.35 | 56.69 | 49.61 | 64.80 | 60.84 | 67.74 | 53.04 | 57.97
GPF-plus | 52.12 | 53.43 | 64.29 | 56.71 | 50.19 | 62.92 | 62.10 | 68.19 | 51.28 | 57.91
SUPT${}_{\text{hard}}$ | 56.58 | 53.12 | 64.66 | 56.74 | 51.54 | 66.35 | 62.20 | 67.99 | 56.57 | 59.53
SUPT${}_{\text{soft}}$ | 56.58 | 54.10 | 64.57 | 56.76 | 51.66 | 66.27 | 62.15 | 68.05 | 56.80 | 59.66
ContextPred | FT | 46.33 | 49.45 | 58.77 | 54.46 | 49.89 | 48.60 | 56.14 | 60.91 | 56.37 | 53.44
GPF | 51.42 | 50.99 | 59.76 | 55.31 | 50.23 | 52.03 | 60.26 | 59.62 | 56.19 | 55.09
GPF-plus | 51.48 | 52.15 | 59.77 | 54.87 | 50.19 | 51.71 | 60.05 | 60.19 | 55.49 | 55.10
SUPT${}_{\text{hard}}$ | 56.12 | 52.29 | 60.34 | 55.46 | 52.25 | 57.87 | 62.54 | 60.64 | 58.97 | 57.39
SUPT${}_{\text{soft}}$ | 56.15 | 52.68 | 60.72 | 55.39 | 52.25 | 53.43 | 61.32 | 60.82 | 58.31 | 56.79
GCL | FT | 49.94 | 54.40 | 48.35 | 50.29 | 53.23 | 54.05 | 46.73 | 60.05 | 49.87 | 51.88
GPF | 46.00 | 53.61 | 50.23 | 53.92 | 50.56 | 53.21 | 51.92 | 62.72 | 61.70 | 53.76
GPF-plus | 46.08 | 53.35 | 50.36 | 52.66 | 50.08 | 60.79 | 49.53 | 63.43 | 61.72 | 54.22
SUPT${}_{\text{hard}}$ | 50.48 | 54.01 | 51.14 | 54.17 | 51.95 | 61.51 | 52.86 | 63.82 | 63.25 | 55.91
SUPT${}_{\text{soft}}$ | 50.32 | 53.96 | 52.21 | 54.20 | 52.00 | 61.62 | 52.94 | 64.05 | 63.61 | 56.10
EdgePred | FT | 47.62 | 48.88 | 60.95 | 55.73 | 51.30 | 57.78 | 66.88 | 64.22 | 61.27 | 57.18
GPF | 52.79 | 51.38 | 64.00 | 57.10 | 49.90 | 67.82 | 67.58 | 62.32 | 62.03 | 59.44
GPF-plus | 52.88 | 54.05 | 64.41 | 57.33 | 50.18 | 64.51 | 67.51 | 62.85 | 60.84 | 59.40
SUPT${}_{\text{hard}}$ | 57.42 | 54.14 | 64.90 | 57.90 | 52.56 | 67.95 | 68.39 | 63.44 | 62.12 | 60.98
SUPT${}_{\text{soft}}$ | 57.26 | 54.36 | 65.19 | 57.88 | 52.45 | 67.70 | 68.73 | 63.25 | 61.70 | 60.95
Table 4. Performance measured by test ROC-AUC in 50-shot scenarios, with
superior performances to FT highlighted in blue and others highlighted in red.
The average values obtained from experiments repeated with 20 random seeds are
reported.

Given a pre-trained GNN $f_{\theta}$ and a task-specific projection head
$h_{\phi}$, Fine-Tuning (FT) involves adjusting the parameters of both
$\theta$ and $\phi$. In contrast, prompt tuning methods such as GPF, GPF-plus,
and SUPT, entail tuning the parameters of $\psi$ for the prompt function
$g_{\psi}$ as well as $\phi$, while maintaining $\theta$ as fixed. In our
experiments, we selected the projection head from an identical range of MLP
layer counts as those used in GPF and GPF-plus. The hyperparameter $k$ of SUPT
in Equation (5), representing the number of subgraphs, was chosen from the set
{1, 2, 3, 4, 5}. The pooling ratio $r$ in Equation (6) for
SUPT${}_{\text{hard}}$ was selected from {0.2, 0.4, 0.6}. We simply set the
number of hops $m$ to 1; thereby $\alpha$ in Equation (5) is calculated using
GCN. The Adam optimizer with weight decay (Kingma and Ba, 2015), as utilized
by Fang et al. and Hu et al., was also employed for tuning SUPT. Additional
information regarding the settings of the hyperparameters is provided in the
appendix. To ensure robustness and reliability of experimental results, we
report the average performance from 20 random seeds. Consequently, the
performance for GPF and GPF-plus reported in this study may deviate from those
previously published, which were contingent upon evaluations conducted with 5
random seeds. In this study, all experiments were conducted using an RTX 8000.
The full table including comparisons with other pre-training specific
prompting methods such as GPPT (Sun et al., 2022) and GraphPrompt (Liu et al.,
2023b), is detailed in the appendix. This is because our primary focus lies on
the comparison of universal prompt tuning methods.

###  4.2. Experiment Results

In the comparison presented in Table 1, it is observed that all prompt tuning
methods generally exhibit performance comparable to that of FT, despite having
a significantly fewer number of tunable parameters.

#### Full-shot scenarios

The experimental results in full-shot scenarios presented in Table 3 reveal
that the SUPT models, based on AttrMasking, ContextPred, and EdgePred,
consistently outperform FT across all cases. Conversely, GPF and GPF-plus
models do not always exhibit superior performance. Specifically,
SUPT${}_{\text{soft}}$, leveraging Infomax, surpasses FT in every dataset,
while SUPT${}_{\text{hard}}$ exceeds FT in all but one dataset. In experiments
employing GCL, both SUPT${}_{\text{soft}}$ and SUPT${}_{\text{hard}}$
underperform relative to FT in two datasets, yet they demonstrate superior
performance in the majority of the remaining datasets. Out of 45 experiments
conducted across 9 datasets and 5 pre-training strategies, GPF and GPF-plus
outperform FT in 27 cases, whereas SUPT${}_{\text{hard}}$ and
SUPT${}_{\text{soft}}$ do so in 42 and 43 cases, respectively. On average, GPF
and GPF-plus exhibit performance improvements of 0.91 and 1.06 over FT,
respectively. In contrast, SUPT${}_{\text{hard}}$ and SUPT${}_{\text{soft}}$
show more notable enhancements, with average performance gains of 1.85 and
2.05, respectively.

| $k$ | PPI | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE
---|---|---|---|---|---|---|---|---|---|---
SUPT${}_{\text{hard}}$ | $1$ | $7.94\times 10^{-5}$ | $8.60\times 10^{-5}$ | $7.67\times 10^{-5}$ | $9.13\times 10^{-5}$ | $7.65\times 10^{-5}$ | $8.80\times 10^{-5}$ | $7.73\times 10^{-5}$ | $7.80\times 10^{-5}$ | $7.54\times 10^{-5}$
$3$ | $1.94\times 10^{-4}$ | $1.86\times 10^{-4}$ | $2.01\times 10^{-4}$ | $1.85\times 10^{-4}$ | $1.86\times 10^{-4}$ | $1.87\times 10^{-4}$ | $2.41\times 10^{-4}$ | $1.86\times 10^{-4}$ | $2.12\times 10^{-4}$
$5$ | $3.01\times 10^{-4}$ | $3.24\times 10^{-4}$ | $2.94\times 10^{-4}$ | $3.22\times 10^{-4}$ | $3.33\times 10^{-4}$ | $2.94\times 10^{-4}$ | $2.93\times 10^{-4}$ | $3.42\times 10^{-4}$ | $2.92\times 10^{-4}$
SUPT${}_{\text{soft}}$ | $1$ | $2.11\times 10^{-5}$ | $2.26\times 10^{-5}$ | $2.24\times 10^{-5}$ | $2.28\times 10^{-5}$ | $1.92\times 10^{-5}$ | $2.11\times 10^{-5}$ | $2.19\times 10^{-5}$ | $2.19\times 10^{-5}$ | $1.95\times 10^{-5}$
$3$ | $2.35\times 10^{-5}$ | $2.27\times 10^{-5}$ | $2.30\times 10^{-5}$ | $2.38\times 10^{-5}$ | $2.30\times 10^{-5}$ | $2.47\times 10^{-5}$ | $2.21\times 10^{-5}$ | $2.26\times 10^{-5}$ | $1.94\times 10^{-5}$
$5$ | $2.42\times 10^{-5}$ | $2.28\times 10^{-5}$ | $2.46\times 10^{-5}$ | $2.45\times 10^{-5}$ | $2.30\times 10^{-5}$ | $2.09\times 10^{-5}$ | $2.24\times 10^{-5}$ | $2.26\times 10^{-5}$ | $1.94\times 10^{-5}$

Table 5. Inference time (seconds) per graph for adding prompts
$g_{\psi}(\mathcal{G})$, averaged over 1,000 repetitions, shows that
SUPT${}_{\text{hard}}$ experiences a more pronounced increase in processing
time with the addition of subgraphs compared to SUPT${}_{\text{soft}}$.

#### Few-shot scenarios

As depicted in Table 4, the results from 50-shot experiments across 45
different settings are reported. Both SUPT${}_{\text{hard}}$ and
SUPT${}_{\text{soft}}$ demonstrate superior performance over FT across all
datasets in experiments based on Infomax and AttrMasking. In the case of
ContextPred and EdgePred, they outperform FT in all but one dataset for each
strategy, and for GCL, they exceed FT in all but two datasets. Generally, SUPT
demonstrates a consistent ability to outperform FT across a variety of pre-
training strategies. On the other hand, GPF falls short of surpassing FT in
four datasets within the GCL strategy. Out of the total 45 experiments, GPF
outperforms FT in 33 cases, GPF-plus in 35 cases, and both
SUPT${}_{\text{hard}}$ and SUPT${}_{\text{soft}}$ in 41 cases. On average, GPF
and GPF-plus show performance improvements of 1.89 and 2.00 over FT,
respectively, while SUPT${}_{\text{hard}}$ and SUPT${}_{\text{soft}}$ exhibit
even higher average performance gains of 3.73 and 3.67, respectively. This
indicates that all four prompt tuning methods yield greater performance
enhancements in the 50-shot setting compared to the full-shot setting.

##  5\. Discussion

#### Efficiency in Parameters and Computation

As reported in Table 1, the number of learning parameters for prompt tuning
methods (denoted as $\psi$ in Equation (2)) is significantly lower than those
required for fine-tuning methods (represented by $\theta$ in Equation (1)),
with all being less than 1% of the latter. Notably, SUPT necessitates more
parameters than GPF but fewer than GPF-plus. While expanding the number of
subgraphs ($k$ in Equation (5)) in SUPT might imply a higher demand for
parameters, empirical evidence demonstrates that SUPT is capable of surpassing
the performance of GPF-plus without necessitating an increase in the number of
subgraphs. This underscores the advantage of employing more intricately
patterned prompts over uniformly applied or node-type-specific ones in
capturing the complex and diverse nature of graph data.

The limitation of SUPT lies in the additional computational step required to
calculate $\alpha$ in Equation (5), which involves operations such as
$\tilde{D}^{-\frac{1}{2}}\tilde{A}\tilde{D}^{-\frac{1}{2}}XW$. This step has a
computational complexity of $\mathcal{O}(|\mathcal{E}|d+|\mathcal{V}|dk)$,
potentially impacting practical inference times. Testing with the BBBP
dataset, where each graph’s prompt addition was repeated 1,000 times, showed
an average added time of $2.27\times 10^{-5}$ seconds per graph for
SUPT${}_{\text{soft}}$ and $1.86\times 10^{-4}$ seconds for
SUPT${}_{\text{hard}}$, with the number of subgraphs $k=3$ as reported in
Table 5. The computational time for SUPT${}_{\text{hard}}$ is longer due to
the top-rank operation in Equation (6). Also, SUPT${}_{\text{hard}}$ exhibits
a greater escalation in processing time as the number of subgraphs increases,
in contrast to SUPT${}_{\text{soft}}$. Despite this, the increase in inference
time is negligible, suggesting that the practical implications are manageable
and the benefits of SUPT’s enhanced performance are significant.

#### Patterns of Subgraph Selection

In Figure 2, the subgraphs of SUPT are depicted as if they are composed solely
of adjacent nodes for the sake of a conceptual comparison, yet there are no
technical constraints mandating such a configuration. However, due to the use
of GCN or SGC in calculating scores for each prompt basis as per Equation (5),
adjacent nodes tend to exhibit similar scoring patterns for each basis. This
phenomenon aligns with numerous studies asserting that GNNs act like low-pass
filters or possess smoothing effects (Xu et al., 2018; Nt and Maehara, 2019;
Oono and Suzuki, 2020). Consequently, even without employing specific
techniques, nodes in proximity tend to be grouped into the same subgraph,
somewhat akin to the illustration in Figure 2. Furthermore, inspired by the
DiffPool (Ying et al., 2018), which also influenced the development of
SUPT${}_{\text{soft}}$, an auxiliary loss $\mathcal{L}_{LP}$ in the form of
link prediction can be incorporated to enhance the model’s performance:

(7) $\mathcal{L}_{LP}=\|A,\alpha\alpha^{\top}\|_{F}$

where $\|\cdot\|_{F}$ denotes the Frobenius norm.

##  6\. Conclusion

In this study, we introduced the Subgraph-level Universal Prompt Tuning (SUPT)
approach, focusing on enhancing the prompt tuning process for graph neural
networks by incorporating prompt features at the subgraph level. This method
maintains the flexibility of prompt tuning across various pre-training
strategies while requiring significantly fewer tuning parameters than
traditional fine-tuning methods. Our experimental results demonstrate the
effectiveness of SUPT, showing notable improvements in performance in both
full-shot and few-shot learning scenarios, outperforming existing methods in
the majority of our tests. These findings suggest that SUPT is a viable and
efficient alternative for adapting pre-trained graph neural networks for
diverse downstream tasks, although the exploration of simple prompts’ ability
to capture complex graph contexts warrants further investigation. The study
contributes to the broader machine learning field by offering insights into
the potential of prompt tuning in graph-based applications, highlighting the
balance between efficiency and effectiveness in model adaptation strategies.

## References

  * (1)
  * Bahng et al. (2022) Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. 2022.  Exploring Visual Prompts for Adapting Large-Scale Models.  _arXiv preprint arXiv:2203.17274_ (2022).
  * Bianchi et al. (2020) Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. 2020\.  Spectral clustering with graph neural networks for graph pooling. In _International conference on machine learning_. PMLR, 874–883.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2023) Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin-Yu Chen, and Sijia Liu. 2023.  Visual Prompting for Adversarial Robustness. In _ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. 1–5.  https://doi.org/10.1109/ICASSP49357.2023.10097245
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Du et al. (2022) Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022\.  A Survey of Vision-Language Pre-Trained Models. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 5436–5443.  https://doi.org/10.24963/ijcai.2022/762 Survey Track.
  * Fang et al. (2023) Taoran Fang, Yunchao Mercer Zhang, Yang Yang, Chunping Wang, and Lei CHEN. 2023\.  Universal Prompt Tuning for Graph Neural Networks. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=0LmWBhIYLi
  * Gao and Ji (2019) Hongyang Gao and Shuiwang Ji. 2019.  Graph u-nets. In _international conference on machine learning_. PMLR, 2083–2092.
  * Gaulton et al. (2012) Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. 2012\.  ChEMBL: a large-scale bioactivity database for drug discovery.  _Nucleic acids research_ 40, D1 (2012), D1100–D1107.
  * Han et al. (2021) Xueting Han, Zhenhuan Huang, Bang An, and Jing Bai. 2021\.  Adaptive transfer learning on graph neural networks. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 565–574.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hu et al. (2020b) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2020b.  Strategies for Pre-training Graph Neural Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=HJlWWJSFDH
  * Hu et al. (2020a) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020a.  Gpt-gnn: Generative pre-training of graph neural networks. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1857–1867.
  * Huang et al. (2023) Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy Liang, and Jure Leskovec. 2023.  PRODIGY: Enabling In-context Learning Over Graphs. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=pLwYhNNnoR
  * Khasahmadi et al. (2020) Amir Hosein Khasahmadi, Kaveh Hassani, Parsa Moradi, Leo Lee, and Quaid Morris. 2020\.  Memory-Based Graph Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=r1laNeBYPB
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).  http://arxiv.org/abs/1412.6980
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Variational Graph Auto-Encoders.  _NIPS Workshop on Bayesian Deep Learning_ (2016).
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=SJU4ayYgl
  * Lee et al. (2019) Junhyun Lee, Inyeop Lee, and Jaewoo Kang. 2019.  Self-attention graph pooling. In _International conference on machine learning_. PMLR, 3734–3743.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021\.  The Power of Scale for Parameter-Efficient Prompt Tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045–3059.  https://doi.org/10.18653/v1/2021.emnlp-main.243
  * Li et al. (2021) Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. 2021.  Training graph neural networks with 1000 layers. In _International conference on machine learning_. PMLR, 6437–6449.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 4582–4597.  https://doi.org/10.18653/v1/2021.acl-long.353
  * Li et al. (2023) Yizhe Li, Yu-Lin Tsai, Chia-Mu Yu, Pin-Yu Chen, and Xuebin Ren. 2023.  Exploring the benefits of visual prompting in differential privacy. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 5158–5167.
  * Liu et al. (2021) Huihui Liu, Yiding Yang, and Xinchao Wang. 2021.  Overcoming catastrophic forgetting in graph neural networks. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 8653–8661.
  * Liu et al. (2020) Meng Liu, Hongyang Gao, and Shuiwang Ji. 2020.  Towards deeper graph neural networks. In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_. 338–348.
  * Liu et al. (2023c) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023c.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _Comput. Surveys_ 55, 9 (2023), 1–35.
  * Liu et al. (2023a) Weihuang Liu, Xi Shen, Chi-Man Pun, and Xiaodong Cun. 2023a.  Explicit visual prompting for low-level structure segmentations. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 19434–19445.
  * Liu et al. (2023d) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023d.  GPT understands, too.  _AI Open_ (2023).
  * Liu et al. (2023b) Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. 2023b.  Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In _Proceedings of the ACM Web Conference 2023_. 417–428.
  * Ma et al. (2019) Yao Ma, Suhang Wang, Charu C Aggarwal, and Jiliang Tang. 2019\.  Graph convolutional networks with eigenpooling. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 723–731.
  * Ma et al. (2020) Zheng Ma, Junyu Xuan, Yu Guang Wang, Ming Li, and Pietro Liò. 2020.  Path integral based convolution and pooling for graph neural networks.  _Advances in Neural Information Processing Systems_ 33 (2020), 16421–16433.
  * Mayr et al. (2018) Andreas Mayr, Günter Klambauer, Thomas Unterthiner, Marvin Steijaert, Jörg K Wegner, Hugo Ceulemans, Djork-Arné Clevert, and Sepp Hochreiter. 2018.  Large-scale comparison of machine learning methods for drug target prediction on ChEMBL.  _Chemical science_ 9, 24 (2018), 5441–5451.
  * Nt and Maehara (2019) Hoang Nt and Takanori Maehara. 2019.  Revisiting graph neural networks: All we have is low-pass filters.  _arXiv preprint arXiv:1905.09550_ (2019).
  * Oh et al. (2023) Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. 2023\.  BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 24224–24235.
  * Oono and Suzuki (2020) Kenta Oono and Taiji Suzuki. 2020.  Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=S1ldO2EFPr
  * Qiu et al. (2020a) Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020a.  Gcc: Graph contrastive coding for graph neural network pre-training. In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_. 1150–1160.
  * Qiu et al. (2020b) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020b.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ 63, 10 (2020), 1872–1897.
  * Sterling and Irwin (2015) Teague Sterling and John J Irwin. 2015.  ZINC 15–ligand discovery for everyone.  _Journal of chemical information and modeling_ 55, 11 (2015), 2324–2337.
  * Sun et al. (2019) Fan-Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. 2019\.  InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization. In _International Conference on Learning Representations_.
  * Sun et al. (2022) Mingchen Sun, Kaixiong Zhou, Xin He, Ying Wang, and Xin Wang. 2022.  Gppt: Graph pre-training and prompt tuning to generalize graph neural networks. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 1717–1727.
  * Sun et al. (2023) Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. 2023.  All in One: Multi-Task Prompting for Graph Neural Networks. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ (¡conf-loc¿, ¡city¿Long Beach¡/city¿, ¡state¿CA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) _(KDD ’23)_. Association for Computing Machinery, New York, NY, USA, 2120–2131.  https://doi.org/10.1145/3580305.3599256
  * Suresh et al. (2021) Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. 2021\.  Adversarial graph augmentation to improve graph contrastive learning.  _Advances in Neural Information Processing Systems_ 34 (2021), 15920–15933.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _International conference on machine learning_. PMLR, 6105–6114.
  * Tsao et al. (2024) Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, Sijia Liu, and Tsung-Yi Ho. 2024.  AutoVP: An Automated Visual Prompting Framework and Benchmark. In _The Twelfth International Conference on Learning Representations_.
  * Veličković et al. (2019) Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019.  Deep Graph Infomax. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rklz9iAcKQ
  * Wu et al. (2019) Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. 2019\.  Simplifying graph convolutional networks. In _International conference on machine learning_. PMLR, 6861–6871.
  * Wu et al. (2023) Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. 2023\.  Demystifying Oversmoothing in Attention-Based Graph Neural Networks. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=Kg65qieiuB
  * Wu et al. (2018) Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. 2018\.  MoleculeNet: a benchmark for molecular machine learning.  _Chemical science_ 9, 2 (2018), 513–530.
  * Xia et al. (2022a) Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. 2022a.  Simgrace: A simple framework for graph contrastive learning without data augmentation. In _Proceedings of the ACM Web Conference 2022_. 1070–1079.
  * Xia et al. (2022b) Jun Xia, Jiangbin Zheng, Cheng Tan, Ge Wang, and Stan Z Li. 2022b.  Towards effective and generalizable fine-tuning for pre-trained molecular graph models.  _bioRxiv_ (2022), 2022–02.
  * Xia et al. (2022c) Jun Xia, Yanqiao Zhu, Yuanqi Du, and Stan Z Li. 2022c.  A survey of pretraining on graphs: Taxonomy, methods, and applications.  _arXiv preprint arXiv:2202.07893_ (2022).
  * Xing et al. (2023) Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, Peng Wang, and Yanning Zhang. 2023.  Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model.  _IEEE Transactions on Multimedia_ (2023), 1–13.  https://doi.org/10.1109/TMM.2023.3291588
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019\.  How Powerful are Graph Neural Networks?. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=ryGs6iA5Km
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018.  Representation learning on graphs with jumping knowledge networks. In _International conference on machine learning_. PMLR, 5453–5462.
  * Xu et al. (2021) Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. 2021.  Self-supervised graph-level representation learning with local and global structure. In _International Conference on Machine Learning_. PMLR, 11548–11558.
  * Ying et al. (2018) Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. 2018\.  Hierarchical graph representation learning with differentiable pooling.  _Advances in neural information processing systems_ 31 (2018).
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020\.  Graph contrastive learning with augmentations.  _Advances in neural information processing systems_ 33 (2020), 5812–5823.
  * You et al. (2022) Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. 2022\.  Bringing your own view: Graph contrastive learning without prefabricated data augmentations. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 1300–1309.
  * Yuan and Ji (2020) Hao Yuan and Shuiwang Ji. 2020.  StructPool: Structured Graph Pooling via Conditional Random Fields. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=BJxg_hVtwH
  * Zhang et al. (2020) Liang Zhang, Xudong Wang, Hongsheng Li, Guangming Zhu, Peiyi Shen, Ping Li, Xiaoyuan Lu, Syed Afaq Ali Shah, and Mohammed Bennamoun. 2020.  Structure-feature based graph self-adaptive pooling. In _Proceedings of The Web Conference 2020_. 3098–3104.
  * Zhang et al. (2021) Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. 2021.  Motif-based graph self-supervised learning for molecular property prediction.  _Advances in Neural Information Processing Systems_ 34 (2021), 15870–15882.
  * Zhou and Cao (2021) Fan Zhou and Chengtai Cao. 2021.  Overcoming catastrophic forgetting in graph neural networks with experience replay. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4714–4722.
  * Zhou et al. (2021) Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. 2021.  Understanding and resolving performance degradation in deep graph convolutional networks. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 2728–2737.
  * Zitnik et al. (2019) Marinka Zitnik, Rok Sosič, Marcus W Feldman, and Jure Leskovec. 2019.  Evolution of resilience in protein interactomes across the tree of life.  _Proceedings of the National Academy of Sciences_ 116, 10 (2019), 4426–4433.

##  Appendix A Proof for Theorem 1

######  Theorem 1 (Universal Capability of SUPT).

Let $f$ be a pre-trained GNN model and $\mathcal{G}:(X,A)$ be an input graph
with sets of nodes and edges, $\mathcal{V}$ and $\mathcal{E}$. Given any
prompting function $\psi_{t}(\cdot)$, if a prompted graph
$\hat{\mathcal{G}}:(\hat{X}\in\mathbb{X},\hat{A}\in\mathbb{A})$ is in the
candidate space $\mathcal{G}^{*}=\Psi_{t}(\mathcal{G})$, then there exists a
vector $\hat{p}$ such that for any non-empty subset of nodes $\mathcal{S}$
(i.e., $\mathcal{S}\subseteq\mathcal{V}$ and $|\mathcal{S}|\geq 1$), the
updated features can be defined as:

$X^{\prime}=\begin{cases}X_{s}+\hat{p}&\text{if }s\in\mathcal{S}\\\
X_{s}&\text{if }s\notin\mathcal{S}\end{cases}$

such that there exists $X^{\prime}$ satisfying:

$f(X^{\prime},A)=f(\hat{X},\hat{A})$

To demonstrate Theorem 1, we detail the architecture of the pre-trained GNN
$f_{\theta}$. For ease of analysis, we initially consider $f_{\theta}$ to be a
single-layer Graph Isomorphism Network (GIN) (Xu et al., 2019), incorporating
a linear transformation, as done in Fang et al.. The embedding is calculated
through the pre-trained GNN model $f_{\theta}$ as:

(8) $\displaystyle Z$ $\displaystyle=(A+(1+\epsilon)\cdot I)\cdot X\cdot W$
(9) $\displaystyle z_{\mathcal{G}}$
$\displaystyle=\sum_{v_{i}\in\mathcal{V}}z_{i}$

where a parameter $\epsilon$ and $W$ for linear projection are elements of
parameter set $\theta$. They have been pre-trained and then fixed in prompt
tuning process. With these assumptions in place, we demonstrate that adding a
prompt feature vector to a subset of nodes can theoretically be equivalent to
the Graph Prompt Feature (GPF) approach (Fang et al., 2023), which
incorporates prompts into all nodes. Theorem 1 is equivalent to Proposition 1.

######  Proposition 1.

Let $f$ be a pre-trained GNN model and $\mathcal{G}:(X,A)$ be an input graph
with sets of nodes and edges, $\mathcal{V}$ and $\mathcal{E}$. There exists a
vector $\hat{p}$ such that for any non-empty subset of nodes $\mathcal{S}$
(i.e., $\mathcal{S}\subseteq\mathcal{V}$ and $|\mathcal{S}|\geq 1$), the
updated features can be defined as:

$X^{\prime}=\begin{cases}X_{s}+\hat{p}&\text{if }s\in\mathcal{S}\\\
X_{s}&\text{if }s\notin\mathcal{S}\end{cases}$

such that there exists $X^{\prime}$ satisfying:

$f(X^{\prime},A)=f(X+p,A)$

where $p$ is a uniform feature vector added to all nodes in $X$.

###### Proof.

For $p=[\rho_{1},...,\rho_{F}]\in\mathbb{R}^{1\times F}$, we have:

(10) $\displaystyle Z_{p}$ $\displaystyle=(A+(1+\epsilon)\cdot
I)\cdot(X+[1]^{N}\cdot p)\cdot W$ (11) $\displaystyle=(A+(1+\epsilon)\cdot
I)\cdot X\cdot W+(A+(1+\epsilon)\cdot I)\cdot[1]^{N}\cdot p\cdot W$ (12)
$\displaystyle=Z+(A+(1+\epsilon)\cdot I)\cdot[1]^{N}\cdot p\cdot W$ (13)
$\displaystyle=Z+[d_{i}+1+\epsilon]^{N}\cdot p\cdot W$

where $[1]^{N}\in\mathbb{R}^{N\times 1}$ and
$[d_{i}+1+\epsilon]^{N}\in\mathbb{R}^{N\times 1}$ are column vectors and
$d_{i}$ denotes the degree of node $v_{i}$.

For a subset of nodes $\mathcal{S}$ and SUPT
$\hat{p}=[\hat{\rho_{1}},...,\hat{\rho_{F}}]\in\mathbb{R}^{1\times F}$, we
have:

(14) $\displaystyle Z_{\hat{p}}$ $\displaystyle=(A+(1+\epsilon)\cdot
I)\cdot(X+[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p})\cdot W$ (15)
$\displaystyle=(A+(1+\epsilon)\cdot I)\cdot X\cdot W+(A+(1+\epsilon)\cdot
I)\cdot[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p}\cdot W$ (16)
$\displaystyle=Z+(A+(1+\epsilon)\cdot
I)\cdot[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p}\cdot W$

where $\mathbf{1}_{\mathcal{S}}(v)$ denotes the indicator function defined to
be 1 if $v\in\mathcal{S}$ and 0 otherwise.

To obtain the identical representation $z_{\mathcal{G}}$, we have:

(17)
$z_{\mathcal{G},\hat{p}}=z_{\mathcal{G},p}\rightarrow\text{Sum}(Z_{\hat{p}})=\text{Sum}(Z_{p})$

where $\text{Sum}(M)=\sum_{i}m_{i}$ denotes the summation of each row of the
matrix. We simplify the above equation as:

(18) $\displaystyle z_{\mathcal{G},\hat{p}}=z_{\mathcal{G},p}$ (19)
$\displaystyle\rightarrow$
$\displaystyle\text{Sum}(Z_{\hat{p}})=\text{Sum}(Z_{p})$ (20)
$\displaystyle\rightarrow$ $\displaystyle\text{Sum}(Z+(A+(1+\epsilon)\cdot
I)\cdot[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p}\cdot W)$ (21)
$\displaystyle=\text{Sum}(Z+[d_{i}+1+\epsilon]^{N}\cdot p\cdot W)$ (22)
$\displaystyle\rightarrow$ $\displaystyle\text{Sum}((A+(1+\epsilon)\cdot
I)\cdot[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p}\cdot W)$ (23)
$\displaystyle=\text{Sum}([d_{i}+1+\epsilon]^{N}\cdot p\cdot W)$

Then, we calculate $\Delta
z_{\mathcal{G},p}=\text{Sum}([d_{i}+1+\epsilon]^{N}\cdot p\cdot
W)\in\mathbb{R}^{d^{\prime}}$ as:

(24) $\displaystyle\Delta z_{\mathcal{G},p}^{i}$
$\displaystyle=\sum_{j=1}^{d}\sum_{k=1}^{N}(d_{i}+1+\epsilon)\cdot\rho_{j}\cdot
W_{j,i}$ (25)
$\displaystyle=\sum_{j=1}^{d}(D+N+N\cdot\epsilon)\cdot\rho_{j}\cdot W_{j,i}$

where $\rho_{j}$ for $j\in[1,d]$ denotes $j$-th parameter of GPF feature
vector, $\Delta z_{\mathcal{G},p}^{i}$ denotes $i$-dimension in $\Delta
z_{\mathcal{G},p}^{i}$, and $D=\sum_{k=1}^{N}d_{k}$ is the sum of degree of
all nodes. Let $(A+(1+\epsilon)\cdot
I)\cdot[\mathbf{1}_{\mathcal{S}}(v)]^{N}\cdot\hat{p}=B\in\mathbb{R}^{N\times
d}$, then we have:

(26) $\Delta
z_{\mathcal{G},\hat{p}}^{i}=\sum_{j=1}^{d}(\sum_{k=1}^{N}\beta_{k,j})\cdot
W_{j,i}$

where $\beta_{k,j}$ for $k\in[1,N],j\in[1,d]$ denotes the learnable parameter
in $B$, which is the parameter of SUPT prompt. Then we have:

(27) $\displaystyle z_{\mathcal{G},\hat{p}}^{i}=z_{\mathcal{G},p}^{i},\text{
for every }i\in[1,d^{\prime}]$ (28) $\displaystyle\rightarrow$
$\displaystyle\Delta z_{\mathcal{G},\hat{p}}^{i}=\Delta z_{\mathcal{G},p}^{i}$
(29) $\displaystyle\rightarrow$
$\displaystyle\sum_{k=1}^{N}\beta_{k,j}=(D+N+N\cdot\epsilon)\alpha_{j},\ \
j\in[1,d]$

Therefore, there exists a SUPT prompt $\hat{p}$ that satisfies the above
conditions to be equivalent to a GPF prompt $p$ for the pre-trained GNN model
$f_{\theta}$. ∎

Theorem 1 and Proposition 1 imply that SUPT can, in theory, achieve
equivalence with GPF, and thus be equivalent to any prompting function. Fang
et al. proved that GPF satisfies Proposition 30 and 3 below.

######  Proposition 2.

Given a pre-trained GNN model $f$, an input graph $\mathcal{G}:(X,A)$, for any
graph-level transformation $g:\mathbb{G}\rightarrow\mathbb{G}$, there exists a
GPF extra feature vector $p$ that satisfies:

(30) $f(X+p,A)=f(g(X,A))$

######  Proposition 3.

Given an input graph $\mathcal{G}:(X,A)$, an arbitrary graph-level
transformation $g:\mathbb{G}\rightarrow\mathbb{G}$ can be decoupled to a
series of following transformations:

  * •

Feature transformations. Modifying the node features and generating the new
feature matrix $X^{\prime}=g_{ft}(X)$.

  * •

Link transformations. Adding or removing edges and generating the new
adjacency matrix $A^{\prime}=g_{lt}(A)$

  * •

Isolated component transformations. Adding or removing isolated components
(sub-graphs) and generating the new adjacency matrix and feature matrix
$X^{\prime},A^{\prime}=g_{ict}(X,A)$.

##  Appendix B Details on Training and Hyperparameters

For SUPT, as defined in Equation (5), the hyperparameter $k$, which denotes
the number of subgraphs, was selected from the set {1, 2, 3, 4, 5}. In the
case of SUPT${}_{\text{hard}}$, the pooling ratio $r$, as outlined in Equation
(6), was chosen from the options {0.2, 0.4, 0.6}. We fixed the number of hops
$m$ at 1, leading to the calculation of $\alpha$ in Equation (5) via a GCN.
The weight decay parameters were determined from the set {1e-3, 1e-4, 1e-5},
learning rates from {1e-3, 5e-4, 1e-4}, the number of layers of projection
head from {1, 2, 3, 4}, the global pooling from {sum, mean}, and the number of
epochs from {50, 100, 300}.

| Tuning | PPI | BBBP | Tox21 | ToxCast | SIDER | ClinTox | MUV | HIV | BACE | Avg.
---|---|---|---|---|---|---|---|---|---|---|---
Infomax | FT | 71.29 | 67.55 | 78.57 | 65.16 | 63.34 | 70.06 | 81.42 | 77.71 | 81.32 | 72.94
GPF | 78.87 | 66.58 | 78.13 | 65.96 | 65.64 | 74.14 | 80.39 | 76.67 | 83.24 | 74.40
GPF-plus | 78.85 | 67.03 | 78.50 | 65.83 | 65.89 | 73.61 | 79.94 | 75.60 | 82.96 | 74.25
SUPT${}_{\text{hard}}$ | 79.64 | 67.29 | 79.14 | 66.48 | 65.78 | 75.65 | 81.51 | 77.96 | 84.05 | 75.28
SUPT${}_{\text{soft}}$ | 79.45 | 67.63 | 78.98 | 66.53 | 65.88 | 76.08 | 81.59 | 77.94 | 84.01 | 75.34
AttrMasking | FT | 73.93 | 66.33 | 78.28 | 65.34 | 66.77 | 74.46 | 81.78 | 77.90 | 80.94 | 73.97
GPF | 81.47 | 67.49 | 78.66 | 66.09 | 68.19 | 72.72 | 81.44 | 77.09 | 84.77 | 75.32
GPF-plus | 81.43 | 67.32 | 78.83 | 66.53 | 68.53 | 74.17 | 81.27 | 77.64 | 85.01 | 75.64
SUPT${}_{\text{hard}}$ | 81.90 | 68.39 | 79.50 | 66.79 | 68.82 | 75.25 | 81.81 | 79.10 | 84.98 | 76.28
SUPT${}_{\text{soft}}$ | 81.91 | 68.37 | 79.73 | 67.02 | 68.68 | 77.36 | 81.93 | 79.15 | 85.22 | 76.60
ContextPred | FT | 72.10 | 69.65 | 78.29 | 66.39 | 64.45 | 73.71 | 82.36 | 79.20 | 84.66 | 74.53
GPF | 80.21 | 69.04 | 79.90 | 67.68 | 65.31 | 73.15 | 84.22 | 77.77 | 85.03 | 75.81
GPF-plus | 80.28 | 68.59 | 79.95 | 67.72 | 66.32 | 72.73 | 84.40 | 78.10 | 84.50 | 75.84
SUPT${}_{\text{hard}}$ | 80.85 | 70.18 | 80.00 | 68.40 | 66.37 | 74.90 | 84.54 | 79.42 | 85.21 | 76.65
SUPT${}_{\text{soft}}$ | 80.86 | 70.06 | 80.12 | 68.50 | 66.48 | 75.55 | 84.67 | 79.39 | 85.27 | 76.77
GCL | FT | 67.76 | 69.49 | 73.35 | 62.54 | 60.63 | 75.17 | 69.78 | 78.26 | 75.51 | 70.27
GPF | 67.91 | 68.11 | 72.15 | 60.73 | 61.00 | 69.48 | 70.81 | 75.12 | 76.67 | 69.11
GPF-plus | 66.71 | 67.77 | 73.06 | 61.47 | 61.57 | 68.89 | 70.54 | 75.90 | 77.84 | 69.31
SUPT${}_{\text{hard}}$ | 68.59 | 69.68 | 73.44 | 63.35 | 62.51 | 69.47 | 70.64 | 76.22 | 79.17 | 70.34
SUPT${}_{\text{soft}}$ | 69.57 | 69.79 | 73.40 | 63.42 | 62.84 | 71.21 | 70.92 | 76.44 | 78.92 | 70.72
EdgePred | FT | 71.54 | 66.56 | 78.67 | 66.29 | 64.35 | 69.07 | 79.67 | 77.44 | 80.90 | 72.72
GPPT | 56.23 | 64.13 | 66.41 | 60.34 | 54.86 | 59.81 | 63.05 | 60.54 | 70.85 | 61.80
GPPT${}_{\text{w/o ol}}$ | 76.85 | 69.43 | 78.91 | 64.86 | 60.94 | 62.15 | 82.06 | 73.19 | 70.31 | 70.97
GraphPrompt | 49.48 | 69.29 | 68.09 | 60.54 | 58.71 | 55.37 | 62.35 | 59.31 | 67.70 | 61.20
GPF | 79.96 | 69.36 | 79.69 | 65.70 | 66.71 | 67.92 | 82.61 | 77.53 | 79.76 | 74.36
| GPF-plus | 79.88 | 69.31 | 79.91 | 65.66 | 66.61 | 69.58 | 82.74 | 77.77 | 80.91 | 74.71
| SUPT${}_{\text{hard}}$ | 80.20 | 69.51 | 80.76 | 66.44 | 66.85 | 69.97 | 82.94 | 77.79 | 81.72 | 75.13
| SUPT${}_{\text{soft}}$ | 80.19 | 69.93 | 80.90 | 66.65 | 66.95 | 70.41 | 82.95 | 77.95 | 81.44 | 75.26

Table 6. Comparison of test ROC-AUC performance in full-shot scenarios across
protein function prediction and molecular prediction benchmarks using various
pre-training strategies, highlighted in blue for performances superior to
fine-tuning (FT) and in red otherwise.