All in One: Multi-task Prompting for Graph Neural Networks
Recently, ''pre-training and fine-tuning'' has been adopted as a standard
workflow for many graph tasks since it can take general graph knowledge to
relieve the lack of graph annotations from each application. However, graph
tasks with node level, edge level, and graph level are far diversified, making
the pre-training pretext often incompatible with these multiple tasks. This gap
may even cause a ''negative transfer'' to the specific application, leading to
poor results. Inspired by the prompt learning in natural language processing
(NLP), which has presented significant effectiveness in leveraging prior
knowledge for various NLP tasks, we study the prompting topic for graphs with
the motivation of filling the gap between pre-trained models and various graph
tasks. In this paper, we propose a novel multi-task prompting method for graph
models. Specifically, we first unify the format of graph prompts and language
prompts with the prompt token, token structure, and inserting pattern. In this
way, the prompting idea from NLP can be seamlessly introduced to the graph
area. Then, to further narrow the gap between various graph tasks and
state-of-the-art pre-training strategies, we further study the task space of
various graph applications and reformulate downstream problems to the
graph-level task. Afterward, we introduce meta-learning to efficiently learn a
better initialization for the multi-task prompt of graphs so that our prompting
framework can be more reliable and general for different tasks. We conduct
extensive experiments, results from which demonstrate the superiority of our
method.
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, Jihong Guan
# All in One: Multi-Task Prompting for Graph Neural Networks

###### Abstract.

Recently, "pre-training and fine-tuning" has been adopted as a standard workflow for many graph tasks since it can take general graph knowledge to relieve the lack of graph annotations from each application. However, graph tasks with node level, edge level, and graph level are far diversified, making the pre-training pretext often incompatible with these multiple tasks. This gap may even cause a "negative transfer" to the specific application, leading to poor results. Inspired by the prompt learning in natural language processing (NLP), which has presented significant effectiveness in leveraging prior knowledge for various NLP tasks, we study the prompting topic for graphs with the motivation of filling the gap between pre-trained models and various graph tasks. In this paper, we propose a novel multi-task prompting method for graph models. Specifically, we first unify the format of graph prompts and language prompts with the prompt token, token structure, and inserting pattern. In this way, the prompting idea from NLP can be seamlessly introduced to the graph area. Then, to further narrow the gap between various graph tasks and state-of-the-art pre-training strategies, we further study the task space of various graph applications and reformulate downstream problems to the graph-level task. Afterward, we introduce meta-learning to efficiently learn a better initialization for the multi-task prompt of graphs so that our prompting framework can be more reliable and general for different tasks. We conduct extensive experiments, results from which demonstrate the superiority of our method.

pre-training; prompt tuning; graph neural networks +
Footnote †: journal: Computer Science and Technology, Tongji Universityjhguan@tongji.edu.cn

+
Footnote †: journal: Computer Science and Technology, Tongji Universityjhguan@tongji.
A promising solution to the above problems is to extend "pre-training and fine-tuning" to "pre-training, prompting, and fine-tuning". Prompt learning is a very attractive idea derived from natural language processing (NLP) and has shown notable effectiveness in generalizing pre-trained language models to a wide range of language applications [20]. Specifically, a language prompt refers to a piece of text appended to the rear of an input text. For example, a sentiment task like "_KDD2023 will witness many high-quality papers_. I feel so [MASK]" can be easily transferred to a word prediction task via a preset prompt ("I feel so [MASK]"). It is highly expected that the language model may predict "[MASK]" as "excited" rather than "upset" without further optimizing parameters for the new sentiment task because this model has already been pre-trained via the pretext of masked words prediction and contains some useful knowledge to answer this question. By this means, some downstream objectives can be naturally aligned with the pre-training target. Inspired by the success of the language prompt, we hope to introduce the same idea to graphs. As shown in Figure 1, prompt tuning in the graph domain is to seek some light-weighted prompt, keep the pre-training model frozen, and use the prompt to reformulate downstream tasks in line with the pre-training task. In this way, the pre-trained model can be easily applied to downstream applications with highly efficient fine-tuning or even without any fine-tuning. This is particularly useful when the downstream task is a few-shot setting.

However, designing the graph prompt is more intractable than language prompts. First, classic language prompts are usually some preset phrases or learnable vectors attached at the end of input texts. As shown in Figure 2, we only need to consider the content for the language prompt, whereas the graph prompt not only requires the prompt "content" but also needs to know how to organize these prompt tokens and how to insert the prompt into the original graph, both of which are undefined problems.

Second, there is a huge difficulty in reconciling downstream problems to the pre-training task. In the NLP area, we usually pre-train a language model via masked prediction and then transfer it to various applications like question answering [22], sentiment classification [17]. The underlying support [21] is that these language tasks usually share a large overlapping task sub-space, making a masked language task easily transferred to other applications. However, how much does the same observation exist (if truly exists) in graph learning? It is crucial but difficult to decide on an appropriate pre-training task and reformulate downstream tasks to improve the capability of model generalization. Currently, we only find very few works [27] studying the graph prompt issue. However, it can only deal with a single-type task (e.g., node classification) using a specific pretext (e.g., edge prediction), which is far from addressing the multi-task setting with different-level tasks.

Last but not least, learning a reliable prompt usually needs huge manpower and is more sensitive to prompt initialization in the multi-task setting [18]. Although there are some works [14; 38] in the NLP area trying to initialize the prompt via hand-crafted content or some discrete features, these methods are task-bounded, which is not sufficient when we confront a new task. This problem may be even worse in our multi-task graph area since graph features vary a lot in different domains and tasks.

_Presented work._ To further fill the gap between graph pre-training and downstream tasks, we introduce the prompt method from NLP to graphs under the multi-task background. Specifically, to address the first challenge, we propose to unify the format of the language prompt and graph prompt in one way so that we can smoothly transfer the prompt idea from NLP to graphs, then we design the graph prompt from prompt tokens, token structures, and prompt inserting patterns. To address the second challenge, we first study the task subspace in graphs and then propose to reformulate node-level and edge-level tasks to graph-level tasks by induced graphs from original graphs. To address the third challenge, we introduce the meta-learning technique over multiple tasks to learn better prompts. We carefully evaluate our method with other approaches and the experimental results extensively demonstrate our advantages.

_Contributions:_

* We unify the prompt format in the language area and graph area, and further propose an effective graph prompt for multi-task settings (section 3.3).
* We propose an effective way to reformulate node-level and edge-level tasks to graph-level tasks, which can further match many pre-training pretexts (section 3.2).
* We introduce the meta-learning technique to our graph prompting study so that we can learn a reliable prompt for improving the multi-task performance (section 3.4).
* We carefully analyze why our method works (section 3.5) and confirm the effectiveness of our method via extensive experiments (section 4).

Figure 1. Fine-tuning, Pre-training, and Prompting.

Figure 2. Our graph prompt inspired by the language prompt.
## 2. Background

**Graph Neural Networks.** Graph neural networks (GNNs) have presented powerful expressiveness in many graph-based applications (Golovolovolov et al., 2012; Kipf and Welling, 2015; Kipf and Welling, 2016; Kipf and Welling, 2017). The nature of most GNNs is to capture the underlying message-passing patterns for graph representation. To this end, there are many effective neural network structures proposed such as graph attention network (GAT) (Gatani and Vedaldi, 2017), graph convolution network (GCN) (Srivastava et al., 2014), Graph Transformer (Kipf and Welling, 2016). Recent works also consider how to make graph learning more adaptive when data annotation is insufficient or how to transfer the model to a new domain, which triggered many graph pre-training studies instead of traditional supervised learning.

**Graph Pre-training.** Graph pre-training (Golovolov et al., 2012) aims to learn some general knowledge for the graph model with easily accessible information to reduce the annotation costs of new tasks. Some effective pre-training strategies include node-level comparison like GCA (Grover et al., 2017), edge-level pretext like edge prediction (Golov et al., 2012), and graph-level contrastive learning such as GraphCL (Srivastava et al., 2014) and SimGRACE (Srivastava et al., 2014). In particular, GraphCL minimizes the distance between a pair of graph-level representations for the same graph with different augmentations whereas SimGRACE tries to perturb the graph model parameter spaces and narrow down the gap between different perturbations for the same graph. These graph-level strategies perform more effectively in graph knowledge learning (Golovolovolov et al., 2012) and are the default strategies of this paper.

**Prompt Learning & Motivations.** Intuitively, the above graph-level pre-training strategies have some intrinsic similarities with the language-masked prediction task: aligning two graph views generated by node/edge/feature mask or other perturbations is very similar to predicting some vacant "blanks" on graphs. That inspires us to further consider: why can't we use a similar format prompt for graphs to improve the generalization of graph neural networks? Instead of fine-tuning a pre-trained model with an adaptive task head, prompt learning aims to reformulate input data to fit the pretext (Golovolovolov et al., 2012). Many effective prompt methods are firstly proposed in the NLP area, including some hand-crafted prompts like GPT-3 (Golovolov et al., 2012), discrete prompts like (Golov et al., 2012; Kipf and Welling, 2016), and trainable prompts in the continuous spaces like (Golov et al., 2012; Kipf and Welling, 2017). Despite significant results achieved, prompt-based methods have been rarely introduced in graph domains yet. We only find very few works like GPPT (Golovolov et al., 2012), trying to design prompts for graphs. Unfortunately, most of them are very limited and are far from sufficient to meet the multi-task demands.

## 3. Multi-Task Prompting on Graphs

### Overview of Our Framework

#### Objective

In this paper, we aim to learn a prompt graph that can be inserted into the original graph, through which we wish to further bridge the gap between a graph pre-training strategy and multiple downstream tasks, and further relieve the difficulties of transferring prior knowledge to different domains.

**Overview:** To achieve our goal, we propose a novel multi-task prompting framework for graph models. First, we unify various graph tasks in the same format and reformulate these downstream tasks as graph-level tasks. Second, with the unified graph-level instances, we further narrow down the gap among multiple tasks by a novel prompt graph with learnable tokens, inner structures, and adaptive inserting patterns. Third, we build a meta-learning process to learn more adaptive graph prompts for multi-task settings. Next, we elaborate on the main components.

### Reformulating Downstream Tasks

#### 3.2.1. Why Reformulate Downstream Tasks

The success of the traditional "pre-training and fine-tuning" framework in the NLP area largely lies in the fact that the pre-training task and downstream tasks share some common intrinsic task subspace, making the pre-training knowledge transferable to other downstream tasks (Figure (a)a). However, things are a little complicated in the graph domain since graph-related tasks are far from similar. As shown in Figure (b)b, it is far-fetched to treat the edge-level task and the node-level task as the same one because node-level operations and edge-level operations are far more different (Golov et al., 2012). This gap limits the performance of pre-training models and might even cause negative transfer (Golov et al., 2012). The same problem also happens in our "pre-training, prompting, and fine-tuning" framework since we aim to learn a graph prompt for multiple tasks, which means we need to further narrow down the gap between these tasks by reformulating different graph tasks in a more general form.

#### 3.2.2. Why Reformulate to the Graph Level

With the above motivation, we revisit the potential task space on the graph and find their hierarchical relation as shown in Figure (b)b. Intuitively, many node-level operations such as "changing node features", "delete/add a node", or edge-level operations such as "add/delete an edge", can be treated as some basic operations at the graph level. For example, "delete a subgraph" can be treated as "delete nodes and edges". Compared with node-level and edge-level tasks, graph-level tasks are more general and contain the largest overlapping task sub-spaces for knowledge transfer, which has been adopted as the mainstream task in many graph pre-training models (Golov et al., 2012; Srivastava et al., 2014; Srivastava et al., 2014). This observation further inspires us to reformulate downstream tasks to look like the graph-level task and then leverage our prompting model to match graph-level pre-training strategies.

#### 3.2.3. How to Reformulate Downstream Tasks

Specifically, we reformulate node-level and edge-level tasks to graph-level tasks by building induced graphs for nodes and edges, respectively. As shown in Figure (a)a, an induced graph for a target node means its local area in the network within \(\tau\) distance, which is also known as its \(\tau\)-ego network. This subgraph preserves the node's local structure by neighboring node connections and its semantic context by neighboring node features, which is the main scope of most graph neural

Figure 3. Task space in NLP and graph
encoders. When we treat the target node's label as this induced graph label, we can easily translate the node classification problem into graph classification; Similarly, we present an induced graph for a pair of nodes in Figure 4b. Here, the pair of nodes can be treated as a positive edge if there is an edge connecting them, or a negative edge if not. This subgraph can be easily built by extending this node pair to their \(\tau\) distance neighbors. We can reformulate the edge-level task by assigning the graph label with the edge label of the target node pair. Note that for unweighted graphs, the \(\tau\) distance is equal to \(\tau\)-hop length; for weighted graphs, the \(\tau\) distance refers to the shortest path distance, where the induced graph can be easily found by many efficient algorithms (Bahdan et al., 2017; Chen et al., 2018).

### Prompt Graph Design

#### 3.3.1. Prompting NLP and Graph in One Way

To seamlessly transfer the prompting idea from NLP to the graph domain, we propose to unify NLP Prompt and Graph Prompt in one perspective. Having compared the demand of NLP and graph area as shown in Figure 2, we found that the prompt in NLP and graph areas should contain at least three components: _(1) prompt token_, which contains the vectorized prompting information with the same size as the input word/node vector; _(2) token structure_, which indicates the connection of different tokens. In the NLP area, prompt tokens (a.k.a prompt words) are preset as a linear relation like a sub-sentence or a phrase; whereas in the graph domain, the connections of different tokens are non-linear and far more complicated than NLP prompts; _(3) inserting pattern_, which presents how to add the prompt to the input data. In the NLP area, the prompt is usually added in the front or the back end of the input sentences by default. However, in the graph area, there are no explicit positions like a sentence to joint graph prompt, making the graph prompting more difficult.

#### 3.3.2. Prompt Tokens

Let a graph instance be \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) where \(\mathcal{V}=\{v_{1},v_{2},\cdots,v_{N}\}\) is the node set containing \(N\) nodes; each node has a feature vector denoted by \(\mathbf{x}_{i}\in\mathbb{R}^{1\times d}\) for node \(v_{i}\); \(\mathcal{E}=\{(v_{i},v_{j})|v_{i},v_{j}\in\mathcal{V}\}\) is the edge set where each edge connects a pair of nodes in \(\mathcal{V}\). With the previous discussion, we here present our prompt graph as \(\mathcal{G}_{p}=(\mathcal{P},\mathcal{S})\) where \(\mathcal{P}=\{p_{1},p_{2},\cdots,p_{|\mathcal{P}|}\}\) denotes the set of prompt tokens and \(|\mathcal{P}|\) is the number of tokens. Each token \(p_{i}\in\mathcal{P}\) can be represented by a token vector \(\mathbf{p}_{i}\in\mathbb{R}^{1\times d}\) with the same size of node features in the input graph; Note that in practice, we usually have \(|\mathcal{P}|\ll N\) and \(|\mathcal{P}|\ll d_{h}\) where \(d_{h}\) is the size of the hidden layer in the pre-trained graph model. With these token vectors, the input graph can be reformulated by adding the \(j\)-th token to graph node \(v_{i}\) (e.g., \(\hat{\mathbf{x}}_{i}=\mathbf{x}_{i}+\mathbf{p}_{j}\)). Then, we replace the input features with the prompted features and send them to the pre-trained model for further processing.

#### 3.3.3. Token Structures

\(\mathcal{S}=\{(p_{i},p_{j})|p_{i},p_{j}\in\mathcal{P}\}\) is the token structure denoted by pair-wise relations among tokens. Unlike the NLP prompt, the token structure in the prompt graph is usually implicit. To solve this problem, we propose three methods to design the prompt token structures: (1) the first way is to learn tunable parameters:

\[\mathcal{A}=\bigcup_{\begin{subarray}{c}i=1\\ j=i+1\end{subarray}}^{|\mathcal{P}|-1}\{a_{ij}\}\]

where \(a_{ij}\) is a tunable parameter indicating how possible the token \(p_{i}\) and the token \(p_{j}\) should be connected; (2) the second way is to use the dot product of each prompt token pair and prune them according to the dot value. In this case, \((p_{i},p_{j})\in\mathcal{S}\) iff \(\sigma(p_{i},p_{j})<\delta\) where \(\sigma(\cdot)\) is a sigmoid function and \(\delta\) is a pre-defined threshold; (3) the third way is to treat the tokens as independent and then we have \(\mathcal{S}=\emptyset\).

#### 3.3.4. Inserting Patterns

Let \(\psi\) be the inserting function that indicates how to add the prompt graph \(\mathcal{G}_{p}\) to the input graph \(\mathcal{G}\), then the manipulated graph can be denoted as \(\mathcal{G}_{m}=\psi(\mathcal{G},\mathcal{G}_{p})\). We can define the inserting pattern as the dot product between prompt tokens and input graph nodes, and then use a tailored connection like \(\hat{\mathbf{x}}_{i}=\mathbf{x}_{i}+\sum_{k=1}^{|\mathcal{P}|}w_{ik}\mathbf{ p}_{k}\) where \(w_{ik}\) is a weighted value to prune unnecessary connections:

\[w_{ik}=\left\{\begin{array}{cc}\sigma(\mathbf{p}_{k}\cdot\mathbf{x}_{i}^{T} ),&\text{if }\sigma(\mathbf{p}_{k}\cdot\mathbf{x}_{i}^{T})>\delta\\ 0,&\text{otherwise}\end{array}\right. \tag{1}\]

As an alternative and special case, we can also use a more simplified way to get \(\hat{\mathbf{x}}_{i}=\mathbf{x}_{i}+\sum_{k=1}^{|\mathcal{P}|}\mathbf{p}_{k}\).

### Multi-task Prompting via Meta Learning

#### 3.4.1. Constructing Meta Prompting Tasks

Let \(\tau_{i}\) be the \(i\)-th task with supporting data \(\mathcal{D}_{\tau_{i}}^{s}\) and query data \(\mathcal{D}_{\tau_{i}}^{q}\); Specifically, for the graph classification task, \(\mathcal{D}_{\tau_{i}}^{s}\) and \(\mathcal{D}_{\tau_{i}}^{q}\) contain labeled graphs; for the node classification task, we generate an induced graph for each node as mentioned in section 3.2.3, align the graph label with the target node label, and treat this graph as a member in \(\mathcal{D}_{\tau_{i}}^{s}\) or \(\mathcal{D}_{\tau_{i}}^{q}\), for the edge classification task, we first generate edge induced graphs for training and testing and the edge label is up to its two endpoints.

#### 3.4.2. Applying Meta-learning to Graph Prompting

Let \(\theta\) be prompt parameters, \(\pi^{\pi}\) be the fixed parameters of the pre-trained graph backbone, and \(\phi\) be the tasker's parameters. We use \(f_{\phi,\phi|\pi^{\pi}}\) to denote the pipeline with prompt graph (\(\theta\)), pre-trained model (\(\pi^{*}\), fixed), and downstream tasker (\(\phi\)). Let \(\mathcal{L}_{\mathcal{D}}(f)\) be the task loss with pipline \(f\) on data \(\mathcal{D}\). Then for each task \(\tau_{i}\), the corresponding parameters

Figure 4. Induced graphs for nodes and edges
can be updated as follows:

\[\begin{split}\theta_{i}^{k}&=\theta_{i}^{k-1}-\alpha \nabla_{\theta_{i}^{k-1}}\mathcal{L}_{\mathcal{D}_{i_{t}}^{n}}\left(\theta_{ \theta_{i}^{k-1},\phi_{i}^{k-1}|\pi^{*}}\right)\\ \phi_{i}^{k}&=\phi_{i}^{k-1}-\alpha\nabla_{\phi_{i}^ {k-1}}\mathcal{L}_{\mathcal{D}_{i_{t}}^{n}}\left(\theta_{\theta_{i}^{k-1},\phi_ {i}^{k-1}|\pi^{*}}\right)\end{split} \tag{2}\]

where the initialization is set as: \(\theta_{i}^{0}=\theta\), and \(\phi_{i}^{0}=\phi\). The goal of this section is to learn effective initialization settings \((\theta,\phi)\) for meta prompting tasks, which can be achieved by minimizing the meta loss on various tasks:

\[\theta^{*},\phi^{*}=\operatorname*{arg\,min}_{\theta,\phi}\sum_{\tau_{i}\in \mathcal{T}}\mathcal{L}_{\mathcal{D}_{i_{t}}^{q}}\left(\theta_{\theta_{i}, \phi_{i}|\pi^{*}}\right) \tag{3}\]

where \(\mathcal{T}\) is the task set. According to the chain rule, we use the second-order gradient to update \(\theta\) (or \(\phi\)) based on query data:

\[\begin{split}\theta\!\leftarrow&\theta-\beta\cdot \beta_{\theta}^{\text{\emph{second}}}\\ =&\theta-\beta\cdot\sum_{\tau_{i}\in\mathcal{T}} \nabla_{\theta_{i}}\mathcal{L}_{\mathcal{D}_{i_{t}}^{q}}\left(\theta_{\theta_{ i},\phi_{i}|\pi^{*}}\right)\end{split} \tag{4}\]

where \(\mathbf{H}_{\theta}(\mathcal{L})\) is the Hessian matrix with \((\mathbf{H}_{\theta}(\mathcal{L}))_{ij}=\vartheta^{2}\mathcal{L}/\partial \theta_{i}\theta_{j}\); and \(\phi\) can be updated in the same way.

Kindly note that in the prompt learning area, the task head is also known as the answering function, which connects the prompt to the answers for downstream tasks to be reformulated. The answering function can be both tunable or hand-craft templates. In section 3.5, we also propose a very simple but effective hand-crafted prompt answering template without any tunable task head.

#### 3.4.3. Overall Learning Process

To improve the learning stability, we organize these tasks as multi-task episodes where each episode contains batch tasks including node classification ("\(n\)" for short), edge classification ("\(\ell\)" for short), and graph classification ("\(g\)" for short). Let \(\mathcal{E}_{i}=(\mathcal{T}_{\mathcal{E}_{i}},\mathcal{L}_{\mathcal{E}_{i}}, \mathcal{S}_{\mathcal{E}_{i}},\mathcal{Q}_{\mathcal{E}_{i}})\) be a multi-task episode. We define task batch \(\mathcal{T}_{\mathcal{E}_{i}}=\{\mathcal{T}_{\mathcal{E}_{i}}^{(g)},\mathcal{T }_{\mathcal{E}_{i}}^{(n)},\mathcal{T}_{\mathcal{E}_{i}}^{(\ell)}\}\) where each subset \(\mathcal{T}_{\mathcal{E}_{i}}^{(\ast)}=\{\tau_{\alpha_{1}},\cdots,\tau_{\alpha_ {\ell}}\}\); loss function sets \(\mathcal{L}_{\mathcal{E}_{i}}=\{\mathcal{L}^{(g)},\mathcal{L}^{(n)},\mathcal{L }^{(\ell)}\}\), supporting data \(\mathcal{S}_{\mathcal{E}_{i}}=\{\mathcal{S}_{\mathcal{E}_{i}}^{(g)},\mathcal{S }_{\mathcal{E}_{i}}^{(n)},\mathcal{S}_{\mathcal{E}_{i}}^{(\ell)}\}\) where each subset \(\mathcal{S}_{\mathcal{E}_{i}}^{(\ast)}=\{\mathcal{D}_{\mathcal{E}_{i}}^{g}, \cdots,\mathcal{D}_{\mathcal{E}_{\tau_{\alpha_{\ell}}}}^{s}\}\), and query data \(\mathcal{Q}_{\mathcal{E}_{i}}=\{\mathcal{Q}_{\mathcal{E}_{i}}^{(g)},\mathcal{Q }_{\mathcal{E}_{i}}^{(n)},\mathcal{Q}_{\mathcal{E}_{i}}^{(\ell)}\}\) where \(\mathcal{S}_{\mathcal{E}_{i}}^{(\ast)}=\{\mathcal{D}_{\mathcal{E}_{i}}^{g}, \cdots,\mathcal{D}_{\mathcal{E}_{\tau_{\alpha_{\ell}}}}^{s}\}\). Then the multi-task prompting is presented in Algorithm 1. We treat each node/edge/graph class as a binary classification task so that they can share the same task head. Note that our method can also deal with other tasks beyond classification with only a few adaptations (see Appendix A).

```
Input: Overall pipeline \(\mathcal{f}_{\theta,\phi|\pi^{*}}\) with prompt parameter \(\theta\), pre-trained model with frozen parameter \(\pi^{*}\), and task head parameterized by \(\phi\); Multi-task episodes \(\mathcal{E}=\{\mathcal{E}_{1},\cdots,\mathcal{E}_{n}\}\); Output: Optimal pipeline \(\mathcal{f}_{\theta^{*},\phi^{*}|\pi^{*}}\)
1 Initialize \(\theta\) and \(\phi\)
2whilenot donedo// inner adaptation Sample \(\mathcal{E}_{i}\in\mathcal{E}\) where \(\mathcal{E}_{i}=(\mathcal{T}_{\mathcal{E}_{i}},\mathcal{L}_{\mathcal{E}_{i}}, \mathcal{S}_{\mathcal{E}_{i}},\mathcal{Q}_{\mathcal{E}_{i}})\) for\(\tau_{\alpha}\in\mathcal{T}_{\mathcal{E}_{i}}\); \(=g,n,\ell\)do\(\theta_{\tau_{\alpha}},\phi_{\tau_{\alpha}}\leftarrow\theta,\phi\) \(\theta_{\tau_{\alpha}}\leftarrow\theta_{\tau_{\alpha}}-\alpha\nabla_{\theta_{ \tau_{\alpha}}}\mathcal{L}_{\mathcal{D}_{\tau_{\alpha}}^{(\ast)}}^{(\ast)}\left( \theta_{\theta_{\tau_{\alpha}},\phi_{\tau_{\alpha}}|\pi^{*}}\right)\) \(\phi_{\tau_{\alpha}}\leftarrow\phi_{\tau_{\alpha}}-\alpha\nabla_{\phi_{\tau_{ \alpha}}}\mathcal{L}_{\mathcal{D}_{\tau_{\alpha}}^{(\ast)}}^{(\ast)}\left( \theta_{\theta_{\tau_{\alpha}},\phi_{\tau_{\alpha}}|\pi^{*}}\right)\)  end for // outer meta update Update \(\theta,\phi\) by Equation (4) on \(\mathcal{Q}_{\mathcal{E}_{i}}=\{\mathcal{D}_{\tau_{\alpha}}^{g}|\tau_{\alpha} \in\mathcal{T}_{\mathcal{E}_{i}},\ast=g,n,\ell\}\)
3 end while return\(\mathcal{f}_{\theta^{*},\phi^{*}|\pi^{*}}\)
```

**Algorithm 1**Overall Learning Process

### Why It Works?

#### 3.5.1. Connection to Existing Work

A prior study on graph prompt is proposed by (Kang et al., 2017), namely GPPT. They use edge prediction as a pre-training pretext and reformulate node classification to the pretext by designing labeled tokens added to the original graph. The compound graph will be sent to the pre-trained model again to predict the link connecting each node to the label tokens. Their work somehow is a special case of our method when our prompt graph only contains isolated tokens, each of which corresponds to a node category. However, there are at least three notable differences: (1) GPPT is not flexible to manipulate original graphs; (2) GPPT is only applicable for node classification; and (3) GPPT only supports edge prediction task as the pretext but is not compatible with more advanced graph-level pre-training strategies such as GraphCL (Wang et al., 2019), UGRAPHEMB (Chen et al., 2019), SimGRACE (Wang et al., 2019) etc. We further discuss these issues w.r.t. flexibility, efficiency, and compatibility as below.

#### 3.5.2. Flexibility

The nature of prompting is to manipulate the input data to match the pretext. Therefore, the flexibility of data operations is the bottleneck of prompting performance. Let \(g\) be any graph-level transformation such as "changing node features", "adding or removing edges/subgraphs" etc., and \(\varphi^{*}\) be the frozen pre-trained graph model. For any graph \(\mathcal{G}\) with adjacency matrix \(\mathbf{A}\) and node feature matrix \(\mathbf{X}\), Fang et al. (Fang et al., 2019) have proved that we can always learn an appropriate prompt token \(p^{*}\) making the following equation stand:

\[\varphi^{*}\left(\mathbf{A},\mathbf{X}+p^{*}\right)=\varphi^{*}(g(\mathbf{A}, \mathbf{X}))+O_{p\varphi} \tag{5}\]

This means we can learn an appropriate token applied to the original graph to imitate any graph manipulation. Here \(O_{p\varphi}\) denotes the error bound between the manipulated graph and the prompting graph w.r.t. their representations from the pre-trained graph model. This error bound is related to some non-linear layers of the model (_unchangeable_) and the quality of the learned prompt (_changeable_), which is promising to be further narrowed down by a more advanced prompt scheme. In this paper, we extend the standalone token to a prompt graph that has multiple prompt tokens with learnable inner structures. Unlike the indiscriminate inserting in Equation (5) ("\(\mathbf{x}+p^{*}\)" means the prompt token should be added to every node of the original graph), the inserting pattern of our proposed prompt graph is highly customized. Let \(\psi(\mathcal{G},\mathcal{G}_{p})\) denote
the inserting pattern defined in section 3.3; \(\mathcal{G}\) is the original graph, and \(\mathcal{G}_{p}\) is the prompt graph, then we can learn an optimal prompt graph \(\mathcal{G}_{p}^{*}\) to extend Equation (5) as follows:

\[\varphi^{*}\left(\mathcal{V}(\mathcal{G},\mathcal{G}_{p}^{*})\right)=\varphi^{* }(\mathbf{g}(\mathbf{A},\mathbf{X}))+O_{p\varphi}^{*} \tag{6}\]

By efficient tuning, the new error bound \(O_{p\varphi}^{*}\) can be further reduced. In section 4.6, we empirically demonstrate that \(O_{p\varphi}^{*}\) can be significantly smaller than \(O_{p\varphi}\) via efficient training. That means our method supports more flexible transformations on graphs to match various pre-training strategies.

#### 3.5.3. Efficiency

Assume an input graph has \(N\) nodes and \(M\) edges and the prompt graph has \(n\) tokens with \(m\) edges. Let the graph model contain \(L\) layers and the maximum dimension of one layer be \(d\). The parameter complexity of the prompt graph is only \(O(nd)\). In contrast, some typical graph models (e.g., GAT (Srivastava et al., 2015)) usually contain \(O(Lkd^{2}+LKd)\) parameters to generate node embedding and additional \(O(Kd)\) parameters to obtain the whole graph representation (\(K\) is the multi-head number). The parameters may be even larger in other graph neural networks (e.g., graph transformer (K
are also included as the backbones of our prompt methods. **(2) Pre-training with fine-tuning**: These methods first pre-train a GNN model in a self-supervised way such as GraphCL (Zhu et al., 2017) and SimGRACE (Zhu et al., 2017), then the pre-trained model will be fine-tuned for a new downstream task. **(3) Prompt methods**: With a pre-trained model frozen and a learnable prompt graph, our prompt method aims to change the input graph and reformulate the downstream task to fit the pre-training strategies.

#### 4.1.3. Implementations

We set the number of graph neural layers as 2 with a hidden dimension of 100. To study the transferability across different graph data, we use SVD (Singular Value Decomposition) to reduce the initial features to 100 dimensions. The token number of our prompt graph is set as 10. We also discuss the impact of token numbers in section 4.4 where we change the token number from 1 to 20. We use the Adam optimizer for all approaches. The learning rate is set as 0.001 for most datasets. In the meta-learning stage, we split all the node-level, edge-level, and graph-level tasks randomly in 1:1 for meta-training and meta-testing. Reported results are averaged on all tested tasks. More implementation details are shown in Appendix A, in which we also analyze the performance on more datasets and more kinds of tasks such as regression, link prediction, and so on.

### Multi-Task Performance with Few-shot Learning Settings (RQ1)

We compared our prompt-based methods with other mainstream training schemes on node-level, edge-level, and graph-level tasks under the few-shot setting. We repeat the evaluation 5 times and report the average results in Table 2, Table 12 (Appendix A), and Table 13 (Appendix A). From the results, we can observe that most supervised methods are very hard to achieve better performance compared with pre-train methods and prompt methods. This is because the empirical annotations required by supervised frameworks in the few-shot setting are very limited, leading to poor performance. In contrast, pre-training approaches contain more prior knowledge, making the graph model rely less on data labels. However, to achieve better results on a specific task, we usually need to carefully select an appropriate pre-training approach and carefully tune the model to match the target task, but this huge effort is not ensured to be applicable to other tasks. The gap between pre-training strategies and downstream tasks is still very large, making the graph model very hard to transfer knowledge on multi-task settings (we further discuss the transferability in section 4.3) Compared with pre-training approaches, our solutions further improve the compatibility of graph models. The reported improvements range from 1.10% to 8.81% on node-level tasks, 1.28% to 12.26% on edge-level tasks, and 0.14% to 10.77% on graph-level tasks. In particular, we also compared our node-level performance with the previously mentioned node-level prompt model GPPT in Table 2. Kindly note that our experiment settings are totally different from GPPT. In GPPT, they study the few-shot problem by masking 30% or 50% data labels. However, in our paper, we propose a more challenging problem: how does the model perform if we further reduce the label data? So in our experiment, each class only has 100 labeled samples. This different setting makes our labeled ratio approximately only 25% on Cora, 18% on CiteSeer, 1.7% on Reddit, 7.3% on Amazon, and 1.5% on Pubmed, which are far less than the reported GPPT (50% labeled).

### Transferability Analysis (RQ2)

To evaluate the transferability, we compared our method with the hard transfer method and the fine-tuning method. Here the hard transfer method means we seek the source task model which has the same task head as the target task and then we directly conduct the model inference on the new task. The fine-tune method means we load the source task model and then tune the task head for the new task. We evaluate the transferability from two perspectives: (1) how effectively is the model transferred to different tasks within the same domain? and (2) how effectively is the model transferred to different domains?

#### 4.3.1. Transferability to Different Level Tasks

Here we pre-train the graph neural network on Amazon, then conduct the model on two source tasks (graph level and node level), and further evaluate the performance on the target task (edge level). For simplicity, both source tasks and the target task are built as binary classifications with \(1:1\) positive and negative samples (we randomly select a class as the positive label and sample negatives from the rest). We report the results in Table 3, from which we have two observations: First, our prompt method significantly outperforms the other approaches and the prediction results make sense. In contrast, the problem of the hard transfer method is that the source model sometimes can not well decide on the target tasks because the target classes may be far away from the source classes. This may even cause negative transfer results (results that are lower than random guess). In most cases, the fine-tuning method can output meaningful results with a few steps of tuning but it can still encounter a negative transfer problem. Second, the graph-level task has better adaptability than the node-level task for the edge-level target, which is in line with our previous intuition presented in Figure 3 (section 3.2).

#### 4.3.2. Transferability to Different Domains

We also conduct the model on Amazon and PubMed as source domains, then load the model states from these source domains and report the performance on the target domain (Cora). Since different datasets have various input feature dimensions, we here use SVD to unify input features from all domains as 100 dimensions. Results are shown in Table 4, from which we can find that the good transferability of our prompt also exists when we deal with different domains.

### Ablation Study (RQ3)

In this section, we compare our complete framework with four variants: "w/o meta" is the prompt method without meta-learning

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline Dataset & \#Nodes & \#Edges & \#Features & \#Labels \\ \hline Cora & 2,708 & 5,429 & 1,433 & 7 \\ CiteSeer & 3,327 & 9,104 & 3,703 & 6 \\ Reddit & 232,965 & 23,213,838 & 602 & 41 \\ Amazon & 13,752 & 491,722 & 767 & 10 \\ Pubmed & 19,717 & 88,648 & 500 & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 1. Statistics of datasets.
step; "w/o h" is our method without task head tuning, which is previously introduced in section 3.5.4; "w/o token structure" is the prompt where all the tokens are treated as isolated without any inner connection; and "w/o inserting" is the prompt without any across links between prompt tokens and the input graphs. We report the performance in Figure 5, from which we can find the meta-learning and token structure all contribute significantly to the final results. In particular, the inserting pattern between a prompt graph and the input graph plays a very crucial role in the final performance. As previously discussed, the purpose of the prompt-based method is to relieve the difficulty of traditional "pre-train, fine-tuning" by filling the gap between the pre-training model and the task head. This means the prompt graph is proposed to further improve the fine-tuning performance. This is particularly important when we transfer the model across different tasks/domains, which proposes harder demand for the task head. As suggested in Figure 5, even when we totally remove the tunable task head, the "w/o h" variant can still perform very competitively, which suggests the powerful capability of bridging upstream and downstream tasks.

### Efficiency Analysis (RQ4)

Figure 6 presents the impact of increasing token number on the model performance, from which we can find that most tasks can reach satisfactory performance with very limited tokens, making

\begin{table}
\begin{tabular}{l l|c c c} \hline \hline Source task & Methods & Accuracy & F1-score & AUC score \\ \hline \multirow{3}{*}{graph level} & hard & 51.50 & 65.96 & 40.34 \\  & fine-tune & 62.50 & 70.59 & 53.91 \\  & prompt & 70.50 & 71.22 & 74.02 \\ \hline \multirow{3}{*}{node level} & hard & 40.50 & 11.85 & 29.48 \\  & fine-tune & 46.00 & 54.24 & 37.26 \\  & prompt & 59.50 & 68.73 & 55.90 \\ \hline \hline \end{tabular}
\end{table}
Table 4. Transferability (%) from different domains. Source domains: Amazon and PubMed. Target domain: Cora

\begin{table}
\begin{tabular}{l l|c c|c c c|c c c|c c c|c c c} \hline \hline Training & \multirow{2}{*}{Methods} & \multicolumn{3}{c|}{Cora} & \multicolumn{3}{c|}{CiteSeer} & \multicolumn{3}{c|}{Reddit} & \multicolumn{3}{c|}{Amazon} & \multicolumn{3}{c}{Pubmed} \\ schemes & & Acc & F1 & AUC & Acc & F1 & AUC & Acc & F1 & AUC & Acc & F1 & AUC & Acc & F1 & AUC \\ \hline \multirow{3}{*}{supervised} & GAT & 74.45 73.21 & 82.97 & 83.00 & 83.20 & 89.33 & 55.64 & 62.03 & 65.38 & 79.00 & 73.42 & 97.81 & 75.00 & 77.56 & 79.72 \\  & GCN & 77.55 77.45 & 83.71 & 88.00 & 81.79 & 94.79 & 54.38 & 52.47 & 56.82 & 95.36 & 93.99 & 96.23 & 53.64 & 66.67 & 69.89 \\  & GT & 74.25 75.21 & 82.04 & 86.33 & 85.62 & 90.13 & 61.50 & 61.38 & 65.56 & 85.50 & 86.01 & 93.01 & 51.50 & 67.34 & 71.91 \\ \hline \multirow{6}{*}{pre-train} & GraphCL+GAT & 76.05 & 76.78 & 81.96 & 87.64 & 88.40 & 89.93 & 57.37 & 66.42 & 74.73 & 76.87 & 72.26 & 95.65 & 76.03 & 70.75 & 80.02 \\  & GraphCL+GCN & 78.75 & 79.13 & 84.90 & 87.49 & 89.36 & 90.25 & 55.00 & 65.52 & 74.65 & 96.00 & 95.92 & 98.93 & 69.70 & 70.04 & 74.74 \\  & GraphCL+GT & 73.80 & 74.12 & 82.87 & 88.50 & 88.92 & 91.25 & 63.50 & 66.06 & 66.08 & 80.44 & 93.93 & 96.92 & 69.97 & 75.00 & 74.74 \\  & SimGRACE+GAT & 76.85 & 77.48 & 83.97 & 90.50 & 91.00 & 91.56 & 56.59 & 65.47 & 67.77 & 84.50 & 84.73 & 89.69 & 77.20 & 68.21 & 81.97 \\  & SimGRACE+GCN & 77.20 & 76.39 & 83.13 & 83.50 & 83.21 & 93.22 & 58.00 & 55.81 & 56.93 & 95.00 & 94.50 & 98.03 & 77.50 & 75.71 & 87.53 \\  & SimGRACE+GT & 77.40 & 78.11 & 82.95 & 87.50 & 87.05 & 91.85 & 66.00 & 69.95 & 70.03 & 79.00 & 73.42 & 97.58 & 70.50 & 73.30 & 74.22 \\ \hline \multirow{6}{*}{prompt} & GraphCL+GAT & 76.50 & 77.72 & 82.99 & 88.00 & 90.52 & 91.82 & 75.84 & 67.02 & 75.33 & 80.01 & 75.62 & 97.96 & 77.50 & 78.26 & 83.02 \\  & GraphCL+GCN & 79.20 & 76.05 & 85.29 & 85.80 & 91.59 & 91.94 & 56.00 & 68.57 & 78.82 & 96.50 & 96.37 & 89.70 & 72.50 & 76.24 & 79.57 \\  & GraphCL+GT & 75.00 & 76.00 & 83.36 & 91.00 & 90.10 & 90.29 & 65.50 & 66.08 & 86.86 & 95.50 & 95.43 & 97.56 & 76.70 & 79.11 & 76.00 \\  & SimGRACE+GAT & 76.95 & 78.51 & 83.55 & 93.00 & 93.14 & 92.44 & 57.63 & 66.64 & 69.43 & 95.00 & 95.43 & 97.56 & 73.00 & 74.04 & 81.89 \\  & SimGRACE+GCN & 77.85 & 76.57 & 83.79 & 90.00 & 89.47 & 94.87 & 59.50 & 55.97 & 59.46 & 95.00 & 95.24 & 98.42 & 78.00 & 78.22 & 87.66 \\  & SimGRACE+GT & 78.75 & 79.53 & 85.03 & 91.00 & 91.26 & 95.62 & 69.50 & 71.43 & 70.75 & 86.00 & 83.72 & 98.24 & 73.00 & 73.79 & 76.64 \\ \hline \multirow{6}{*}{reported Acc of GPTT (Label Ratio 50\%)} & 1.47 & 1.94 & 1.10 & 3.81 & 5.25 & 2.05 & 3.97 & 5.04 & 6.98 & 4.49 & 5.84 & 2.24 & 8.81 & 4.55 & 4.62 \\ \cline{1-1}  & \multirow{2}{*}{reported Acc of GPTT (Label Ratio 50\%)} & 77.16 & \(-\) & 65.81 & \(-\) & 92.13 & \(-\) & \(-\) & 86.80 & \(-\) & \(-\) & 72.23 & \(-\) & \(-\) \\ \cline{1-1}  & & \multirow{2}{*}{\(\sim\) 25\%} & & \(\sim\) 18\% & & \(\sim\) 1.7\% & & \(\sim\) 7.3\% & & \(\sim\) 1.5\% & \\ \hline \hline \end{tabular}
\end{table}
Table 2. Node-level performance (%) with 100-shot setting. IMP (%): the average improvement of prompt over the rest.

Figure 5. Effectiveness of main components
the complexity of the prompt graph very small. The limited token numbers make our tunable parameter space far smaller than traditional methods, which can be seen in Table 5. This means our method can be efficiently trained with a few steps of tuning. As shown in Figure 7, the prompt-based method converges faster than traditional pre-train and supervised methods, which further suggests the efficiency advantages of our method.

### Flexibility on Graph Transformation (RQ5)

As discussed in section 3.5.2, the flexibility of data transformation is the bottleneck of prompt-based methods. Here we manipulate several graphs by dropping nodes, dropping edges, and masking features, then we calculate the error bound mentioned in Equation 5 and 6. We compare the original error with the naive prompt mentioned in Equation 5, and our prompt graph with 3, 5, and 10 tokens. As shown in Table 6, our designed prompt graph significantly reduces the error between the original graph and the manipulated graph. This means our method is more powerful to stimulate various graph transformations and can further support significant improvement for downstream tasks. This capability can also be observed in the graph visualization from two approaches. As shown in Figure 8, the graph representations from a pre-trained model present lower resolution to node classes compared with our prompted graph.

## 5. Conclusion

In this paper, we study the multi-task problem of graph prompts with few-shot settings. We propose a novel method to reformulate different-level tasks to unified ones and further design an effective prompt graph with a meta-learning technique. We extensively evaluate the performance of our method. Experiments demonstrate the effectiveness of our framework.