Self-Pro: A Self-Prompt and Tuning Framework for Graph Neural Networks
Graphs have become an important modeling tool for web applications, and Graph
Neural Networks (GNNs) have achieved great success in graph representation
learning. However, the performance of traditional GNNs heavily relies on a
large amount of supervision. Recently, ``pre-train, fine-tune'' has become the
paradigm to address the issues of label dependency and poor generalization.
However, the pre-training strategies vary for graphs with homophily and
heterophily, and the objectives for various downstream tasks also differ. This
leads to a gap between pretexts and downstream tasks, resulting in ``negative
transfer'' and poor performance. Inspired by prompt learning in Natural
Language Processing (NLP), many studies turn to bridge the gap and fully
leverage the pre-trained model. However, existing methods for graph prompting
are tailored to homophily, neglecting inherent heterophily on graphs.
Meanwhile, most of them rely on the randomly initialized prompts, which
negatively impact on the stability. Therefore, we propose Self-Prompt, a
prompting framework for graphs based on the model and data itself. We first
introduce asymmetric graph contrastive learning for pretext to address
heterophily and align the objectives of pretext and downstream tasks. Then we
reuse the component from pre-training phase as the self adapter and introduce
self-prompts based on graph itself for task adaptation. Finally, we conduct
extensive experiments on 11 benchmark datasets to demonstrate its superiority.
We provide our codes at https://github.com/gongchenghua/Self-Pro.
Chenghua Gong, Xiang Li, Jianxiang Yu, Cheng Yao, Jiaqi Tan, Chengcheng Yu
# Prompt Tuning for Multi-View Graph Contrastive Learning

###### Abstract.

In recent years, "pre-training & fine-tuning" has emerged as a promising approach in addressing the issues of label dependency and poor generalization performance in traditional GNNs. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompt tuning is a popular alternative to "pre-training & fine-tuning" in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives. However, existing study of prompting on graphs is still limited, lacking a framework that can accommodate commonly used graph pre-training methods and downstream tasks. In this paper, we propose a multi-view graph contrastive learning method as pretest and design a prompting tuning for it. Specifically, we first reformulate graph pre-training and downstream tasks into a common format. Second, we construct multi-view contrasts to capture relevant information of graphs by GNN. Third, we design a prompting tuning method for our multi-view graph contrastive learning method to bridge the gap between pretexts and downstream tasks. Finally, we conduct extensive experiments on benchmark datasets to evaluate and analyze our proposed method.

2017 acmcopy 2023 Association for Computing Machinery, ACM ISBN 978-x-xx-xxx-xx/YY/MM... $15.00[https://doi.org/10.1145/mnmnn.mmn](https://doi.org/10.1145/mnmnn.mmn) +
Footnote †: journal: Computer Vision and Pattern Recognition
node classification) using edge prediction as pretext and design a learnable prompt. This pretext overly emphasizes the topological information, thereby neglecting the important semantic information within the graph. Especially when dealing with node classification tasks on heterophilous graphs, the performance is naturally suboptimal due to the differing labels of neighboring nodes.

In this paper, our emphasis lies on the formulation between pre-training and downstream tasks and prompting methods for GNNs. Specifically, we strive for a unified framework that offers flexibility in accommodating various types of graphs and diverse downstream tasks. However, the appliacation of prompts on graphs encounters the following three challenges. First, graph data is more complex compared to text data. Graph data usually contains both graph structure and node features, which play different roles in various tasks. For example, node features in social networks have a great impact on the classification results, while in chemical substance networks, graph structure plays a more essential part. Therefore, prompt on graphs should consider both node features and graph structure and have the ability to transform any of them adaptively. Second, to fully leverage knowledge of pre-training model and transfer it, the optimization objective of pre-training step should be compatible with that of downstream tasks. The key to solving the problem is how to reformulate pre-training and downstream tasks in the same template. Specifically, prompting in NLP usually reformulate both pre-training and downstream tasks as masked language modeling (Chen et al., 2017). However, the pre-training strategies vary for graphs under homophily and heterophily. Meanwhile, graph downstream tasks include node-level, edge-level and graph-level ones, and they often have different objectives. It is crucial but challenging to convert the pretexts and different-level task to a unified framework. Third, it is equally important to design specific prompt under unified framework to identify the distinction between different downstream tasks. Prompt tuning in NLP employed handcrafted tokens (Wang et al., 2018; Wang et al., 2019) or learnable word vectors (Wang et al., 2019; Wang et al., 2019) to give different hints to different tasks. Accordingly, we should design task-specific prompting methods for graphs to guide downstream tasks to extract relevant prior knowledge of pre-trained models.

To address the aforementioned challenges, we proposed a multi-view graph contrastive learning method and corresponding prompt tuning strategies, namely, PGCL. More specifically, to address the first challenge, we present a common template through constructing graph instances and focus on two crucial graph components: node feature and graph structure. So we establish two views, namely, semantic view and contextual view to capture relevant information on graphs, respectively. To address the second challenge, we reformulate pre-training and downstream task into the same format, which aims to compute the similarity of graph representations from both semantic view and contextual view. In this work, we adopt contrastive learning as the self-supervised pre-training task, which can be understood as calculating the similarity between anchor and positive/negative sample with the goal of reducing the distance between anchor and positive sample, while enlarging that between anchor and negative sample in the latent space. Accordingly, downstream tasks can be reformulated to calculate the similarity of semantic and contextual views among the graph instances, which bridges the gap between the pretext and different downstream tasks. To address the third challenge, we distinguish different down-stream tasks by way of a fusion operation and a learnable prompt. First, we fuse representations from semantic view and contextual view into a comprehensive representation. Then we use a multi-view learnable prompt to extract the most relevant prior knowledge from representation of both views for downstream tasks. In summary, our main contributions can be summarized as follows:

\(\bullet\) We propose PGCL, a novel multi-view pre-train and unified downstream tasks prompting framework. To our best knowledge, PGCL is the first graph prompt framework that can accommodate various types of graphs and downstream tasks.

\(\bullet\) We propose a prompting strategy for PGCL, hinging on a fusion operation and a learnable prompt design to transfer the pre-trained knowledge to different downstream tasks for improving performance.

\(\bullet\) We conduct extensive experiments on \(12\) benchmark datasets to evaluate the performance of PGCL. Our results show its superiority over other state-of-the-art competitors.

## 2. Related Work

### Graph Neural Networks

Recently, GNNs have received significant attention for Web applications and there have been many GNN models proposed (Krizhevsky et al., 2014; Kipf and Welling, 2015; Kipf and Welling, 2016) on homophilic graphs. Their key idea boils down to a message-passing framework, in which each node derives its representation by receiving and aggregating messages from its neighboring nodes recursively. Moreover, there are also many studies on designing GNNs for heterophilic graphs (Chen et al., 2017; Wang et al., 2019; Wang et al., 2019; Wang et al., 2019; Wang et al., 2019). Existing graph neural network methods on heterophilic graphs can mainly be divided into two categories. One is to capture information from distant nodes (Wang et al., 2019; Wang et al., 2019; Wang et al., 2019), the other is to adaptively aggregate useful information from the neighborhood by refining the GNN architecture (Chen et al., 2017; Wang et al., 2019). At graph levels, representation learning requires an additional readout operation (Chen et al., 2017; Wang et al., 2019; Wang et al., 2019) which summarizes the global information of a graph by aggregating node representations. Recent research has also focused on addressing the challenges posed by insufficient data labels in order to make graph learning more adaptive. Furthermore, there is a growing interest in improving the model's generalization when it is transferred to new domains. To tackle these issues, many studies have shifted their focus towards graph unsupervised learning and pre-training methods, as opposed to traditional supervised learning approaches.

### Graph Pre-training

Inspired by pre-trained techniques in NLP, tremendous efforts have been devoted to pre-trained graph models. Some effective pre-training strategies include two types of unsupervised learning: generation-based methods and contrast-based methods. Generation-based methods such as GAE (Krizhevsky et al., 2014), GraphMAE (Krizhevsky et al., 2014) and SeeGera (Krizhevsky et al., 2014) reconstruct the graph data from the perspectives of feature and structure of the graph, and use the input data as the supervision signal. Contrast-based methods construct representations under different views and maximize their agreement. In particular, GRACE (Wang et al., 2019) pulls the representations of the same node closer under different augmentations and pushes away the representations
of other nodes. GraphCL (Gordes and Riedl, 2017) brings the graph-level representations closer under different views to ensure perturbation invariance. DGI (Wang et al., 2017) and MVGRL (Wang et al., 2018) maximize the mutual information between node-level representations and graph-level representations. For heterophilic graphs, GREET (Wang et al., 2019) discriminates homophilic edges from heterophilic edges and uses low-pass and high-pass filters to capture the corresponding information. NWR-GAE (Wang et al., 2019) emphasizes the graph topology and reconstructs the neighborhoods based on the local structure and features. However, the above approaches do not consider the gap between pre-training and downstream objectives, which limits their generalization ability.

### Graph Prompt Learning

Recognizing the existing gap between pre-training and downstream tasks, recent studies have aimed to bridge this gap and shifted their focus towards prompt learning. Many effective prompt methods are firstly proposed in the NLP area, including some hand-crafted prompts (Wang et al., 2017; Wang et al., 2018) and continuous prompts (Wang et al., 2018; Wang et al., 2019; Wang et al., 2019). However, we only find very few works like GPPT (Wang et al., 2019) and GraphPrompt (Wang et al., 2019) in the graph domain. GPPT pre-trains a GNN model based on link prediction and leverages a sophisticated design of learnable prompts but it only works with node classification. GraphPrompt introduces a unification framework for pretext and down-stream tasks and employs a learnable prompt to assist downstream tasks. Both of them employ link prediction as the pre-training task, but this pretext is limited, which excessively prioritizes the emphasis on the graph topology without considering the importance of node attributes.

## 3. Preliminaries

In this section, we introduce the notations used in this paper and also some GNN basics.

### Notations

Let \(G=(V,E)\) be an undirected and unweighted graph, where \(V\) is the set of nodes and \(E\) is the set of edges. \(\mathbf{X}\in\mathbb{R}^{|V|\times d}\) is the feature matrix where the \(i\)-th row \(x_{i}\) is the \(d\)-dimensional feature vector of node \(u_{i}\in V\). \(\mathbf{A}\in\mathbb{R}^{|V|\times|V|}\) denotes the binary adjacent matrix with \(\mathbf{A}_{i,j}=1\) if \(e_{i,j}\in E\) and \(\mathbf{A}_{i,j}=0\) otherwise. The neighboring set of node \(v\) is denoted as \(\mathcal{N}(v)\). In addition, we denote a set of graphs as \(\mathcal{G}=\{G_{1},G_{2},...\}\).

### Graph Neural Networks

GNNs adopt a message passing mechanism, where the representation of each node is updated by aggregating messages from its local neighboring nodes, and then combining the aggregated messages with the node's own representation. Generally, given a GNN model \(f(\cdot)\), message passing in the \(l\)-th layer can be divided into two operations: one is to aggregate information from a node's neighbors while the other is to update a node's representation. Given a node \(v\), these two operations are formulated as:

\[m_{v}^{(l)}=\texttt{AGGREGATE}^{(l)}\{h_{v}^{(l-1)},\forall u\in\mathcal{N }(v)\}, \tag{1}\]

\[h_{v}^{(l)}=\texttt{COBINE}^{(l)}\{h_{v}^{(l-1)},m_{v}^{(l)}\}, \tag{2}\]

where \(m_{v}^{(l)}\) and \(h_{v}^{(l)}\) denote the message vector and representation of node \(v\) in the \(l\)-th layer, respectively. \(\texttt{AGGREGATE}^{(l)}(\cdot)\) and \(\texttt{COMBINE}^{(l)}(\cdot)\) are two functions in each GNN layer. Note that in the first layer, the input node embedding \(h_{v}^{0}\) can be initialized as the node features in \(\mathbf{X}\). The total learnable GNN parameters can be denoted as \(\Theta\). For brevity, we simply denote the output node representations of the last layer as \(h_{v}\).

## 4. Methodologies

In this section, we introduce the proposed framework of MVP which is illustrated in Fig. 1. First, we present a common template and unify various graph tasks in the same format. Second, we propose a multi-view pre-training approach to capture relevant information from different views. Third, we design prompts for specific tasks to further narrow down the gap between pre-training and downstream phrase. Next, we elaborate on the main components.

### Unified Framework

In the following, we present how the framework facilitates a unified perspective on pre-training and downstream tasks.

#### 4.1.1. Graph instances under semantic and contextual view

**Motivation of constructing graph instances.** The success of prompts in NLP relies on the shared task template. However, graph-related tasks including node-level, edge-level and graph-level are far from similar. Generally, operations on graphs such as "modifying features" at node-level or "adding/deleting edges" at edge-level can both be considered as basic operations of graph-level. Compared with node-level and edge-level, graph-level tasks are more general and graph-level knowledge can be effectively transferred to other levels (Wang et al., 2019). So we follow (Wang et al., 2019; Wang et al., 2019) and uniformly perform the node-level, edge-level tasks on graph-level through constructing graph instances. At node-level, we expand the target node \(v\) on a graph \(G=(V,E)\) into a subgraph \(S_{v}=(V(S_{v}),E(S_{v}))\) of its local area, where its set of nodes and edges are respectively given by

\[V(S_{v})=\{d(u,v)\leq\delta\mid u\in V\}, \tag{3}\]

\[E(S_{v})=\{(u,u^{\prime})\in E\mid u\in V(S_{v}),u^{\prime}\in V(S_{v})\}, \tag{4}\]

where \(d(u,v)\) gives the shortest distance between nodes \(u\) and \(v\) on \(G\) and \(\delta\) is a predetermined threshold. \(S_{v}\) consists of nodes within \(\delta\) hops from the node \(v\), and the edges between those nodes. At graph-level, the maximum subgraph \(S_{G}\) of a graph \(G\) is the graph itself ( i.e., \(S_{G}=G\)), which naturally embodies the information of all nodes in \(G\).

**Semantic and contextual views.** Since we use graph instances to represent both nodes and graphs, the (sub)graph where the graph instance resides preserves rich self-information and contextual information by neighboring node features and connections (Gordes and Riedl, 2017; Wang et al., 2019; Wang et al., 2019), which play distinct roles in various downstream tasks. Therefore, we establish two views, namely the semantic view and the contextual view, to capture the node feature information and the topological structure information of graph instances, respectively. The semantic view for a graph instance mainly focuses on the features of nodes. This view describes the nodes in the (sub)graph with their intrinsic properties. The contextual view for a graph instance primarily emphasizes the information derived from neighboring connections. This view characterizes nodes in the (sub)graph by considering their local neighborhoods, which is the main scope of most GNN encoders. By establishing and complementing semantic
and contextual views, we aim to capture information from graphs in a more comprehensive manner.

**Embedding of graph instances.** To obtain the embedding \(z_{x}\) of graph instance \(x\), we follow the standard approach and employ a READOUT operation to aggregate the representations of nodes in the (sub)graph \(S_{x}\) of \(x\). Considering a \(K\)-layer GNN \(f(\cdot)\) and node representation \(h_{u}^{(k)}\) generated by it,

\[z_{x}=f(S_{x})=\texttt{READOUT}(h_{u}^{(k)}:v\in V(S_{x}),k\in K). \tag{5}\]

The choice of the aggregation scheme for READOUT is flexible, including sum, max and mean pooling and more advanced techniques [44, 45]. We simply use sum pooling in our implementation.

#### 4.1.2 Unified task template.

**Motivation of adopting constrastive learning.** Since we expand the target node into a graph instance, naturally we treat the target node's label as this graph instance's label. The same at edge-level, pair of nodes can be treated as a positive label if there is an edge connecting them, or a negative label if not. When we treat the edge's label as this pair graph instances' label, we can translate the edge-level task into the relationship learning between two graph instances. As can be seen in Fig. 2, link prediction is anchored on the similarity of representations for pairs of nodes. Intuitively, the representations of two adjacent nodes shall be more similar than those of non-adjacent nodes. For classification tasks, the graph/node representations of the same class shall exhibit higher similarity compared to those from different classes. It is worth noting that the core idea of contrastive learning is calculating the similarity between anchor and positive/negative sample with the goal of reducing the distance between the anchor and positive sample, while enlarging that between the anchor and negative sample. By similarity calculation in Fig. 2, we unify the graph contrastive learning as pretext with downstream tasks into a common task template.

Next, we will formally define the template for downstream tasks. Let \(z_{x}\) be the representation vector of (sub)graph \(S_{x}\) for graph instance \(x\). How to fuse semantic and contextual information to obtain the representation of graph instance will be described in Section 4.3. Let \(sim(\cdot,\cdot)\) be the cosine similarity function. Three downstream tasks (node classification, graph classification and link prediction) can be mapped to the similarity computation, which is formalized below.

**Graph instance classification.** Under our framework, node classification and graph classification are unified as graph instance classification. Consider a set of graph instances \(\mathcal{G}\) with a set of classes \(C\), and a set of labeled graph instances \(\mathcal{D}=\{(G_{1},Y_{1}),(G_{2},Y_{2}),...\}\) where \(G_{i}\in\mathcal{G}\) and \(Y_{i}\) is the corresponding label of \(G_{i}\). We follow the \(k\)-shot setting in [27], there are exactly \(k\) pairs of \((G_{i},L_{i}=c)\in\mathcal{D}\) for each class \(c\in C\). We define a class-prototype represented by \(\tilde{z}_{c}\) for each class \(c\). It is worth noting that the class-prototype is a "virtual" graph instance in the same latent space.

Figure 1. The overall framework of PGCL

Figure 2. Illustration of the motivation.
The class-prototype representation can be obtained through representation learning (Srivastava et al., 2017). However, the representation learning of class-prototypes in few-shot settings poses challenges due to the limited annotated data. Therefore, we follow (Zhu et al., 2017) and employ the mean representation of labeled graph instances for each class as class-prototypes:

\[\widetilde{z_{c}}=\frac{1}{k}\sum_{(G_{i},Y_{i})\in\mathcal{D},Y_{i}=c}z_{G_{i}}. \tag{6}\]

Given a graph instance \(G_{j}\) not in the labeled set \(\mathcal{D}\), its class label \(Y_{j}\) shall be

\[Y_{j}=\arg\max_{c\in C}sim(z_{G_{j}},\widetilde{z_{c}}) \tag{7}\]

Intuitively, the graph instance shall belong to the class whose class-prototype is the most similar to itself.

**Link prediction.** Under our framework, link prediction are model as the similarity computation between pair graph instances. Given a graph \(G=(V,E)\), a triplet of nodes \((v,a,b)\) such that \((a,b)\in E\) and \((v,b)\notin E\), we shall have

\[sim(z_{v},z_{a})>sim(z_{v},z_{b}) \tag{8}\]

It is worth noting that the link prediction task is based on the homophily assumption, i.e., the representation for \(S_{v}\) of \(v\) shall be more similar to that of a node adjacent to \(v\) than that of another non-adjacent node.

In summary, we reformulated the pretext and downstream tasks on graphs into a common task template: the similarity learning among graph instances, which lays the foundation of our pre-training and prompting strategies as we will introduce in the following subsection.

### Multi-View Pre-Training

In this section, we will introduce our multi-view pre-training strategy, which captures relevant information on graphs through semantic and contextual contrasts.

**Semantic View.** Semantic-level contrast aims to encourage the learned representations of graph instances with similar features to be consistent. Given a graph instance \(x\) and its (sub)graph \(S_{x}=(V,E)\), we employ a perturbation \(\tau_{\alpha}\) on the initial feature matrix \(\mathbf{X}\) to generate a new feature matrix as the positive sample:

\[\widetilde{\mathbf{X}}\sim\tau_{\alpha}(\mathbf{X}), \tag{9}\]

where \(\widetilde{\mathbf{X}}\) is the augmented feature matrix. We apply perturbations by altering only the features of nodes while keeping the graph structure unchanged. Specifically, we randomly mask the initial node features in different dimensionality with a probability. To independently encode the features without taking into account the structure information, we construct a unit matrix \(\mathbf{I}\in\mathbb{R}^{|V|\times|V|}\) as the adjacency matrix and feed it into GNN \(f(\cdot)\) along with features. The semantic representation obtained is denoted by \(z^{s}\) and corresponding augmentation are denoted by \(\widetilde{z^{s}}\):

\[z^{s}=f_{0}\left(\mathbf{X},\mathbf{I}\right),\widetilde{z^{s}}=f_{0}\left( \widetilde{\mathbf{X}},\mathbf{I}\right), \tag{10}\]

where \(\Theta\) denotes the parameters of GNN encoder. Then, a non-linear projection \(g(\cdot)\) is used to map the representations to the latent space where the contrastive loss is applied, as advocated in (Bordes et al., 2017). Specially, a two-layer perceptron (MLP) is applied to obtain \(\mathbf{z^{s}}\) and \(\widetilde{\mathbf{z^{s}}}\):

\[\mathbf{z^{s}}=g(z^{s}),\widetilde{\mathbf{z^{s}}}=g(\widetilde{\mathbf{z^{s}}}). \tag{11}\]

We construct the semantic-level contrastive loss based on the normalized temperature-scaled cross entropy loss (NT-Xent) (Bordes et al., 2017). During pre-training phrase, we randomly select \(N\) graph instances from the whole dataset as a mini-batch. For each graph instance \(G_{i}\) in the mini-batch, we construct positive pairs \(\left(\mathbf{z^{s}}_{G_{i}},\widetilde{\mathbf{z^{s}}}_{G_{i}}\right)\), negative pairs are not explicitly sampled but generated from the other graphs and their corresponding augmentations within the same minibatch. For convenience of expression, the semantic contrastive loss is formulated as:

\[\mathcal{L}_{s}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{exp(sim\left(\mathbf{z^{s}}_{G _{i}},\widetilde{\mathbf{z^{s}}_{G_{i}}}\right)/\tau)}{\sum_{j=1}^{N}exp(sim\left( \mathbf{z^{s}}_{G_{i}},\widetilde{\mathbf{z^{s}}_{G_{j}}}\right)/\tau)}, \tag{12}\]

where \(\tau\) denotes the temperature parameter to control the shape of the output distribution.

**Contextual View.** Context-level contrast aims to encourage graph instances with similar topology to be consistent. When constructing positive samples under the contextual view, it should be ensured that the semantic information of graph instances remains unchanged. Therefore, we only introduce a perturbation \(\tau_{\beta}\) on the adjacent matrix \(\mathbf{A}\) while preserving the semantic information:

\[\widetilde{\mathbf{A}}\sim\tau_{\beta}\left(\mathbf{A}\right). \tag{13}\]

Specifically, we randomly drop some edges in the graph with a probability, which is equivalent to removing a small portion of nodes to alter the contextual information. To capture the contextual information of graph instance, we use feature matrix and adjacency matrix as the input of GNN \(f(\cdot)\). The contextual representation obtained is denoted by \(z^{c}\) and the corresponding augmentation are denoted by \(\widetilde{z^{c}}\),

\[z^{c}=f_{0}\left(\mathbf{X},\mathbf{A}\right),\widetilde{z^{c}}=f_{0}\left( \mathbf{X},\widetilde{\mathbf{A}}\right). \tag{14}\]

Similar as the contrastive loss under the semantic view, we construct the contextual contrastive loss after the non- linear projection \(g(\cdot)\):

\[\mathcal{L}_{c}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{exp(sim\left(\mathbf{z^{c}}_{G _{i}},\widetilde{\mathbf{z^{c}}_{G_{i}}}\right)/\tau)}{\sum_{j=1}^{N}exp(sim\left( \mathbf{z^{c}}_{G_{i}},\widetilde{\mathbf{z^{c}}_{G_{j}}}\right)/\tau)}. \tag{15}\]

Note that we share the same GNN encoder between two views and the loss is parameterized by \(\Theta\). To sum up, the pre-training loss \(\mathcal{L}_{pre}\) can be defined as:

\[\mathcal{L}_{pre}=\mathcal{L}_{s}+\lambda\mathcal{L}_{c}, \tag{16}\]

where \(\lambda\) are weight factors for adjusting the importance of two views. After pre-training, we freeze the model and then perform prompting on the outputs of model.

### Prompt Design for Downstream Tasks

Aligning the pretext with downstream tasks can enable more effective knowledge transfer. We next show how to unify pre-training and downstream prompt-tuning.

**Representation fusion and prompt design.** From the macroscopic perspective, node features and graph topology are both
essential components of a graph. To further enhance representation learning on graphs, we consider that the contextual information and the semantic information complement each other. We thus fuse them into a holistic representation that can adapt to different tasks. Here we simply use the CONCAT operation in our implementation. Formally, for each graph instance \(x\), we fuse its semantic representation \(z_{x}^{s}\) and contextual representation \(z_{x}^{c}\) to obtain \(x\)'s representation \(z_{x}=\texttt{CONCAT}(z_{x}^{s},z_{x}^{c})\). In this way, the similarity between two graph instances can be captured by their similarity in both semantic and contextual information. This also aligns with the pre-training objectives.

From the microscopic perspective, semantic view and contextual view play different roles in various tasks. For example, node classification (especially on heterophilous graphs) generally pays more attention to node features, while link prediction and graph classification focus on the structure information. Therefore, we should design prompts to extract the most relevant knowledge from representations of the two views. A naive way is to directly apply a linear transformation \(\mathbf{P}\) as learnable prompt, which can be formulated as:

\[z_{x}^{p}=\mathbf{P}\cdot\texttt{CONCAT}(z_{x}^{s},z_{x}^{c}), \tag{17}\]

where \(z_{x}^{p}\) is the prompted representation for graph instance \(x\). The linear transformation \(\mathbf{P}\) can be regarded as a reweighting between semantic and contextual view to adjust the representation for downstream tasks.

Further, we can also introduce a prompt vector \(\mathbf{p}^{s}\) for semantic view and \(\mathbf{p}^{c}\) for contextual view, respectively. After that, for each view, we perform element-wise reweighting to extract more fine-grained irrelevant information and derive prompted semantic representation \(z_{x}^{ps}\) and contextual representation \(z_{x}^{pc}\):

\[z_{x}^{ps}=z_{x}^{s}\odot\mathbf{p}^{s},z_{x}^{pc}=z_{x}^{c}\odot\mathbf{p}^{ c}, \tag{18}\]

where \(\odot\) denotes the element-wise multiplication. Considering that the importance of these two views could vary in different tasks, we further employ a hyper-parameter \(\alpha\) to control their weights and overload \(z_{x}^{p}\) as:

\[z_{x}^{p}=\texttt{CONCAT}(z_{x}^{ps},az_{x}^{pc}). \tag{19}\]

Note that \(\alpha\) can be learned by using the attention mechanism. To align it with the pre-training objective, we set \(\alpha=\lambda\) for simplicity in our implementation, where \(\lambda\) is used in Equation 16 as a balance coefficient for the two views.

**Prompt tuning.** To optimize the learnable prompt parameters, we next formulate the loss functions in prompt tuning.

For node/graph classification, given a labeled training set \(\mathcal{T}=\{(G_{1},Y_{1}),(G_{2},Y_{2}),\dots\}\) with a set of classes \(C\), where \(G_{i}\) is a labeled graph instance (e.g., a node or a graph) and \(Y_{i}\) is the class label of \(G_{i}\), the loss function for prompt tuning is then defined as:

\[\mathcal{L}_{prompt}=-\sum_{(G_{i},Y_{i})\in\mathcal{T}}\log\frac{exp(sim\left( z_{G_{i}}^{p},\widetilde{z_{Y_{i}}^{p}}\right)/\tau)}{\sum_{e\in Y}exp(sim \left(z_{G_{i}}^{p},\widetilde{z_{e}^{p}}\right)/\tau)}, \tag{20}\]

where \(\widetilde{z_{e}^{p}}\) denotes the prompted representation of prototype for class \(c\in C\). For link prediction, given a node \(v\) on graph \(G\), we randomly sample one positive node \(a\) from \(v\)'s adjacent neighbors, and one negative node \(b\) that does not directly link to \(v\), forming a triplet \((v,a,b)\). Our objective is to increase the similarity between graph instance \(S_{v}\) and \(S_{a}\), while decreasing that between \(S_{v}\) and \(S_{b}\). We sample a number of triplets from the graph to construct a training set \(\mathcal{T}\). Then our objective is given as:

\[\mathcal{L}_{prompt}=-\sum_{(v,a,b)\in\mathcal{T}}\log\frac{exp(sim\left(z_{v }^{p},z_{a}^{p}\right)/\tau)}{\sum_{u\in\{x,y\}}exp(sim\left(z_{v}^{p},z_{u} ^{p}\right)/\tau)}. \tag{21}\]

Note that, unlike fine-tuning, prompt tuning freezes parameters in the pre-training stage, while updating only a few parameters, e.g., \(\mathbf{P}\) and \(\mathbf{p}^{s},\mathbf{p}^{c}\) in our cases. This significantly decreases the difficulty in model training. More analysis about model complexity are presented in Appendix 6.

## 5. Experiments

In this section, we perform experiments on benchmark datasets to evaluate the proposed PGCL.

### Experimental Settings

**Datasets.** We employ 12 datasets in total, which can be divided into three groups. The first group is homophilous graphs, which include Cora, Citeseer, PubMed and DBLP (Cordes and Manning, 2015; Chen et al., 2016). These datasets are citation networks that are widely used for node classification and link prediction. In these datasets, nodes represent publications and edges are citations between them. Further, node features are the bag-of-words representations of keywords contained in the publications. The second group is heterophilous graphs. We adopt four public datasets from: Chameleon, Cornell, Texas, Wisconsin (Cordes and Manning, 2015). Specifically, these datasets are web networks, where nodes are web pages and edges are hyperlinks. The last group is graph classification dataset including PROTEINS, COX2, ENZYMES, BZR (Cordes and Manning, 2015; Chen et al., 2016; Chen et al., 2016). These datasets are a collection of molecular structure graphs.

**Baselines.** To evaluate the effectiveness of PGCL, we mainly compare it with the state-of-the-art approaches from three main categories (**1) End-to-end GNN methods**: GCN (Kipf and Welling, 2015), GraphSAGE (Kipf and Welling, 2015), GAT (Yang et al., 2016), GIN (Yang et al., 2016) and H\({}_{2}\)GCN (Yang et al., 2016). These methods directly train a GNN model on a specific task and work in an end-to-end manner.

**(2) Graph pre-training methods**: GAE (VGAE) (Vgg and Welling, 2015), DGI (Yang et al., 2016), InfoGraph (Zhu et al., 2017), GRACE (Wang et al., 2017), MVGRL (Kipf and Welling, 2015) and GraphCL (Yang et al., 2016). These methods pre-train a GNN model in a self-supervised way work and fine-tune for the downstream task. **(3) Graph prompt methods**: GPPT (Yang et al., 2016) and GraphPrompt (Zhu et al., 2017). These methods utilize the link prediction task for pre-training, and reformulate downstream tasks into the common complete. Note that meta-learning methods cannot be compared in our setting, as they require labeled data in their base classes for the meta-training phase.

**Setup.** To evaluate the goal of PGCL in better utilization of the capabilities of pre-trained model and generalizability of different tasks. We mainly consider three typical types of downstream tasks, i.e., node classification and graph classification in few-shot settings and link prediction. For node classification and graph classification, we follow GraphPrompt (Zhu et al., 2017) and construct a series of \(k\)-shot classification tasks. The details of task construction will be elaborated later when reporting the results. For task evaluation, as the \(k\)-shot tasks are balanced classification, we employ accuracy as the evaluation metric. For all the baselines, based on the authors' code and
default settings, we further tune their hyper-parameters to optimize their performance.

### Performance Comparison

We conduct various types of downstream task, namely, few-shot node classification on homophilous and heterophilous graphs, few-shot graph classification, and link prediction and compare the performance of PGCL with other methods.

**Few-shot on homophilous graphs.** We conduct this node classification on homophilous graphs on four datasets, i.e., Cora, Citeseer, PubMed and DBLP. Following the \(k\)-shot setup (Zhou et al., 2017), we generate a series of few-shot tasks for model training and validation. In particular, we randomly generate ten 1-shot node classification tasks (i.e., we randomly sample 1 node per class) for training and validation, respectively. Each training task is paired with a validation task, and the remaining nodes not sampled by the pair of training and validation tasks will be used for testing. Table 1 illustrates the results of few-shot node classification on homophilous graphs. We have the following observations:

(1) End-to-end GNN models achieve poor performance in most cases, demonstrating that they heavily depend on task-specific labeled data as supervision.

(2) Compared to end-to-end GNN models, pre-traing GNN models achieve even worse performance on PubMed. This implies that they have not effectively transferred knowledge to downstream tasks.

(3) PGCL outperforms all the baselines across all datasets, demonstrating that the unified framework can fully leverage the capabilities of pre-trained models by aligning the objectives of pre-training and downstream tasks.

**Few-shot on heterophilous graphs.** We conduct this node classification on heterophilous graphs on four datasets, i.e., Chameleon, Cornell, Texas, Wisconsin. We following the \(k\)-shot setup of experiments on homophilous graphs. Note that we additionally consider MLP as our baseline in our experiments. Table 2 shows the results of few-shot node classification on heterophilous graphs. We have the following observations:

(1) PGCL consistently leads to the best results on all the datasets and shows superiority over H\({}_{2}\)GCN, which is specially designed for graphs with heterophily.

(2) GAE, GPT and GraphPrompt achieve poor performance on heterophilic datasets. This implies that link prediction as pretext overemphasizes the structure information, thereby neglecting the important semantic information on graphs.

(3) MLP achieves good results on Cornell, Texas and Wisconsin, indicating that node features only can play a vital role. Basic GCN performs relatively well on Chameleon, suggesting that contextual information also counts too. Therefore, capturing information on graphs through semantic and contextual views in PGCL provides a more comprehensive approach.

**Few-shot on graph classification.** We conduct this graph classification on four datasets, i.e., PROTEINS, ENZYMES, COX2, BZR. For each dataset, we randomly generate 100 5-shot classification tasks for training and validation, following a similar process for node classification tasks. We illustrate the results of few-shot graph classification in Table 3, and have the following observations:

(1) GraphPrompt and PGCL outperform other baselines, once again demonstrating that the effectiveness of the prompt design and unified framework for pre-training and downstream tasks.

(2) Note that we use the raw features in these datasets to initialize input feature vectors, so PGCL can simultaneously captures information from both features and graph structure. The superior results on graph classification demonstrate that the fusion operation of PGCL provides more inter-class information in few-shot setting.

**Link prediction.** We conduct this link prediction on three datasets, i.e., Cora, Citeseer and PubMed. We follow previous studies (Cong et al., 2019) to construct the train/valid/test edge sets. To be specific, we randomly split all edges into three sets, i.e., the training set (85%), the validation set (5%), and the test set (10%), and evaluate the performance based on AUC and AP scores. We compare our PGCL with two generation-based methods: GAE and VGAE, and two contrastive methods: GRACE and MVGRL. Table 4 list the results and note that PGCL_np is our method without prompt tuning. We have the following observation:

(1) Generative baselines perform generally better than contrastive methods and PGCL_np, which is consistent with common belief.

(2) PGCL leads to a much larger performance gap compared with MVP_np and performs generative baselines on Cora and Citeseer and achieves comparable results on PubMed, indicating that the

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline
**Method** & **Cara** & **Citeseer** & **PubMed** & **DBLP** \\ \hline GCN & 43.04\(\pm\)8.06 & 33.83\(\pm\)9.53 & 54.38\(\pm\)7.59 & 42.93\(\pm\)10.85 \\ GAT & 45.33\(\pm\)7.01 & 36.72\(\pm\)8.52 & 55.74\(\pm\)9.12 & 38.63\(\pm\)9.83 \\ GraphSAGE & 43.27\(\pm\)8.05 & 32.68\(\pm\)10.23 & 54.32\(\pm\)10.11 & 39.91\(\pm\)12.68 \\ \hline GAE & 42.51\(\pm\)6.53 & 40.68\(\pm\)9.23 & 51.72\(\pm\)8.92 & 39.73\(\pm\)12.05 \\ DGI & 49.41\(\pm\)7.72 & 43.19\(\pm\)8.73 & 50.30\(\pm\)7.89 & 42.38\(\pm\)9.57 \\ MVGRL & 56.02\(\pm\)7.86 & 46.25\(\pm\)8.98 & 54.29\(\pm\)8.93 & 45.14\(\pm\)10.71 \\ GRACE & 55.56\(\pm\)8.76 & 46.64\(\pm\)5.98 & 54.16\(\pm\)9.59 & 47.78\(\pm\)11.90 \\ \hline GPPT & 51.63\(\pm\)8.76 & 42.89\(\pm\)7.93 & 50.98\(\pm\)9.12 & 39.48\(\pm\)12.08 \\ GraphPrompt & 55.32\(\pm\)9.56 & 44.19\(\pm\)9.73 & 53.49\(\pm\)6.68 & 43.32\(\pm\)10.87 \\ PGCL & **57.51\(\pm\)8.09** & **48.12\(\pm\)7.23** & **59.94\(\pm\)8.16** & **49.07\(\pm\)7.31** \\ \hline \hline \end{tabular}
\end{table}
Table 1. Node classification accuracy (%) on homophilous graphs. We highlight the best score for each dataset in bold.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline
**Method** & **Chameleon** & **Cornell** & **Texas** & **Wisconsin** \\ \hline GCN & 26.84\(\pm\)7.52 & 23.93\(\pm\)9.33 & 19.48\(\pm\)11.12 & 19.84\(\pm\)13.67 \\ GAT & 25.75\(\pm\)5.46 & 23.77\(\pm\)10.25 & 20.40\(\pm\)12.07 & 18.27\(\pm\)10.19 \\ MLP & 22.67\(\pm\)2.88 & 29.61\(\pm\)11.34 & 33.74\(\pm\)8.82 & 36.54\(\pm\)10.72 \\ H\({}_{2}\)GCN & 26.95\(\pm\)3.73 & 28.38\(\pm\)12.52 & 34.37\(\pm\)15.82 & 35.21\(\pm\)15.35 \\ \hline GAE & 22.80\(\pm\)3.63 & 21.44\(\pm\)8.97 & 26.89\(\pm\)16.91 & 17.71\(\pm\)8.79 \\ DGI & 25.42\(\pm\)4.82 & 24.35\(\pm\)9.74 & 21.21\(\pm\)12.71 & 24.64\(\pm\)11.89 \\ MVGRL & 26.45\(\pm\)4.01 & 26.23\(\pm\)9.62 & 24.55\(\pm\)11.49 & 25.82\(\pm\)13.35 \\ GRACE & 24.68\(\pm\)3.79 & 19.51\(\pm\)8.07 & 18.58\(\pm\)9.36 & 19.95\(\pm\)9.94 \\ \hline GPPT & 25.05\(\pm\)3.68 & 27.57\(\pm\)9.13 & 22.96\(\pm\)12.89 & 30.44\(\pm\)10.77 \\ GraphPrompt & 25.62\(\pm\)4.66 & 28.67\(\pm\)7.24 & 23.13\(\pm\)11.89 & 28.54\(\pm\)7.64 \\ PGCL & **30.45\(\pm\)3.14** & **39.52\(\pm\)8.71** & **40.17\(\pm\)7.88** & **47.48\(\pm\)10.24** \\ \hline \hline \end{tabular}
\end{table}
Table 2. Node classification accuracy (%) on heterophilous graphs. For each dataset, we highlight the best result in bold.
prompt design effectively tune output of pre-trained model to adapt to downstream task.

## 6. Model Analysis

**Time complexity.** We analyze the time complexity of main components in our model. Assume an input graph has \(n\) nodes and \(m\) edges, let the GNN encoder contain \(l\) layers. During the pre-training phrase, the time complexities of GNN encoder is \(O(dml+nd^{\prime}dl)\), where \(d\) is the dimensionality of initial features, \(d^{\prime}\) is the dimensionality of final representations. Let \(q\) be the number of selected negative samples in a batch, the complexity of contrastive loss is \(O(nqd^{\prime})\). During the prompt tuning phrase, the time complexity for \(k\)-shot classification task with \(c\) classes is \(O(kc^{2}d^{\prime})\), and the time complexity for link prediction is \(O(nd^{\prime})\). During the inference phrase, the time complexity for classification task is \(O(cd^{\prime})\), and the time complexity for link prediction is \(O(d^{\prime})\).

**Parameter efficiency.** We also compare the number of parameters that needs to be updated in a downstream classification task with a few representative models. As can be seen in Table 5, GCN works in an end-to-end manner, it is obvious that it involves the largest number of parameters for updating. GRACE and MVGRL employ a linear classifier for node classification, so the parameters in the classifier need to be updated in the downstream task. For our proposed PGCL, it not only outperforms the baselines GCN, GRACE and MVGRL as we have seen earlier, but also requires the least parameters, demonstrating the superiority of graph prompting.

## 7. Conclusions

In this paper, we proposed a multi-view pre-train and unified downstream tasks prompting method PGCL. In particular, to narrow the gap between pre-training and downstream objectives on graph, we reformulated pre-training pretexts and downstream tasks on graphs into a common template. In the pre-training phase, we proposed a multi-view contrastive learning method to capture the semantic and contextual structure information on graphs. In the prompt tuning stage, we introduced a learnable prompt strategy to transfer the pre-trained knowledge to different downstream tasks. Finally, we extensively evaluated the performance of our method on 12 public datasets. Our experimental results demonstrate the effectiveness of our framework.
