\begin{table}[t]
\caption{{Comparison with finetuning by MAML++ and ALFA.}}
\label{tab:node-alfa}
\vspace{1mm}
\small
\renewcommand\arraystretch{1}
\centering
\resizebox{0.4\textwidth}{!}{\begin{tabular*}{0.45\textwidth}{c| @{\extracolsep{\fill}} |c|c}
\toprule
{Model} &  3-shot & 5-shot \\
\midrule
MAML++       &55.9$\pm$2.1 &59.4$\pm$3.2 \\
ALFA       &54.8$\pm$3.6 &58.8$\pm$3.7 \\
Ours (with MAML)       &55.2$\pm$2.5 &58.7$\pm$2.7  \\
\bottomrule
\end{tabular*}
}
\end{table}
\textbf{GFL Setting and Problem}. Let $\mathcal{C}_{base}$ and $\mathcal{C}_{novel}$ denote the base classes set and novel (new) classes set in training data $\mathcal{T}_{train}$ and testing data $\mathcal{T}_{test}$, respectively. 
Similar to the general meta-learning problem~\cite{finn2017model}, the purpose of graph few-shot learning (GFL) is to train a GNN encoder $f_\theta(\cdot)$ over $\mathcal{C}_{base}$, such that the trained GNN encoder can be quickly adapted to $\mathcal{C}_{novel}$ with few labels per class. Note that there is no overlapping between base classes and novel classes, i.e., $\mathcal{C}_{base} \cap \mathcal{C}_{novel} = \emptyset$. 
In $K$-shot setting, during the meta-training phase,
a batch of classes (tasks) is randomly sampled from $\mathcal{C}_{base}$, where $K$ labeled instances per class are sampled to form the support set $\mathcal{S}$ for model training and the remaining instances are taken as the query set $\mathcal{Q}$ for model evaluation. After sufficient training, the model is further transferred to the meta-testing phase to conduct $N$-way classification over $\mathcal{C}_{novel}$ ($N$ is the number of novel classes), where each class is only with $K$ labeled instances. GFL applies to different graph mining problems, depending on the class meaning. Each class corresponds to a node label for the node classification problem or corresponds to a graph label for the graph classification problem. In this work, we will study both node classification and graph classification problems under few-shot setting, which are formally defined as follows:

where $\tau \in [0, 1]$ is the decay rate. Note that the target GNN stops the backpropagation from $\mathcal{L}_{\theta,\xi}$, and it is only updated by EMA.
\textbf{Contrastive Distillation}.
With the pre-trained GNN $f_\theta(\cdot)$ obtained in the previous step, we introduce a self-distillation step to elevate $f_\theta(\cdot)$. This is inspired by the Born-again strategy~\cite{furlanello2018born}, which implies a well-trained teacher can boost a random initialized identical student. The distillation step adopts a similar contrastive framework as the previous step, as shown in Figure~\ref{fig:ssl}(a). Specifically, we load the pre-trained GNN $f_\theta(\cdot)$ and take it as the teacher model $f_{\theta_{t}}(\cdot)$. The teacher model is frozen and applied to distill a student model $f_{\theta_{s}}(\cdot)$.
Two augmented views ($G^{*}$, $G^{**}$) of a graph $G$ are generated and fed to $f_{\theta_{t}}(\cdot)$ and $f_{\theta_{s}}(\cdot)$, respectively. Later,
the student's normalized output is forced to approximate the teacher's normalized output as follows:

where $x_{i}$ denotes a random variable of $i$-th node attribute in the graph. 
Then, we introduce a  noise perturbation-based method to approximate $H(x_{i}|{\bf {z}})$. Specifically,  let $\widetilde{x_i}=x_i+{\bf \epsilon}_i$ (${\bf \epsilon}_i \sim \mathcal{N}(0, {\bf \Sigma}_i={\bf \sigma}^2_i{\bf I})$) and we aim to optimize the following loss function:

where $\bm{\sigma}$ = $[\sigma_1, \sigma_2, \cdots, \sigma_n]$ are learnable parameters; 
$\beta$ is trade-off weight. In particular, the first term of the above objective minimizes difference between the encoded embedding of noisy input and the hidden state while the second term encourages a high conditional entropy according to the maximum entropy principle. In this way, we have $p(\widetilde{x_i}|{\bf z})=p({\bf \epsilon}_i)$ and $H(x_i|{\bf z})$ is approximated by $H(\widetilde{x}_i|{\bf z})$:

where $C = \frac{1}{2}\log (2\pi e)$. By taking Eqn.~\ref{eq:infoloss-7} into Eqn.~\ref{eq:infoloss-6}, 
we can optimize $\mathcal{L}(\bm{\sigma})$ with Adam optimizer and obtain the optimal $\bm{\sigma}$ for computing the overall discarded information $H(G|Z)$. Furthermore, we can explain GFL model's capability by comparing discarded information of different models. Ideally, we expect the GFL model to encode valid node (or graph) embeddings as much as possible and therefore discard information as small as possible. 

\subsection{Experimental Setup}
\label{sec:exp-set}
\textbf{Datasets}.
We use multiple graph datasets to conduct experiments: for the node classification task, we use ogbn-arxiv~\citep{hu2020open}, Tissue-PPI~\citep{hamilton2017inductive}, Fold-PPI~\citep{zitnik2017predicting}, Cora~\citep{sen2008collective}, and Citeseer~\citep{sen2008collective}; for the graph classification task, we use datasets in ~\citep{chauhan2020few}, i.e., Letter-High, Triangles, Reddit-12K, and Enzymes. The detailed information of datasets is illustrated in Appendix~\ref{app: dataset}. 
\\
\textbf{Baseline Methods}. 
We employ a variety of baseline methods for model comparison in two tasks. For few-shot node classification, we use node2vec~\citep{grover2016node2vec}, DeepWalk~\citep{perozzi2014deepwalk}, Meta-GNN~\citep{zhou2019meta}, FS-GIN~\citep{xu2018powerful}, FS-SGC~\citep{wu2019simplifying}, No-Finetune~\citep{triantafillou2019meta},
Finetune~\citep{triantafillou2019meta},
KNN~~\citep{triantafillou2019meta},  ProtoNet~\citep{snell2017prototypical}, MAML~\citep{finn2017model}, G-Meta~\citep{huang2020graph}, and TENT~\citep{wang2022task}. For few-shot graph classification, we utilize WL~\citep{shervashidze2011weisfeiler}, Graphlet~\citep{shervashidze2009efficient}, AWE~\citep{ivanov2018anonymous}, Graph2Vec~\citep{narayanan2017graph2vec}, Diffpool~\citep{lee2019self}, CapsGNN~\citep{xinyi2018capsule}, GIN~\citep{xu2018powerful}, GIN-KNN~\citep{xu2018powerful}, GSM-GCN~\citep{chauhan2020few}, GSM-GAT~\citep{chauhan2020few}, and AS-MAML~\citep{ma2020adaptive}. Details of baselines are illustrated in Appendix~\ref{app: implementation}. 
\\
\textbf{Experimental Settings}.
In CGFL, we use GCN~\citep{kipf2016semi} as the GNN backbone for the node classification task, including two-layer graph convolution and one-layer FC. The graph classification task adds an extra average pooling operation as a readout layer. 
In the pretraining phase, we pre-train GNN on the unlabeled graph dataset with contrastive learning. 
For graph data augmentation, the node drop rate is 15\%, the edge removing rate is 15\%; and the feature masking rate is 20\%. The mini-batch size is set to 2,048, and the learning rate is set to 0.05 with a decay factor = 0.9. Meanwhile, the $\tau$ in exponential moving average is 0.999. We consider both inductive and transductive settings. For the inductive setting (\textbf{CGFL-I}), we only use unlabeled data in the training set; for the transductive setting (\textbf{CGFL-T}), we use unlabeled data in both training data and testing data. Additionally, for pre-trained models without knowledge distillation elevation, we refer to them as \textbf{Teacher-I} and \textbf{Teacher-T} for two settings, respectively. 
In GFL phase, we employ MAML to fine-tune the model.
We implement CGFL by PyTorch and train it on NVIDIA V100 GPUs. 
The code is in the supplementary material. 

\\
\textbf{Impact of Shot Number}.
In Figure~\ref{fig:node-shot}, we show the performance of our model (CGFL-T) under different shot numbers (1 to 5) compared with some selected baselines. It is easy to see that CGFL achieves better accuracy across different shot numbers.
The result demonstrates our model's performance is robust for node classification. Note that we only show results on two datasets and the results on the other datasets (i.e., ogbn-arxiv and Tissue-PPI) are shown in Appendix~\ref{sec:app:fsn}. The same goes for the following analyses. 
\\
\textbf{Impact of Training Label Rate}.
We further evaluate CGFL's performance under different training label rates (10\%, 20\%, 30\%, 40\%, 50\%, 100\%) compared with baseline methods for 3-shot node classification, as shown in Figure~\ref{fig:node-lr}. It is easy to see that (1) CGFL has better results across different label rates; (2) CGFL achieves larger improvement over baseline models when label rate becomes lower (e.g., 10\%), showing contrastive pre-training and self-distillation of CGFL lead to more significant improvement for label sparsity case. 
\\
\textbf{Impact of Data Augmentation}.
As a key step for contrastive learning in CGFL, graph augmentation plays an important role in affecting model performance. Here we conduct experiments to evaluate the model performance with different augmentation strategies. We consider three graph augmentations - node dropping (\textbf{ND}), feature masking (\textbf{FM}), edge removing (\textbf{ER}), and their combinations for CGFL phase and report their performances in Figure~\ref{fig:node-da}. It is easy to find that the combination of three augmentation strategies works better than a single augmentation or the combination of two augmentations. It demonstrates that various graph augmentations are able to generate sufficient contrastive pairs for a better model. 
\\

\subsection{{Apply to MAML++ and ALFA in Few-shot Learing Phase}}
{We implement MAML++ and ALFA (extended from MAML) in our CGFL to replace MAML under transductive setting on ogbn-arxiv dataset. The results are reported in  Table~\ref{tab:node-alfa}. Our aim is to design a contrastive pretraining framework which is plug-and-play for graph few-shot learning. So our CGFL is not limited to MAML. It is easy to be extended with more new methods (e.g., MAML++). Replacing MAML to MAML++ in CGFL brings better accuracy, which implies our framework is general and extendable.}
{"value": "55.9", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"3-shot": "true"}, "model": "MAML++", "model settings": {"xx": "yy"}} {"value": "59.4", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"5-shot": "true"}, "model": "MAML++", "model settings": {"xx": "yy"}} {"value": "54.8", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"3-shot": "true"}, "model": "ALFA", "model settings": {"xx": "yy"}} {"value": "58.8", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"5-shot": "true"}, "model": "ALFA", "model settings": {"xx": "yy"}} {"value": "55.2", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"3-shot": "true"}, "model": ["CGFL MAML", "Ours (with MAML)"], "model settings": {"xx": "yy"}} {"value": "58.7", "type": "Result", "training data/set": "ogbn-arxiv", "test data/set": "ogbn-arxiv", "task": ["Few-shot node classification", "node classification"], "metric": "Accuracy", "experimental settings": {"5-shot": "true"}, "model": ["CGFL MAML", "Ours (with MAML)"], "model settings": {"xx": "yy"}}