Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and
  Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework
Recent years have witnessed the great success of Graph Neural Networks (GNNs)
in handling graph-related tasks. However, MLPs remain the primary workhorse for
practical industrial applications due to their desirable inference efficiency
and scalability. To reduce their gaps, one can directly distill knowledge from
a well-designed teacher GNN to a student MLP, which is termed as GNN-to-MLP
distillation. However, the process of distillation usually entails a loss of
information, and ``which knowledge patterns of GNNs are more likely to be left
and distilled into MLPs?" becomes an important question. In this paper, we
first factorize the knowledge learned by GNNs into low- and high-frequency
components in the spectral domain and then derive their correspondence in the
spatial domain. Furthermore, we identified a potential information drowning
problem for existing GNN-to-MLP distillation, i.e., the high-frequency
knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency
knowledge during distillation; we have described in detail what it represents,
how it arises, what impact it has, and how to deal with it. In this paper, we
propose an efficient Full-Frequency GNN-to-MLP (FF-G2M) distillation framework,
which extracts both low-frequency and high-frequency knowledge from GNNs and
injects it into MLPs. Extensive experiments show that FF-G2M improves over the
vanilla MLPs by 12.6% and outperforms its corresponding teacher GNNs by 2.6%
averaged over six graph datasets and three common GNN architectures.
Lirong Wu, Haitao Lin, Yufei Huang, Tianyu Fan, Stan Z. Li
Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

###### Abstract

Recent years have witnessed the great success of Graph Neural Networks (GNNs) in handling graph-related tasks. However, MLPs remain the primary workhorse for practical industrial applications due to their desirable inference efficiency and scalability. To reduce their gaps, one can directly distill knowledge from a well-designed teacher GNN to a student MLP, which is termed as GNN-to-MLP distillation. However, the process of distillation usually entails a loss of information, and _"which knowledge patterns of GNNs are more likely to be left and distilled into MLPs?"_ becomes an important question. In this paper, we first factorize the knowledge learned by GNNs into low- and high-frequency components in the spectral domain and then derive their correspondence in the spatial domain. Furthermore, we identified a potential _information drowning_ problem for existing GNN-to-MLP distillation, i.e., the high-frequency knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge during distillation; we have described in detail what it represents, how it arises, what impact it has, and how to deal with it. In this paper, we propose an efficient _Full-Frequency GNN-to-MLP_ (FF-62M) distillation framework, which extracts both low-frequency and high-frequency knowledge from GNNs and injects it into MLPs. Extensive experiments show that FF-62M improves over the vanilla MLPs by 12.6% and outperforms its corresponding teacher GNNs by 2.6% averaged over six graph datasets and three common GNN architectures. Codes are publicly available at:

## Introduction

In many real-world applications, including social networks, chemical molecules, and citation networks, data can be naturally modeled as graphs. Recently, the emerging Graph Neural Networks (GNNs) [16, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] have demonstrated their powerful capability to handle various graph-related tasks [25, 26, 27, 28]. However, practical deployments of GNNs in the industry are still less popular due to inference efficiency and scalability challenges incurred by data dependency [21, 22]. In other words, GNNs generally rely on message passing to aggregate features from the neighborhood, but fetching and aggregating these nodes during inference can burden latency-sensitive applications. In contrast, Multi-Layer Perceptrons (MLPs) involve no data dependence between pairs of nodes and infer much faster than GNNs, but often with less competitive performance. Motivated by these complementary strengths and weaknesses, one solution to reduce their gaps is to perform GNN-to-MLP knowledge distillation [25, 26, 27, 28], which extracts knowledge from a well-trained teacher GNN and then distills it into a student MLP with the same network architecture (e.g., layer number and layer size).

Most of the existing GNN-to-MLP distillation methods [25, 26, 27] focus on special designs on either student MLPs or teacher GNNs, but default to distill knowledge in a node-to-node fashion. For example, CPF [26] combines Label Propagation (LP) [13] into the student MLPs to improve classification performance and thus still suffers from the neighborhood-fetching latency caused by label propagation, which defeats the original intention of MLPs to be inference-efficient. In contrast, GLNN [26] directly distills knowledge from arbitrary GNNs to vanilla MLPs with the same network architecture. While the distilled MLPs of GLNN can be greatly improved by employing more powerful teacher GNNs, the process of distillation usually entails a loss of information [28], which may lead to sub-optimal student MLPs. In this paper, we look away from specific instantiations of teacher GNNs and student MLPs, but rather explore two fundamental questions: (1) _Can existing GNN-to-MLP distillation ensure that sufficient knowledge is distilled from teacher GNNs to student MLPs?_ If not, (2) _Which knowledge patterns of GNNs are more likely to be distilled into student MLPs?_

**Present Work.** In this paper, we identify a potential _information drowning_ problem for existing GNN-to-MLP distillation, i.e., the high-frequency knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge during distillation. To illustrate this, we first factorize GNN knowledge into low- and high-frequency components using graph signal processing theory in the spectral domain and then derive their correspondence in the spatial domain. Moreover, we conduct a comprehensive investigation of the
roles played by low- and high-frequency components in the distillation process and describe in detail what _information drowning_ represents, how it arises, what impact it has, and how to deal with it. Extensive experiments have shown that high-frequency and low-frequency knowledge are complementary to each other, and they can further improve performance on top of each other. In this paper, we propose a novel _Full-Frequency GNN-to-MLP_ (FF-G2M) distillation framework, which extracts both low- and high-frequency knowledge from teacher GNNs and injects it into student MLPs.

## Related Work

**Graph Neural Networks (GNNs).** The early GNNs define graph convolution kernels in the spectral domain [1, 13] based on the graph signal processing theory, known as ChebyNet [13] and Graph Convolutional Networks (GCN) [14]. The later GNNs directly define updating rules in the spatial space and focus on the design of neighborhood aggregation functions. For instance, GraphSAGE [1] employs a generalized induction framework to generate embeddings for previously unseen nodes by aggregating known node features. Moreover, GAT [15] introduces the self-attention mechanism to assign different importance scores to neighbors for better information aggregation. We refer interested readers to the surveys [16, 17, 18] for more GNN architectures.

**Graph Knowledge Distillation.** Despite the great progress, most existing GNNs share the de facto design that relies on message passing to aggregate features from neighborhoods, which may be one major source of latency in GNN inference. To address this problem, there are previous works that attempt to distill knowledge from large teacher GNNs to smaller student GNNs, termed as GNN-to-GNN [1, 17, 16, 18]. For example, the student model in RDD [19] and TinyGNN [16] is a GNN with fewer parameters but not necessarily fewer layers than the teacher GNN, which makes both designs still suffer from the neighborhood-fetching latency caused by data dependency.

To enjoy the low-latency of MLPs and high-accuracy of GNNs, the other branch of graph knowledge distillation is to directly distill from large teacher GNNs to student MLPs, termed as GNN-to-MLP. The existing work on GNN-to-MLP distillation can be mainly divided into two branches: student MLPs-focused and teacher GNNs-focused. The former branch, such as CPF [16], _directly_ improves student MLPs by adopting deeper and wider network architectures or incorporating label propagation, both of which burden the inference latency. The other branch, such as GLNN [19], distills knowledge from teacher GNNs to vanilla MLPs with the same network architectures but without other computing-consuming operations; while the performance of their distilled MLPs can be _indirectly_ improved by employing more powerful GNNs, they still cannot match their corresponding teacher GNNs. Moreover, PGKD [17] proposes a Prototype-Guided Knowledge Distillation (PGKD) method, which does not require graph edges yet learns structure-aware MLPs. In this paper, we aim to develop a **model-agnostic** GNN-to-MLP distillation that is applicable to various GNN architectures.

**Noticons.** Let \(\mathcal{G}=(\mathcal{V},\mathcal{E},\mathbf{X})\) be an attributed graph, where \(\mathcal{V}\) is the set of \(N\) nodes with features \(\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N}]\in\mathbb{R} ^{N\times d}\) and \(\mathcal{E}\) denotes the edge set. Each node \(v_{i}\in\mathcal{V}\) is associated with a \(d\)-dimensional features vector \(\mathbf{x}_{i}\), and each edge \(e_{i,j}\in\mathcal{E}\) denotes a connection between node \(v_{i}\) and \(v_{j}\). The graph structure is denoted by an adjacency matrix \(\mathbf{A}\in[0,1]^{N\times N}\) with \(\mathbf{A}_{i,j}=1\) if \(e_{i,j}\in\mathcal{E}\) and \(\mathbf{A}_{i,j}=0\) if \(e_{i,j}\notin\mathcal{E}\). Consider a semi-supervised node classification task where only a subset of node \(\mathcal{V}_{L}\) with labels \(\mathcal{Y}_{L}\) are known, we denote the labeled set as \(\mathcal{D}_{L}=(\mathcal{V}_{L},\mathcal{Y}_{L})\) and unlabeled set as \(\mathcal{D}_{U}=(\mathcal{V}_{U},\mathcal{Y}_{U})\), where \(\mathcal{V}_{U}=\mathcal{V}\backslash\mathcal{V}_{L}\). The node classification aims to learn a mapping \(\Phi:\mathcal{V}\rightarrow\mathcal{Y}\) with labeled data, so that it can be used to infer the labels \(\mathcal{Y}_{U}\).

**Graph Neural Networks (GNNs).** Most existing GNNs rely on message passing to aggregate features from the neighborhood. A general GNN framework consists of two key computations for each node \(v_{i}\): (1) \(\mathrm{AGGREGATE}\): aggregating messages from neighborhood \(\mathcal{N}_{i}\); (2) \(\mathrm{UPDATE}\): updating node representation from its representation in the previous layer and aggregated messages. Considering a \(L\)-layer GNN, the formulation of the \(l\)-th layer is as follows

\[\mathbf{m}_{i}^{(l)}= \,\mathrm{AGGREGATE}^{(l)}\left(\left\{\mathbf{h}_{i}^{(l-1)}:v_{ j}\in\mathcal{N}_{i}\right\}\right) \tag{1}\] \[\mathbf{h}_{i}^{(l)}= \,\mathrm{UPDATE}^{(l)}\left(\mathbf{h}_{i}^{(l-1)},\mathbf{m}_ {i}^{(l)}\right)\]

where \(1\leq l\leq L\), \(\mathbf{h}_{i}^{(0)}=\mathbf{x}_{i}\) is the input feature, and \(\mathbf{h}_{i}^{(l)}\) is the representation of node \(v_{i}\) in the \(l\)-th layer.

**Multi-Layer Perceptrons (MLPs).** To achieve efficient inference, the vanilla MLPs (with the same network architecture as the teacher GNNs) are used as the student model by default in this paper. For a \(L\)-layer MLP, the \(l\)-th layer is composed of a linear transformation, an activation function \(\sigma=\mathrm{ReLu}(\cdot)\), and a dropout function \(\mathrm{Dropout}(\cdot)\), as

\[\mathbf{z}_{i}^{(l)}=\mathrm{Dropout}\left(\sigma\big{(}\mathbf{z}_{i}^{(l-1)} \mathbf{W}^{(l-1)}\big{)}\big{)},\quad\mathbf{z}_{i}^{(0)}=\mathbf{x}_{i} \tag{2}\]

where \(\mathbf{W}^{(0)}\in\mathbb{R}^{d\times F}\) and \(\mathbf{W}^{(l)}\in\mathbb{R}^{F\times F}\)\((1\leq l<L)\) are weight matrices with the hidden dimension \(F\). In this paper, the network architecture of MLPs, such as the layer number \(L\) and layer size \(F\), is set the same as that of teacher GNNs.

**GNN-to-MLP Knowledge Distillation.** The knowledge distillation is first introduced in [10] to handle mainly image data, where knowledge is transferred from a cumbersome teacher model to a simpler student model. The later works on GNN-to-MLP distillation [16, 19, 16, 17] extend it to the graph domain by imposing KL-divergence constraint \(\mathcal{D}_{KL}(\cdot,\cdot)\) between the softmax label distributions generated by teacher GNNs and student MLPs and directly optimizing the objective function as follows

\[\mathcal{L}_{\mathrm{KD}}=\frac{1}{|\mathcal{V}|}\sum_{i\in\mathcal{V}} \mathcal{D}_{KL}\left(\mathrm{softmax}\left(\mathbf{z}_{i}^{(L)}\right),\mathrm{ softmax}\left(\mathbf{h}_{i}^{(L)}\right)\right) \tag{3}\]
## Knowledge Factorization from the Perspective of Spectral and Spatial Domain

In this section, we first theoretically factorize the knowledge learned by GNNs into low- and high-frequency components in the spectral domain based on graph signal processing theory (Shuman et al., 2013). The normalized graph Laplacian matrix of graph \(\mathcal{G}\) is defined as \(\mathbf{L}=\mathbf{I}_{N}-\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{ \mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}\), where \(\widetilde{\mathbf{A}}=\mathbf{A}+\mathbf{I}_{N}\in\mathbb{R}^{N\times N}\) is an adjacency matrix with self-loop, \(\widetilde{\mathbf{D}}\in\mathbb{R}^{N\times N}\) is a diagonal degree matrix with \(\widetilde{\mathbf{D}}_{i,i}=\sum_{j}\widetilde{\mathbf{A}}_{i,j}\), and \(\mathbf{I}_{N}\) denotes the identity matrix. Since \(\mathbf{L}\) is a real symmetric matrix, it can be eigendecomposed as \(\mathbf{L}=\mathbf{U}\Lambda\mathbf{U}^{\top}\), where \(\Lambda=\mathrm{diag}\left([\lambda_{1},\lambda_{2},\cdots,\lambda_{N}]\right)\) with each eigenvalue \(\lambda_{l}\in[0,2]\) corresponding to an eigenvectors \(\mathbf{u}_{l}\) in \(\mathbf{U}\)(Chung and Graham, 1997). According to graph signal processing theory, we can directly take the eigenvector \(\{\mathbf{u}_{l}\}_{l=1}^{N}\) as bases. Given signal \(\mathbf{x}\in\mathbb{R}^{d}\), the graph Fourier transform and inverse Fourier transform (Sandryhaila and Moura, 2013; Ricaud et al., 2019) are defined as \(\widehat{\mathbf{x}}=\mathbf{U}^{\top}\mathbf{x}\) and \(\mathbf{x}=\mathbf{U}\widehat{\mathbf{x}}\). Thus, the convolutional \(*_{G}\) between the signal \(\mathbf{x}\) and convolution kernel \(\mathcal{F}\) can be defined as follows

\[\mathcal{F}*_{G}\mathbf{x}=\mathbf{U}\left(\left(\mathbf{U}^{\top}\mathcal{F }\right)\odot\left(\mathbf{U}^{\top}\mathbf{x}\right)\right)=\mathbf{U} \mathbf{g}_{\theta}\mathbf{U}^{\top}\mathbf{x} \tag{4}\]

where \(\odot\) denotes the element-wise product and \(\mathbf{g}_{\theta}\) is a parameterized diagonal matrix. Most of the existing GNNs architectures can be regarded as a special instantiation on the convolutional kernel \(\mathcal{F}\) (i.e., the matrix \(\mathbf{g}_{\theta}\)). For example, GCN-Cheby parameterizes \(g_{\theta}\) with a polynomial expansion \(\mathbf{g}_{\theta}=\sum_{k=0}^{K-1}\alpha_{k}\Lambda^{k}\), and GCN defines the convolutional kernel as \(\mathbf{g}_{\theta}\!=\!\mathbf{I}_{N}-\Lambda\). Considering a special convolution kernel \(\mathcal{F}_{A}\!=\!\mathbf{I}_{N}\), we have \(\mathcal{F}_{A}*_{G}\mathbf{x}\!=\!\mathbf{U}\mathbf{I}_{N}\mathbf{U}^{\top} \mathbf{x}\!=\!\mathbf{U}\widehat{\mathbf{x}}\!=\!\mathbf{x}\), i.e., this is an identity mapping, where all information can be preserved. Next, we decompose the graph knowledge into low-frequency and high-frequency components (Bo et al., 2021; Wu et al., 2019) by factorizing \(\mathcal{F}_{A}\!=\!\mathbf{I}_{N}\) as follows

\[\mathcal{F}_{A}\!=\!\mathbf{I}_{N}\!=\!\frac{1}{2}\!\left(\left(\underbrace{ \mathbf{I}_{N}+\widetilde{\mathbf{D}}^{-\frac{1}{2}}\widetilde{\mathbf{A}} \widetilde{\mathbf{D}}^{-\frac{1}{2}}}_{\text{Low-Phase Filter}\,\,\mathcal{F}_{M}} \right)+\left(\underbrace{\mathbf{I}_{N}-\widetilde{\mathbf{D}}^{-\frac{1}{2}} \widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}}}_{\text{High- Phase Filter}\,\,\mathcal{F}_{H}}\right)\right)\]

For a given signal \(\mathbf{x}\in\mathbb{R}^{d}\), e.g., node feature, we have \(\mathcal{F}_{A}*_{G}\mathbf{x}=\frac{1}{2}\left(\mathcal{F}_{M}+\mathcal{F}_{H} \right)*_{G}\mathbf{x}=\frac{1}{2}(\mathcal{F}_{M}*_{G}\mathbf{x}+\mathcal{F} _{H}*_{G}\mathbf{x})=\mathbf{x}\), which means that any signal \(\mathbf{x}\) can be decomposed into the average of two components \(\mathcal{F}_{M}*_{G}\mathbf{x}\) and \(\mathcal{F}_{H}*_{G}\mathbf{x}\).

**Analysis on the Spectral Domain.** The Proposition 1 states what the two components \(\mathcal{F}_{M}*_{G}\mathbf{x}\) and \(\mathcal{F}_{H}*_{G}\mathbf{x}\) represent.

**Proposition 1**: _The convolution kernel \(\mathcal{F}_{M}\) works as a low-pass filter, which filters out high-frequency information, and \(\mathcal{F}_{M}*_{G}\mathbf{x}\) represents low-frequency knowledge; \(\mathcal{F}_{H}\) works as a high-pass filter, which filters out low-frequency information, and \(\mathcal{F}_{H}*_{G}\mathbf{x}\) represents high-frequency knowledge._

**Proof 1**: _For a \(L\)-layer GNN, the signal \(\mathbf{x}\) is filtered by the \(L\)-order convolution kernel \(\mathcal{F}_{M}^{L}=(\mathbf{I}_{N}+\widetilde{\mathbf{D}}^{-\frac{1}{2}} \widetilde{\mathbf{A}}\widetilde{\mathbf{D}}^{-\frac{1}{2}})^{L}=(2\mathbf{I}_ {N}-\mathbf{L})^{L}\) to output \(\mathcal{F}_{M}^{L}*_{G}\mathbf{x}=\mathbf{U}(2\mathbf{I}_{N}-\Lambda)^{L} \mathbf{U}^{\top}\mathbf{x}\) with \(g_{\theta}^{L}(\lambda_{i})=(2-\lambda_{i})^{L}\). As shown in Fig. 1, \(g_{\theta}^{L}(\lambda_{i})\) decreases monotonically in the range \(\lambda_{i}\in[0,2]\) and reaches \(g_{\theta}^{L}(\lambda_{i}\!=\!2)=0\) at \(\lambda_{i}=2\), which mainly amplifies the low-frequency information and filters out the high-frequency information. Similarly, the \(L\)-order convolution kernel \(\mathcal{F}_{H}^{L}\) has \(g_{\theta}^{L}(\lambda_{i})=\lambda_{i}^{L}\). As shown in Fig. 1, \(g_{\theta}^{L}(\lambda_{i})\) increases monotonically in the range \(\lambda_{i}\in[0,2]\) and reaches \(g_{\theta}^{L}(\lambda_{i}=0)=0\) at \(\lambda_{i}=0\), which mainly filters out the low-frequency information but in turn amplifies the high-frequency information._

**Correspondence on the Spatial Domain.** We have derived that \(\mathcal{F}_{M}*_{G}\mathbf{x}\) and \(\mathcal{F}_{H}*_{G}\mathbf{x}\) represent mainly the low- and high-frequency components of signal \(\mathbf{x}\), and we next can derived their correspondences in the spatial domain, as follows

\[\mathcal{F}_{M}*_{G}\mathbf{x}_{i} \rightarrow \mathbf{x}_{i}^{(low)}=\mathbf{x}_{i}+\sum_{j\in\mathcal{N}_{i}} \frac{\mathbf{x}_{j}}{\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}} \tag{5}\] \[\mathcal{F}_{H}*_{G}\mathbf{x}_{i} \rightarrow \mathbf{x}_{i}^{(high)}=\mathbf{x}_{i}-\sum_{j\in\mathcal{N}_{i}} \frac{\mathbf{x}_{j}}{\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}}\]

By the derivation in Eq. (5), the low-frequency knowledge \(\mathcal{F}_{M}*_{G}\mathbf{x}\) is the sum of node feature and its neighborhood features in the spatial domain. On the other hand, the high-frequency knowledge \(\mathcal{F}_{H}*_{G}\mathbf{x}\) represents the differences between the target node feature with its neighborhood features. There have recently been some novel GNN models (Bo et al., 2021; Pei et al., 2020; Zhu et al., 2020; Chien et al., 2021) that can capture both low- and high-frequency information simultaneously or adaptively. However, in this paper, we focus on the design of distillation objective functions and do not consider indirect performance improvements by employing these more powerful but complex GNNs. Instead, we consider the most commonly used GNNs, such as GCN (Kipf and Welling, 2016), GraphSAGE (Hamilton, Ying, and Leskovec, 2017), and GAT (Velickovic et al., 2017), all of which rely on multi-layer message passing to aggregate features of neighboring nodes that are multiple hops away, i.e., they essentially work as a low-pass filter \(\mathcal{F}_{M}^{L}\) or its variants.

## Roles Played by Low- and High-Frequency Knowledge during Distillation

### Rethinking the Core of Knowledge Distillation

We rethink the core of knowledge distillation from three shallow-to-deep perspectives to highlight our motivations.

* _Firstly_, knowledge distillation enables the representations of MLPs to "mimic" those of GNNs as closely as possible by imposing KL-divergence constraints between their softmax distribution probabilities. However, such a

Figure 1: Eigenvalues _vs._ Amplitudes
mimicking (or fitting) process is inevitably accompanied by a loss of information, especially high-frequency information, which explains why the performance of student MLPs is always hard to match with that of teacher GNNs.
* _Secondly_, for a neural network framework, any change in the final representations is achieved indirectly by optimizing the mapping function, i.e., the network parameters. In this sense, knowledge distillation essentially optimizes the parameter matrices \(\{\mathbf{W}^{(l)}\}_{l=0}^{L-1}\) of the student MLPs to make it functionally approximates the convolution kernel of the teacher GNNs, which makes the student MLPs also serve as a low-pass filter \(\widetilde{\mathcal{F}}_{M}^{L}\) for graph data.
* _Finally_, the low-pass filter in the spectral domain is equivalent to neighborhood aggregation in the spatial domain as derived in Eq. (5), which in essence can be considered as a special use of the graph topology.

To explore the roles played by graph topology during GNN-to-MLP distillation, we plot the _mean cosine similarity_ of nodes with their first-order neighbors for vanilla GCNs, vanilla MLPs, and Distilled MLPs (GLNN) on the Cora dataset in Fig. 2(a), from which we observe that the mean similarity of GCNs and GLNN gradually increases with training, while that of vanilla MLPs gradually decreases, which indicates that knowledge distillation has introduced graph topology as an inductive bias (as GCNs has done), while vanilla MLPs do not. As a result, the distilled MLPs can enjoy the benefits of topology-awareness in training but without neighborhood-fetching latency in inference.

### High-Frequency Information Drowning

Next, we discuss a potential high-frequency information drowning problem from both spectral and spatial domains, i.e., the high-frequency information of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge during the process of GNN-to-MLP knowledge distillation.

**How information drowning arises?**_From the perspective of spectral domain_, the knowledge distillation optimizes the network parameters of the student MLPs to make it functionally approximate the convolution kernel of the teacher GNNs, i.e., \(\widetilde{\mathcal{F}}_{M}^{L}\approx\mathcal{F}_{M}^{L}\). The information loss induced by such approximation may be inconsequential for high-amplitude low-frequency information but can be catastrophic for those high-frequency information with very low amplitude, as shown in Fig. 2(c). As a result, compared to low-frequency information, high-frequency information is more likely to be drowned by these optimization errors.

**What impact does information drowning have?**_From the perspective of spatial domain_, the information drowning may lead to distilled MLPs that, despite preserving neighborhood smoothing well, can easily neglect differences between nodes, such as pairwise distances. To illustrate this, we consider a target node \(v_{i}\) and its two neighboring nodes \(v_{j}\) and \(v_{k}\) in Fig. 2(d), where they are mapped closely by GNNs. In the process of knowledge distillation, the representations of these three nodes may be mapped around the representations of teacher GNNs, i.e., they are still mapped closely with most of their low-frequency information preserved; however, the relative distances between nodes, i.e., their high-frequency information, may be drowned dramatically. For example, node \(v_{i}\) is adjacent to node \(v_{j}\) but far from node \(v_{k}\) in the representation space of teacher GNNs. However, in the representation space of student MLPs, node \(v_{i}\) becomes closer to node \(v_{k}\) and farther from node \(v_{j}\).

The curves of the pairwise distance differences between the teacher GCNs and the student MLP in Fig. 2(b) show that common knowledge distillation (e.g., GLNN) is not good at capturing high frequency information, compared to our proposed FF-G2M. Moreover, extensive qualitative and quantitative experiments have been provided to demonstrate the harmfulness of the identified high-frequency information drowning problem in the experimental section. The detailed experimental settings, including hyperparameters and evaluation metric definitions, are available in **Appendix B&E**.

## Full-Frequency GNN-to-MLP (FF-G2M) Knowledge Distillation

The above discussions reached two important insights: (1) the inductive bias of graph topology plays an important role, and (2) it is mainly the low-frequency knowledge of graph data that has been distilled from the teacher GNNs to the student MLPs. Inspired by these two insights, we propose _Low-Frequency Distillation_ (LFD) and _High-Frequency Distillation_ (HFD) to fully capture the low-frequency and high-frequency knowledge learned by GNNs, respectively. An high-level overview of the proposed _Full-Frequency GNN-to-MLP_ (FF-G2M) framework is shown in Fig. 3.

Figure 2: (a) Mean cosine similarity (the higher, the better) between nodes with their first-order neighbors on Cora. (b) Pairwise distance differences (the lower, the better) between teacher GCNs and student MLPs on Cora. (c)(d) Illustrations of how the high-frequency information drowning arises and what potential impact it has in the spectral and spatial domains, respectively.
### Low-Frequency Distillation (LFD)

The node representations of teacher GNNs are generated by explicit message passing, so it mainly captures the low-frequency information of the graph data as analyzed earlier. Unlike aggregating features from neighborhoods as in GNNs, we directly distill (diffuse) knowledge from teacher GNNs into the neighborhoods of student MLPs in order to better utilize topological information and low-frequency knowledge captured by GNNs, formulated as follows

\[\mathcal{L}_{\mathrm{LFD}}\!=\!\frac{1}{|\mathcal{E}|}\sum_{i\in\mathcal{V}}\sum_ {j\in\mathcal{N}_{i}\cup i}\mathcal{D}_{KL}\Big{(}\sigma(\mathbf{z}_{j}^{(L)} /\tau_{1}),\sigma(\mathbf{h}_{i}^{(L)}/\tau_{1})\Big{)} \tag{6}\]

where \(\tau_{1}\) is the low-frequency distillation temperature, and \(\sigma=\mathrm{softmax}(\cdot)\) denotes an activation function.

### High-Frequency Distillation (HFD)

As derived in Eq. (5), the high-frequency components in the spectral domain represent the differences between node feature and its neighborhood features in the spatial domain. Inspired by this, we propose High-Frequency Distillation (HFD), a GNN knowledge objective that trains student MLPs to preserve the neighborhood pairwise differences from the representation space of teacher GNNs. The neighborhood pairwise differences around node \(v_{i}\) are defined as the differences between the target node feature \(\mathbf{s}_{i}\) and its neighborhood features \(\{\mathbf{s}_{j}\mid j\in\mathcal{N}_{i}\}\), which can be computed by the kernel \(\mathcal{K}\left(\mathbf{s}_{i},\mathbf{s}_{j}\right)=|\mathbf{s}_{i}-\mathbf{ s}_{j}|\), where \(|\cdot|\) denotes the element-wise absolute values. The high-frequency distillation trains the student model to mimic the neighborhood pairwise differences from the teacher GNNs via KL-divergence constraints, which can be defined as follows

\[\mathcal{L}_{\mathrm{HFD}}\!=\!\frac{1}{|\mathcal{E}|}\sum_{i\in \mathcal{V}}\sum_{j\in\mathcal{N}_{i}}\mathcal{D}_{\mathrm{KL}}\Big{(}\sigma \big{(}\mathcal{K}\big{(}\mathbf{z}_{i}^{(L)},\mathbf{z}_{j}^{(L)}\big{)}/ \tau_{2}\big{)}, \tag{7}\] \[\sigma\big{(}\mathcal{K}\big{(}\mathbf{h}_{i}^{(L)},\mathbf{h}_{ j}^{(L)}\big{)}/\tau_{2}\big{)}\Big{)}\]

where \(\mathcal{K}\left(\cdot,\cdot\right)\) denotes the element-wise absolute values, and \(\tau_{2}\) is the high-frequency distillation temperature.

### Training Strategy

The pseudo-code of the FF-G2M framework is summarized in **Appendix C**. To achieve GNN-to-MLP knowledge distillation, we first pre-train the teacher GNNs with the classification loss \(\mathcal{L}_{\mathrm{label}}=\frac{1}{|\mathcal{V}_{L}|}\sum_{i\in\mathcal{V}_ {L}}\mathcal{H}\big{(}y_{i},\sigma(\mathbf{h}_{i}^{(L)})\big{)}\), where \(\mathcal{H}(\cdot)\) denotes the cross-entropy loss and \(y_{i}\) is the ground-truth label of node \(v_{i}\). Finally, the total objective function to distill the low- and high-frequency knowledge from the teacher GNNs into the student MLPs is defined as follows

\[\mathcal{L}_{\mathrm{total}}=\frac{\lambda}{|\mathcal{V}_{L}|}\sum_{i\in \mathcal{V}_{L}}\mathcal{H}\big{(}y_{i},\sigma(\mathbf{z}_{i}^{(L)})\big{)}\! +\!\big{(}1\!-\!\lambda\big{)}\big{(}\mathcal{L}_{\mathrm{LFD}}\!+\!\mathcal{L }_{\mathrm{HFD}}\big{)}\]

where \(\lambda\) is the weights to balance the influence of the classification loss and two knowledge distillation losses. The time complexity analysis of FF-G2M is available in **Appendix D**.

### Discussion and Comparision

In this subsection, we compare the proposed FF-G2M framework with the commonly used node-to-node distillation (e.g., GLNN) in Fig. 3. While the node-to-node distillation can map neighboring nodes closely in the representation space of MLPs, i.e., preserving low-frequency knowledge, it completely confounds the relative distance between node pairs, i.e., high-frequency knowledge is drowned, leading to a different (incorrect) class-boundary with the teacher GNNs. In terms of the proposed FF-G2M framework, the Low-Frequency Distillation distills (diffuses) the features aggregated from the neighborhood in the teacher GNNs back into their neighborhood of the student MLPs to better utilize the extracted low-frequency knowledge. Besides, High-Frequency Distillation directly distills the neighborhood pairwise differences from teacher GNNs into the student MLPs to better capture the high-frequency knowledge patterns, i.e., the relative positions between pairs of nodes.

## Experiments

**Datasets.** The effectiveness of the FF-G2M framework is evaluated on six public real-world datasets, including Cora [14], Citeseer [11], Pubmed [15], Coauthor-CS, Coauthor-Physics, and Amazon-Photo [12]. For each dataset, following the data splitting settings of [13, 14], we select 20 nodes per class to construct a training set, 500 nodes for validation, and 1000 nodes for testing. A statistical overview

Figure 3: Illustration of the Full-Frequency GNN-to-MLP (FF-G2M) distillation framework, where the dotted red lines denote the predicted class-boundary, the solid black lines denote feature aggregation from the neighborhood, and the dashed black lines denote the distillation of knowledge (neighborhood features and pairwise distances) from teacher GNNs to student MLPs.
of these datasets is placed in **Appendix A**. Besides, we defer the implementation details and hyperparameter settings for each dataset to **Appendix B** and supplementary materials.

**Baselines.** Three basic components in knowledge distillation are (1) teacher model, (2) student model, and (3) distillation loss. As a model-agnostic general framework, FF-G2M can be combined with any teacher GNN architecture. In this paper, we consider three types of teacher GNNs, including GCN [11], GraphSAGE [10], and GAT [21]. As for the student model, we default to using pure MLPs (with the same network architecture as the teacher GNNs) as the student model for a fair comparison. Finally, the focus of this paper is on designing distillation objectives rather than powerful teacher and student models. Therefore, we only take GLNN [13] as an important baseline to compare FF-G2M with the conventional node-to-node distillation approach. The experiments of all baselines and FF-G2M are implemented based on the standard implementation in the DGL library [23] using PyTorch 1.6.0 library on NVIDIA V100 GPU. Each set of experiments is run five times with different random seeds, and the average are reported as metrics.

### Classification Performance Comparison

This paper aims to explore which knowledge patterns _should_ and _how to_ be distilled into student MLPs, rather than designing more powerful teacher GNNs. Therefore, we consider three classical GNNs, including GCN, GraphSAGE, and GAT, as the teacher models and distill their knowledge into MLPs with the same network architecture. The experimental results on six datasets are reported in Table. 1, from which we can make the following observations: (1) In general, more powerful teacher GNNs can lead to student MLPs with better classification performance. However, such improvements are usually very limited and do not work for all datasets and GNN architectures. For example, on the Citeseer dataset, the performance of GLNN drops over the vanilla implementation of teacher GNNs by 0.4% (GraphSAGE) and 0.6% (GAT), respectively. (2) The proposed FF-G2M framework can consistently improve the performance of student MLPs across three GNN architectures on all six datasets. For example, FF-G2M can outperform vanilla teacher GNNs by 2.63% (GCN), 2.58% (GraphSAGE), and 2.55% (GAT) averaged over six datasets, respectively.

### Qualitative and Quantitative Analysis

Extensive qualitative and quantitative experiments are conducted to explore the existence and harmfulness of the information drowning problem and how to solve it by FF-G2M.

**Qualitative Analysis on Visualizations.** We consider GCNs as the teacher model and compare its visualization with that of vanilla MLPs, GLNN, and FF-G2M on the Cora dataset (due to space limitations, more results can be found in **Appendix F**). We select a target node (id 27 for Cora) and analyze its relative position relationship with its neighbors in Fig.4, from which we observe that: (1) The vanilla MLPs map neighboring nodes apart, which indicates that it is not even good at capturing low-frequency information. (2) GLNN fails to capture the relative positions between the target node and its neighbors, i.e., high-frequency information. (3) FF-G2M well preserves the relative positions between nodes while mapping neighboring nodes closely, which suggests that it is good at capturing both low- and high-frequency information. For example, on the Cora dataset, the target node (id 27) is the closest to node (id 1810) and the farthest from node (id 2678) in the visualizations of both teacher GCNs and FF-G2M's student MLPs.
high-frequency knowledge, measured by _KL-divergence_ between the pairwise distances of teacher GNNs and student MLPs, respectively. The detailed mathematical definitions of these two evaluation metrics are available in **Appendix E**. From the experimental results on the Cora dataset reported in Fig. 5, we make four observations: (1) The vanilla MLP does not consider the inductive bias of the graph topology at all and thus fails to capture the low-and high-frequency knowledge in the graph data. (2) GLNN is capable of successfully capturing low-frequency information, i.e., neighborhood smoothing, but is not good at capturing high-frequency knowledge, i.e., difference information between pairs of nodes. (3) The proposed low- and high-frequency distillation has an advantage over GLNN in capturing one type of individual frequency but lags behind in another frequency. (4) The proposed FF-G2M combines both the two distillation and is better at capturing both low- and high-frequency knowledge than GLNN, especially the latter.

### Roles of Low- and High-frequency Knowledge

To evaluate the roles played by low- and high-frequency knowledge in GNN-to-MLP distillation, we consider distillation with only \(\mathcal{L}_{\mathrm{LFD}}\) and \(\mathcal{L}_{\mathrm{HFD}}\), in addition to the full FF-G2M model. The experiments (with GCNs as the teacher model) on six datasets are reported in Table. 2, from which we observe that: (1) The proposed low-frequency distillation \(\mathcal{L}_{\mathrm{LFD}}\) makes fuller use of the graph topology and the low-frequency information from GNNs, and in turn outperforms GLNN that adopts node-to-node distillation on all six datasets. (2) While both low- and high-frequency distillation can work alone to improve the performance of vanilla MLPs, the former plays a _primary_ role and the latter a _secondary_ (auxiliary) role. More importantly, these two distillations are complementary to each other and can further improve performance on top of each other. (3) The FF-G2M (full model) considers both low- and high-frequency distillation and is capable of capturing full-frequency knowledge, and thus can far outperform GLNN on all six datasets.

## Conclusion

In this paper, we factorize the knowledge learned by GNNs into low- and high-frequency components in the spectral and spatial domains and then conduct a comprehensive investigation on their roles played in GNN-to-MLP distillation. Our key finding is existing GNN-to-MLP distillation may suffer from a potential _information drowning_ problem, i.e., the high-frequency knowledge of the pre-trained GNNs may be overwhelmed by the low-frequency knowledge during distillation. Therefore, we propose a novel _Full-Frequency GNN-to-MLP_ (FF-G2M) knowledge distillation framework, which extracts both low- and high-frequency knowledge from GNNs and injects it into MLPs. As a simple but general framework, FF-G2M outperforms other leading methods across various GNN architectures and graph datasets. Limitations still exist; for example, this paper pays little attention to the special designs on teacher GNNs, and designing more expressive teachers to directly capture full-frequency knowledge may be another promising direction.

\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **Cora** & **Citeseer** & **Pubmed** & **Amazon-Photo** & **Coauthor-CS** & **Coauthor-Phy** \\ \hline Vanilla GCN & 82.2 \(\pm\) 0.5 & 71.6 \(\pm\) 0.4 & 79.3 \(\pm\) 0.3 & 91.8 \(\pm\) 0.6 & 89.9 \(\pm\) 0.7 & 91.9 \(\pm\) 1.2 \\ Vanilla MLP & 59.7 \(\pm\) 1.0 & 60.7 \(\pm\) 0.5 & 71.5 \(\pm\) 0.5 & 77.4 \(\pm\) 1.2 & 87.5 \(\pm\) 1.4 & 89.2 \(\pm\) 0.9 \\ GLNN (Zhang et al., 2021) & 82.8 \(\pm\) 0.5 & 72.7 \(\pm\) 0.4 & 80.2 \(\pm\) 0.6 & 91.4 \(\pm\) 1.0 & 92.7 \(\pm\) 1.0 & 93.2 \(\pm\) 0.5 \\ \hline \hline Low-Frequency KD w/ \(\mathcal{L}_{\mathrm{LFD}}\) & 83.4 \(\pm\) 0.9 & 73.7 \(\pm\) 0.6 & 81.0 \(\pm\) 0.5 & 92.1 \(\pm\) 0.8 & 93.2 \(\pm\) 0.8 & 93.7 \(\pm\) 0.8 \\ High-Frequency KD w/ \(\mathcal{L}_{\mathrm{HFD}}\) & 68.5 \(\pm\) 0.8 & 63.2 \(\pm\) 0.7 & 74.4 \(\pm\) 0.4 & 82.5 \(\pm\) 1.3 & 89.3 \(\pm\) 1.7 & 91.0 \(\pm\) 1.6 \\ FF-G2M (full model) & **84.3 \(\pm\) 0.4** & **74.0 \(\pm\) 0.5** & **81.8 \(\pm\) 0.4** & **94.2 \(\pm\) 0.4** & **93.8 \(\pm\) 0.5** & **94.4 \(\pm\) 0.9** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification accuracy \(\pm\) std (%) on six real-world datasets. The best metrics are marked by **bold**.

Figure 4: Representation 2D-Visualizations (by UMAP (McInnes et al., 2018)) of the teacher model and three student models on Cora. Each node is colored by its ground-truth label, and the numbers around the nodes denote the node ids.

Figure 5: (a) Curves of mean cosine similarity (**the higher, the better**) between nodes with their first-order neighbors. (b) Curves of pairwise distance differences (**the lower, the better**) between the teacher GNNs and student MLPs.
## Acknowledgement

This work is supported in part by Ministry of Science and Technology of the People's Republic of China (No. 2021YFA1301603) and National Natural Science Foundation of China (No. U21A20427).
