Graph Neural Networks for Text Classification: A Survey
Text Classification is the most essential and fundamental problem in Natural
Language Processing. While numerous recent text classification models applied
the sequential deep learning technique, graph neural network-based models can
directly deal with complex structured text data and exploit global information.
Many real text classification applications can be naturally cast into a graph,
which captures words, documents, and corpus global features. In this survey, we
bring the coverage of methods up to 2023, including corpus-level and
document-level graph neural networks. We discuss each of these methods in
detail, dealing with the graph construction mechanisms and the graph-based
learning process. As well as the technological survey, we look at issues behind
and future directions addressed in text classification using graph neural
networks. We also cover datasets, evaluation metrics, and experiment design and
present a summary of published performance on the publicly available
benchmarks. Note that we present a comprehensive comparison between different
techniques and identify the pros and cons of various evaluation metrics in this
survey.
Kunze Wang, Yihao Ding, Soyeon Caren Han
# Graph Neural Networks for Text Classification: A Survey

###### Abstract.

Text Classification is the most essential and fundamental problem in Natural Language Processing. While numerous recent text classification models applied the sequential deep learning technique, graph neural network-based models can directly deal with complex structured text data and exploit global information. Many real text classification applications can be naturally cast into a graph, which captures words, documents, and corpus global features. In this survey, we bring the coverage of methods up to 2023, including corpus-level and document-level graph neural networks. We discuss each of these methods in detail, dealing with the graph construction mechanisms and the graph-based learning process. As well as the technological survey, we look at issues behind and future directions addressed in text classification using graph neural networks. We also cover datasets, evaluation metrics, and experiment design and present a summary of published performance on the publicly available benchmarks. Note that we present a comprehensive comparison between different techniques and identify the pros and cons of various evaluation metrics in this survey.

Graph Neural Networks, Text Classification. Representation Learning +
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †:
Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †: journal: ACM

+
Footnote †:

Footnote †: journal: ACM

+
Footnote †:

Footnote †:

Footnote †:

Footnote †:

+

[MISSING_PAGE_POST]
document-level graphs and each of them represents a document. The corpus-level graph can capture the global structural information of the entire corpus, while the document-level graph can capture the word-to-word relationships within a document explicitly. Both ways of applying graph neural networks to text classification achieve good performance.

This paper mainly focuses on GNN-based text classification techniques, datasets, and their performance. The graph construction approaches for both corpus-level and document-level graphs are addressed in detail. Papers on the following aspects will be reviewed:

* GNNs-based text classification approaches. Papers that design GNN-based frameworks to enhance the feature representation or directly apply GNNs to conduct sequence text classification tasks will be summarized, described and discussed. GNNs applied for token-level classification (Natural Language Understanding) tasks, including NER, slot filling, etc, will not be discussed in this work.
* Text classification benchmark datasets and their performance applied by GNN-based models. The text classification datasets with commonly used metrics used by GNNs-based text classification models will be summarized and categorized based on task types, together with the model performance on these datasets.

### Related Surveys and Our Contribution

Before 2019, the text classification survey papers [2; 35; 116; 129; 46] have focused on covering traditional machine learning-based text classification models. Recently, with the rapid development of deep learning techniques, [55; 82; 147; 149] review the various deep learning based approaches. In addition, some papers not only review the SoTA model architectures, but summarize the overall workflow [10; 39; 42; 49; 83] or specific techniques for text classification including word embedding [102], feature selection [22; 96; 103], term weighting [3; 91] and etc. Meanwhile, some growing potential text classification architectures are surveyed, such as CNNs [132], attention mechanisms [78]. Since the powerful ability to represent non-Euclidean relation, GNNs have been used in multiple practical fields and reviewed e.g. financial application [117], traffic prediction [72], bio-informatics [142], power system [60], recommendation system [27; 59; 131]. Moreover, [14; 126; 139; 8; 144] comprehensively review the general algorithms and applications of GNNs, as well as certain surveys mainly focus on specific perspectives including graph construction [105; 112], graph representation [34], training [128], pooling [65] and more. However, only [55; 82] briefly introduce certain SoTA GNN-based text classification models. A recent short review paper [75] reviews several SoTA models without providing a comprehensive overview in this area. The contribution of this survey includes:

* This is the first survey focused only on graph neural networks for text classification with a comprehensive description and critical discussion on more than twenty GNN text classification models.
* We categorize the existing GNN text classification models into two main categories with multiple sub-categories, and the tree structure of all the models shows in Figure 1.
* We compare these models in terms of graph construction, node embedding initialization, and graph learning methods. And We also compare the performance of these models on the benchmark datasets and discuss the key findings.
* We discuss the existing challenges and some potential future work for GNN text classification models.
### Text Classification Tasks

Text classification involves assigning a pre-defined label to a given text sequence. The process typically involves encoding pre-processed raw text into numerical representations and using classifiers to predict the corresponding categories. Typical sub-tasks include sentiment analysis, topic labelling, news categorization, and hate speech detection. Certain frameworks can be extended to advanced applications such as information retrieval, summarising, question answering, and natural language inference. This paper focuses specifically on GNN-based models used for typical text classification.

* **Sentiment Analysis** is a task that aims to identify the emotional states and subjective opinions expressed in the input text, such as reviews, micro-blogs, etc. This can be achieved through binary or multi-class classification. Effective sentiment analysis can aid in making informed business decisions based on user feedback.
* **Topic Classification** is a supervised deep learning task to automatically understand the text content and classified into multiple domain-specific categories, typically more than two. The data sources may gather from different domains, including Wikipedia pages, newspapers, scientific papers, etc.
* **Junk Information Detection** involves detecting inappropriate social media content. Social media providers commonly use approaches like hate speech, abusive language, advertising or spam detection to remove such content efficiently.

### Text Classification Development

Many traditional machine learning methods and deep learning models are selected as the baselines for comparing with the GNN-based text classifiers. We mainly summarized those baselines into three types:

_Traditional Machine Learning_: In earlier years, traditional methods such as Support Vector Machines (SVM) (Krizhevsky et al., 2014) and Logistic Regression (Krizhevsky et al., 2014) utilized sparse representations like Bag of Words (BoW) and TF-IDF. However, recent advancements (Krizhevsky et al., 2014; Krizhevsky et al., 2014; Krizhevsky et al., 2014) have focused on dense representations, such as Word2vec, GloVe, and Fasttext, to mitigate the limitations of sparse representations. These dense representations are also used as inputs for sophisticated methods, such as Deep Averaging Networks (DAN) (Krizhevsky et al., 2014) and Paragraph Vector (Doc2Vec) (Krizhevsky et al., 2014), to achieve new state-of-the-art results.

_Sequential Models_: RNNs and CNNs have been utilized to capture local-level semantic and syntactic information of consecutive words from input text bodies. The upgraded models, such as LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Hochreiter and Schmidhuber, 1997), have been proposed to address the vanishing or exploding gradient problems caused by vanilla RNN. CNN-based structures have been applied to capture N-gram features by using one or more convolution and pooling layers, such as Dynamic CNN (Krizhevsky et al., 2014) and TextCNN (Krizhevsky et al., 2014). However, these models can only capture local dependencies of consecutive words. To capture longer-term or non-Euclidean relations, improved RNN structures, such as Tree-LSTM (Hochreiter and Schmidhuber, 1997) and MT-LSTM (Krizhevsky et al., 2014), and global semantic information, like TopicRNN (Krizhevsky et al., 2014), have been proposed. Additionally, graph (Krizhevsky et al., 2014) and tree structure (Krizhevsky et al., 2014) enhanced CNNs have been proposed to learn more about global and long-term dependencies.

_Attentions and Transformers_: attention mechanisms (Chen et al., 2016) have been widely adopted to capture long-range dependencies, such as hierarchical attention networks (Chen et al., 2016) and attention-based hybrid models (Krizhevsky et al., 2014). Self-attention-based transformer models have achieved state-of-the-art performance on many text classification benchmarks via pre-training on some tasks to generate strong contextual word representations. However, these models only focus on learning the
relation between input text bodies and ignore the global and corpus level information. Researchers have proposed combining the benefits of attention mechanisms and Graph Neural Networks (GNNs) to learn both the relation between input text bodies and the global and corpus level information, such as VGCN-BERT (Wang et al., 2019) and BERTGCN (Wang et al., 2019).

### Outline

The outline of this survey is as follows:

* Section 1 presents the research questions and provides an overview of applying Graph Neural Networks to text classification tasks, along with the scope and organization of this survey.
* Section 2 provides background information on text classification and graph neural networks and introduces the key concepts of applying GNNs to text classification from a designer's perspective.
* Section 3 and Section 4 discuss previous work on Corpus-level Graph Neural Networks and Document-level Graph Neural Networks, respectively, and provide a comparative analysis of the strengths and weaknesses of these two approaches.
* Section 5 introduces the commonly used datasets and evaluation metrics in GNN for text classification.
* Section 6 reports the performance of various GNN models on a range of benchmark datasets for text classification and discusses the key findings.
* The challenges for the existing methods and some potential future works are discussed in Section 7.
* In Section 8, we present the conclusions of our survey on GNN for text classification and discuss potential directions for future work.

## 2. Backgrounds of Gnn

### Definition of Graph

A graph in this paper is represented as \(G=(V,E)\), where \(V\) and \(E\) represent a set of nodes (vertices) and edges of \(G\), respectively. A single node in the node set is represented \(v_{i}\in V\), as well as \(e_{ij}=(v_{i},v_{j})\in E\) donates an edge between node \(v_{i}\) and \(v_{j}\). The adjacent matrix of graph \(G\) is represented as \(A\), where \(A\in\mathbb{R}^{n\times n}\) and \(n\) is the number of nodes in graph \(G\). If \(e_{ij}\in E\), \(A_{ij}=1\), otherwise \(A_{ij}=0\). In addition, we use \(\mathbf{X}\) and \(\mathbf{E}\) to represent the nodes and edges representations in graph \(G\), where \(\mathbf{X}\in\mathbb{R}^{n\times m}\) and \(\mathbf{E}\in\mathbb{R}^{n\times c}\). \(\mathbf{x}_{i}\in\mathbb{R}^{m}\) represents the \(m\)-dimensional vector of node \(v_{i}\) and \(\mathbf{e}_{ij}\in\mathbb{R}^{c}\) represents the \(c\)-dimensional vector of edge \(e_{ij}\) (most of the recent studies set \(c=1\) to represent a weighting scalar). \(\mathbf{A}\) donates the edge feature weighted adjacent matrix.

### Traditional Graph-based Algorithms

Before GNNs were broadly used for representing irregular relations, traditional graph-based algorithms have been applied to model the non-Euclidean structures in text classification e.g. Random Walk (197; 146), Graph Matching (101; 104), Graph Clustering (79) which has been well summarized in (124). There are three common limitations of those traditional graph-based algorithms. Firstly, most of those algorithms mainly focus on capturing graph-level structure information without considering the significance of node and edge features. For example, Random Walk based approaches (107; 146) mainly focus on using distance or angle between node vectors to calculate transition probability while ignoring the information represented by node vectors. Secondly, since the traditional graph-based
algorithms are only suitable for specific tasks, there is no unified learning framework for addressing various practical tasks. For example, [44] proposes a graph clustering method that requires a domain knowledge-based ontology graph. Lastly, the traditional graph based methods are comparative time inefficient like the Graph Edit Distance-based graph matching methods have exponential time complexity [104].

### Foundations of GNN

To tackle the limitation of traditional graph-based algorithms and better represent non-Euclidean relations in practical applications, Graph Neural Networks are proposed by [100]. GNNs have a unified graph-based framework and simultaneously model the graph structure, node, and edge representations. This section will provide the general mathematical definitions of Graph Neural Networks. The general forward process of GNN can be summarised as follows:

\[\mathbf{H}^{(I)}=\mathcal{F}(\mathbf{A},\mathbf{H}^{(l-1)}) \tag{1}\]

where \(\mathbf{A}\in\mathbb{R}^{n\times n}\) represents the weighted adjacent matrix and \(\mathbf{H}^{(I)}\in\mathbb{R}^{n\times d}\) is the updated node representations at the \(l\)-th GNN layers by feeding \(l-1\)-th layer node features \(\mathbf{H}^{(l-1)}\in\mathbb{R}^{n\times k}\) ( \(k\) is the dimensions of previous layers node representations ) into pre-defined graph filters \(\mathcal{F}\).

Figure 1: Categorizing the graph neural network text classification models.
The most commonly used graph filtering method is defined as follows:

\[\mathbf{H}^{(I)}=\phi(\tilde{\mathbf{A}}\mathbf{H}^{(I-1)}\mathbf{W}) \tag{2}\]

where \(\tilde{\mathbf{A}}=\mathbf{D}^{-\frac{1}{2}}\mathbf{A}\mathbf{D}^{-\frac{1}{2}}\) is the normalized symmetric adjacency matrix. \(\mathbf{A}\in\mathbb{R}^{n\times n}\) is the adjacent matrix of graph \(G\) and \(\mathbf{D}\) is the degree matrix of \(\mathbf{A}\), where \(D_{ii}=\Sigma_{j}A_{ij}\). \(\mathbf{W}\in\mathbb{R}^{k\times d}\) is the weight matrix and \(\phi\) is the activation function. If we design a two layers of GNNs based on the above filter could get a vanilla Graph Convolutional Network (GCN) (Kunze et al., 2017) framework for text classification:

\[\mathbf{Y}=softmax(\tilde{\mathbf{A}}(ReLU(\tilde{\mathbf{A}}\mathbf{H}\mathbf{W}^{(0)}))\mathbf{W}^{ (1)}) \tag{3}\]

where \(\mathbf{W}^{0}\) and \(\mathbf{W}^{1}\) represent different weight matrix for different GCN layers and \(\mathbf{H}\) is the input node features. \(ReLU\) function is used for non-linearization and \(softmax\) is used to generated predicted categories \(\mathbf{Y}\).

### GNN for Text Classification

In this paper, we mainly discuss how GNNs are applied in Text Classification tasks. Before we present the specific applications in this area, we first introduce the key concepts of applying GNNs to text classification from a designer's view. We suppose for addressing a text classification task need to design a graph \(G=(V,E)\). The general procedures include _Graph Construction_, _Initial Node Representation_, _Edge Representations_, _Training Setup_.

#### 2.4.1. Graph Construction

Some applications have explicit graph structures including constituency or dependency graphs (Kunze et al., 2017), knowledge graphs (Kunze et al., 2017; Kunze et al., 2018), social networks (Kunze et al., 2017) without constructing graph structure and defining corresponding nodes and edges. However, for text classification, the most common graph structures are implicit, which means we need to define a new graph structure for a specific task such as designing a word-word or word-document co-occurrence graph. In addition, for text classification tasks, the graph structure can be generally classified into two types:

* _Corpus-level/Document-level_. Corpus-level graphs intend to construct the graph to represent the whole corpus such as (Kunze et al., 2017; Kunze et al., 2018; Kunze et al., 2019), while the document-level graphs focus on representing the non-Euclidean relations existing in a single text body like (Kunze et al., 2017; Kunze et al., 2019; Kunze et al., 2019). Supposing a specific corpus \(\mathcal{C}\) contains a set of documents (text bodies) \(\mathcal{C}=\{D_{1},D_{2},..,D_{j}\}\) and each \(D_{i}\) contains a set of tokens \(D_{i}=\{t_{i_{1}},t_{i_{2}},...,t_{i_{k}}\}\). The vocabulary of \(\mathcal{C}\) can be represented as \(\mathcal{D}=\{t_{1},t_{2},...,t_{l}\}\), where \(l\) is the length of \(\mathcal{D}\). For the most commonly adopted corpus-level graph \(G_{corpus}=(V_{corpus},E_{corpus})\), a node \(v_{i}\) in \(V_{corpus}\) follows \(v_{i}\in\mathcal{C}\cup\mathcal{D}\) and the edge \(e_{ij}\in E_{corpus}\) is one kind of relations between \(v_{i}\) and \(v_{j}\). Regarding the document level graph \(G_{doc_{i}}=(V_{doc_{i}},E_{doc_{i}})\), a node \(v_{i_{j}}\) in \(V_{doc_{i}}\) follows \(v_{i_{j}}\in D_{i}\).

After designing the graph-scale for the specific tasks, specifying the graph types is also important to determine the nodes and their relations. For text classification tasks, the commonly used graph construction ways can be summarized into:

* _Homogeneous/Heterogeneous Graphs_: homogeneous graphs have the same node and edge type while heterogeneous graphs have various node and edge types. For a graph \(G=(V,E)\), we use \(\mathbf{N}^{\mathbf{\alpha}}\) and \(\mathbf{N}^{\mathbf{e}}\) to represent the number of types of \(V\) and \(E\). If \(\mathbf{N}^{\mathbf{\alpha}}=\mathbf{N}^{\mathbf{e}}=1\), \(G\) is a homogeneous graph. If \(\mathbf{N}^{\mathbf{\alpha}}>1\) or \(\mathbf{N}^{\mathbf{e}}>1\), \(G\) is a heterogeous graph.
* _Static/Dynamic Graphs_: Static graphs aim to use the constructed graph structure by various external or internal information to leverage to enhance the initial node representation such as dependency or constituency graph
[110], co-occurrence between word nodes [143], TF-IDF between word and document nodes [53, 123, 133] and so on. However, compared with the static graph, the dynamic graph initial representations or graph topology are changing during training without certain domain knowledge and human efforts. The feature representations or graph structure can jointly learn with downstream tasks to be optimised together. For example, [120] proposed a novel topic-award GNN text classification model with dynamically updated edges between topic nodes with others (e.g. document, word). Piao et al. [95] also designed a dynamic edge based graph to update the contextual dependencies between nodes. Additionally, [16] propose a dynamic GNN model to jointly update the edge and node representation simultaneously. We provide more details about above mentioned models in Section 3 and Section 4.

Another widely used pair of graph categories are _directed_ or _undirected_ graphs based on whether the directions of edges are bi-directional or not. For text classification, most of the GNN designs are following the unidirectional way. In addition, those graph type pairs are not parallel which means those types can be combined.

#### 2.4.2. Initial Node Representation.

Based on the pre-defined graph structure and specified graph type, selecting the appropriate initial node representations is the key procedure to ensure the proposed graph structure can effectively learn node. According to the node entity type, the existing node representation approaches for text classification can be generally summarised into:

* _Word-level Representation_: non-context word embedding methods such as GloVe [93], Word2vec [81], FastText [13] are widely adopted by many GNN-based text classification framework to numerically represent the node features. However, those embedding methods are restricted to capturing only syntactic similarity and fail to represent the complex semantic relationships between words, as well as, they cannot capture the meaning of out-of-vocabulary (OOV) words, and their representations are fixed. Therefore, there are some recent studies selecting ELMo [94], BERT [23], GPT [97] to get contextual word-level node representation. Notably, even if one-hot encoding is the simplest word representation method, there are many GNN-based text classifiers using one-hot encoding and achieving state-of-the-art performance. Few frameworks use randomly initialised vectors to represent the word-level node features.
* _Document-level Representation_: similar to other NLP applications, document level representations are normally acquired by aggregating the word level representation via some deep learning frameworks. For example, some researchers select by extracting the last-hidden state of LSTM or using the [CLS] token from BERT to numerically represent the input text body. Furthermore, it is also a commonly used document-level node representation way to use TF-IDF based document vectors.

Most GNN based text classification frameworks will compare the performance between different node representation methods to conduct quantitative analysis, as well as provide reasonable justifications for demonstrating the effectiveness of the selected initial node representation based on defined graph structure.

#### 2.4.3. Edge Features.

Well-defined edge features can effectively improve the graph representation learning efficiency and performance to exploit more explicit and implicit relations between nodes. Based on the predefined graph types, the edge feature types can be divided into _structural features_ and _non-structural features_. The structural edge features are acquired from explicit relations between nodes such as dependency or constituency relation between words, word-word adjacency
relations, etc. Those relations between nodes are explicitly defined and are also widely employed in other NLP applications. However, more commonly used edge features are non-structural features which implicitly existed between the nodes and specifically applied to specific graph-based frameworks. The typically non-structural edge features are firstly defined by (Wang et al., 2017) for GNNs-based text classification tasks including:

* **PMI** measures the co-occurrence between two words in a sliding window \(W\) and is calculated as: \[\text{PMI}(i,j) =log\frac{p(i,j)}{p(i)p(j)}\] (4) \[p(i,j) =\frac{\#W(i,j)}{\#W}\] (5) \[p(i) =\frac{\#W(i)}{\#W}\] (6) where \(\#W\) is the number of windows in total, and \(\#W(i)\), \(\#W(i,j)\) shows the number of windows containing word \(i\) and both word \(i\) and \(j\) respectively.
* **TF-IDF** is the broadly used weight of the edges between document-level nodes and word-level nodes.

Except for those two widely used implicit edge features, some specific edge weighting methods are proposed to meet the demands of particular graph structures for exploiting more information of input text bodies.

#### 2.4.4. Training Setup.

After specifying the graph structure and types, the graph representation learning tasks and training settings also need to be determined to decide how to optimise the designed GNNs. Generally, the graph representation learning tasks can be categorised into three levels including _Node-level_, _Graph-level_ and _Edge-level_. Node-level and Graph-level tasks involve node or graph classification, clustering, regression, etc, while Edge-level tasks include link prediction or edge classification for predicting the relation existence between two nodes or the corresponding edge categories.

Similar to other deep learning model training settings, GNNs also can be divided into _supervised_, _semi-supervised_ and _unsupervised training settings_. Supervised training provides labelled training data, while unsupervised training utilises unlabeled data to train the GNNs. However, compared with supervised or unsupervised learning, semi-supervised learning methods are broadly used by GNNs designed for text classification applications which could be classified into two types:

* _Inductive Learning_ adjusts the weights of proposed GNNs based on a labelled training set for learning the overall statistics to induce the general trained model for following processing. The unlabeled set can be fed into the trained GNNs to compute the expected outputs.
* _Transductive Learning_ intends to exploit labelled and unlabeled sets simultaneously for leveraging the relations between different samples to improve the overall performance.

## 3. Corpus-Level GNN for Text Classification

We define a corpus-level Graph Neural Network as "constructing a graph to represent the whole corpus", thus, only one or several graphs will be built for the given corpus. We categorize Corpus-level GNN into four subcategories based on the types of nodes shown in the graph.
### Document and Word Nodes as a Graph

Most corpus-level graphs include word nodes and document nodes and there are word-document edges and word-word edges. By applying \(K\)(normally \(K\)=2 or 3) layer GNN, word nodes will serve as a bridge to propagate the information from one document node to another.

#### 3.1.1. PMI and TF-IDF as graph edges: TextGCN, SGC, S\({}^{2}\)GC, NMGC, TG-Transformer, BertGCN.

**TextGCN[133]** Yao et al. [133] builds a corpus-level graph with training document nodes, test document nodes and word nodes. Before constructing the graph, a common preprocessing method[47] has been applied and words shown fewer than 5 times or in NLTK[11] stopwords list have been removed. The edge value between the document node and the word node is TF-IDF and that between the word nodes is PMI. The adjacency matrix of this graph shows as follows.

\[A_{ij}=\begin{cases}\text{PMI}(i,j)&i,j\text{ are words},\text{PMI}(i,j)>0\\ \text{TF-IDF}_{i,j}&i\text{ is document},j\text{ is word}\\ 1&i=j\\ 0&\text{otherwise}\end{cases} \tag{7}\]

A two-layer GCN is applied to the graph, and the dimension of the second layer output equals to the number of classes in the dataset. Formally, the forward propagation of TextGCN shows as:

\[Z=\text{softmax}(\tilde{A}(\text{ReLU}(\tilde{A}X\mathbf{W}^{(0)}))\mathbf{W}^{(1)}) \tag{8}\]

where \(\tilde{A}\) is the normalized adjacency of \(A\) and \(X\) is one-hot embedding. \(W_{0}\) and \(W_{1}\) are learnable parameters of the model. The representation on training documents is used to calculate the loss and that on test documents is for prediction.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Notations** & **Descriptions** \\ \hline \(G\) & A graph. \\ \hline \(V\) & The set of nodes in a graph. \\ \hline \(E\) & The set of edges in a graph. \\ \hline \(e_{ij}\) & An edge between node \(i\) and node \(j\). \\ \hline \(N_{i}\) & The neighbors of a node \(i\). \\ \hline \(\mathbf{A}\) & The graph adjacency matrix. \\ \hline \(\tilde{\mathbf{A}}\) & The normalized matrix \(\mathbf{A}\). \\ \hline \(\tilde{\mathbf{A}}^{k},k\in Z\) & The \(k^{th}\) power of \(\tilde{\mathbf{A}}\). \\ \hline \([\mathbf{A}||B]\) & The concatenation of \(A\) and \(B\). \\ \hline \(D\) & The degree matrix of \(\mathbf{A}\). \(D_{ii}=\Sigma_{j=1}^{n}A_{ij}\). \\ \hline \(W^{(l)}\) & The weight matrix of layer \(l\). \\ \hline \(H\in R^{n\times d}\) & The feature matrix of a graph. \\ \hline \(H^{(l)}\in R^{n\times d}\) & The feature matrix of a graph at layer \(l\). \\ \hline \(\tilde{\mathbf{h}}_{i}\in R^{n}\) & The feature vector of the node \(i\) \\ \hline \(\tilde{\mathbf{h}}_{i}^{(l)}\in R^{n}\) & The feature vector of the node \(i\) at layer \(l\). \\ \hline \(Z\in R^{n\times d}\) & The output feature matrix of a graph. \\ \hline \(z_{i}\in R^{n}\) & The output feature vector of the node \(i\) \\ \hline \hline \end{tabular}
\end{table}
Table 1. Commonly used notations in Graph Neural Networks
TextGCN is the first work that treats a text classification task as a node classification problem by constructing a corpus-level graph and has inspired many following works.

Based on TextGCN, several works follow the same graph construction method and node initialization but apply different graph propagation models.

**SGC[123]** To make GCN efficient, SGC (Simple Graph Convolution) removes the nonlinear activation function in GCN layers, therefore, the K-layer propagation of SGC shows as:

\[Z=\text{softmax}(\tilde{\mathbf{A}}...(\tilde{\mathbf{A}}(\tilde{\mathbf{A}}\mathbf{X}\mathbf{W}^{ (0)})\mathbf{W}^{(1)})...\mathbf{W}^{(K)}) \tag{9}\]

which can be reparameterized into

\[Z=\text{softmax}(\tilde{\mathbf{A}}^{K}\mathbf{X}\mathbf{W}) \tag{10}\]

and \(K\) is 2 when applied to text classification tasks. With a smaller number of parameters and only one feedforward layer, SGC saves computation time and resources while improving performance.

**S\({}^{2}\)GC[148]** To solve the oversmoothing issues in GCN, Zhu and Koniusz [148] propose Simple Spectral Graph Convolution(S\({}^{2}\)GC) which includes self-loops using Markov Diffusion Kernel. The output of S\({}^{2}\)GC is calculated as:

\[Z=\text{softmax}(\frac{1}{K}\Sigma_{k=0}^{K}\tilde{\mathbf{A}}^{k}\mathbf{X}\mathbf{W}) \tag{11}\]

and can be generalized into:

\[Z=\text{softmax}(\frac{1}{K}\Sigma_{k=0}^{K}((1-\alpha)\tilde{\mathbf{A}}^{k}\mathbf{ X}+\alpha\mathbf{X})\mathbf{W}) \tag{12}\]

Similarly, \(K=2\) on text classification tasks and \(\alpha\) denotes the trade-off between self-information of the node and consecutive neighbourhood information. S\({}^{2}\)GC can also be viewed as introducing skip connections into GCN.

**NMGC[53]** Other than using the sum of each GCN layer in S\({}^{2}\)GC, NMGC applies min pooling using the Multi-hop neighbour Information Fusion (MIF) operator to address oversmoothing problems. A MIF function is defined as:

\[\text{MIF}(K)=\min(\tilde{\mathbf{A}}\mathbf{X}\mathbf{W},\tilde{\mathbf{A}}^{2}\mathbf{X}\mathbf{W},...,\tilde{\mathbf{A}}^{K}\mathbf{X}\mathbf{W}) \tag{13}\]

NMGC-K firstly applies a MIF(\(K\)) layer then a GCN layer and K is 2 or 3. For example, when \(K=3\), the output is:

\[Z=\text{softmax}(\tilde{\mathbf{A}}(\text{ReLU min}(\tilde{\mathbf{A}}\mathbf{X}\mathbf{W}^{ (0)},\tilde{\mathbf{A}}^{2}\mathbf{X}\mathbf{W}^{(0)},\tilde{\mathbf{A}}^{3}\mathbf{X}\mathbf{W}^{(0) }))\mathbf{W}^{(1)}) \tag{14}\]

NMGC can also be treated as a skip-connection in Graph Neural Networks which makes the shallow layer of GNN contribute to the final representation directly.

**TG-Transformer[137]** TextGCN treats the document nodes and word nodes as the same type of nodes during propagation, and to introduce heterogeneity into the TextGCN graph, TG-Transformer (Text Graph Transformer) adopts two sets of weights for document nodes and word nodes respectively. To cope with a large corpus graph, subgraphs are sampled from the TextGCN graph using PageRank algorithm[88]. The input embedding of is the sum of three types of embedding: pretrained GloVe embedding, node type embedding, and Weisfeiler-Lehman structural encoding[85]. During propagation, self-attention[114] with graph residual[138] is applied.

**BertGCN[63]** To combine BERT[45] and TextGCN, BertGCN enhances TextGCN by replacing the document node initialization with the BERT [CLS] output of each epoch and replacing the word input vector with zeros. BertGCN trains BERT and TextGCN jointly by interpolating the output of TextGCN and BERT:

\[Z=\lambda Z_{GCN}+(1-\lambda)Z_{BERT} \tag{15}\]
where \(\lambda\) is the trade-off factor. To optimize the memory during training, a memory bank is used to track the document input and a smaller learning rate is set to BERT module to remain the consistency of the memory bank. BertGCN shows that with the help of TextGCN, BERT can achieve better performance.

#### 3.1.2. Multi-Graphs/Multi-Dimensional Edges: TensorGCN, ME-GCN.

**TensorGCN**[68] Instead of constructing a single corpus-level graph, TensorGCN builds three independent graphs: Semantic-based graph, Syntactic-based graph, and Sequential-based graph to incorporate semantic, syntactic and sequential information respectively and combines them into a tensor graph.

Three graphs share the same set of TF-IDF values for the word-document edge but different values for word-word edges. Semantic-based graph extracts the semantic features from a trained Long short-term memory(LSTM)[36] model and connects the words sharing high similarity. Syntactic-based graph uses Stanford CoreNLP parser[76] and constructs edges between words when they have a larger probability of having dependency relation. For Sequential-based graph, PMI value is applied as TextGCN does.

The propagation includes intra-graph propagation and inter-graph propagation. The model first applies the GCN layer on three graphs separately as intra-graph propagation. Then the same nodes on three graphs are treated as a virtual graph and another GCN layer is applied as inter-graph propagation.

**ME-GCN**[118] To fully utilize the corpus information and analyze rich relational information of the graph, ME-GCN (Multi-dimensional Edge-Embedded GCN) builds a graph with multi-dimensional word-word, word-document and document-document edges. Word2vec and Doc2vec embedding is firstly trained on the given corpus and the similarity of each dimension of trained embedding is used to construct the multi-dimensional edges. The trained embedding also serves as the input embedding of the graph nodes. During propagation, GCN is firstly applied on each dimension and representations on different dimensions are either concatenated or fed into a pooling method to get the final representations of each layer.

#### 3.1.3. Making TextGCN Inductive: HeteGCN, InducT-GCN, T-VGAE.

**HeteGCN**[98] HeteGCN (Heterogeneous GCN) optimizes the TextGCN by decomposing the TextGCN undirected graph into several directed subgraphs. Several subgraphs from TextGCN graph are combined sequentially as different layers: feature graph (word-word graph), feature-document graph (word-document graph), and document-feature graph (document-word graph). Different combinations were tested and the best model is shown as:

\[Z=\text{softmax}(\mathbf{A}_{w-d}(\text{ReLU}(\mathbf{A}_{w-w}\mathbf{X}_{w}\mathbf{W}^{(0)} ))\mathbf{W}^{(1)}) \tag{16}\]

where \(\mathbf{A}_{w-w}\) and \(\mathbf{A}_{w-d}\) show the adjacency matrix for the word-word subgraph and word-document subgraph. Since the input of HeteGCN is the word node embeddings without using document nodes, it can also work in an inductive way while the previous corpus-level graph text classification models are all transductive models.

**InducT-GCN**[119] InducT-GCN (InducTive Text GCN) aims to extend the transductive TextGCN into an inductive model. Instead of using the whole corpus for building the graph, InducT-GCN builds a training corpus graph and makes the input embedding of the document as the TF-IDF vectors, which aligns with the one-hot word embeddings. The weights are learned following TextGCN but InducT-GCN builds virtual subgraphs for prediction on new test documents.

**T-VGAE**[127] T-VGAE (Topic Variational Graph Auto-Encoder) applies Variational Graph Auto-Encoder on the latent topic of each document to make the model inductive. A vocabulary graph \(A_{\varnothing}\) which connects the words using PMI values is constructed while each document is represented using the TF-IDF vector. All the document vectors are
stacked into a matrix which can also be treated as a bipartite graph \(A_{d}\). Two graph auto-encoder models are applied on \(A_{p}\) and \(A_{d}\) respectively. The overall workflow shows as:

\[Z_{v}=\text{Encoder}_{GCN}(\mathbf{A}_{v},\mathbf{X}_{v}) \tag{18}\] \[Z_{d}=\text{Encoder}_{UDMP}(\mathbf{A}_{d},Z_{v})\] (19) \[\mathbf{A}_{v}^{*}=\text{Decoder}(Z_{v})\] (20) \[\mathbf{A}_{d}^{*}=\text{Decoder}(\mathbf{Z}_{d},\mathbf{Z}_{v}) \tag{17}\]

where \(X^{p}\) is an Identity Matrix. The \(\text{Encoder}_{GCN}\) and the decoders are applied following \(VGAE\)(Kunze et al., 2017) while \(\text{Encoder}_{UDMP}\) is an unidirectional message passing variant of \(\text{Encoder}_{GCN}\). The training objective is minimising the reconstruction error and \(Z_{d}\) is used for the classification task.

### Document Nodes as a Graph

To show the global structure of the corpus directly, some models only adopt document nodes in the non-heterogeneous graph.

**knn-GCN(Chen et al., 2017)** knn-GCN constructs a k-nearest-neighbours graph by connecting the documents with their \(K\) nearest neighbours using Euclidean distances of the embedding of each document. The embedding is generated in an unsupervised way: either using the mean of pretrained GloVe word vectors or applying LDA(Kunze et al., 2017). Both GCN and Attention-based GNN(Kunze et al., 2017) are used as the graph model.

**TextGTL(Wang et al., 2017)** Similar to TensorFlowGCN, TextGTL (Text-oriented Graph-based Transductive Learning) constructs three different document graphs: Semantics Text Graph, Syntax Text Graph, and Context Text Graph while all the graphs are non-heterogeneous. Semantics Text Graph uses Generalized Canonical Correlation Analysis(Chen et al., 2017) and trains a classifier to determine the edge values between two document nodes. Syntax Text Graph uses the Stanford CoreNLP dependency parser(Sutskever et al., 2017) to construct units and also trains a classifier. Context Text Graph defines the edge values by summing up the PMI values of the overlapping words in two documents. Two GCN layers are applied and the output of each graph is mixed as the output of this layer and input for the next layer for all three graphs:

\[\mathbf{H}^{(1)}=\sigma(\mathbf{A}\mathbf{H}^{(0)}\mathbf{W}^{(0)}) \tag{22}\] \[\mathbf{H}^{(2)}=\sigma(\mathbf{A}[\mathbf{H}^{(1)}_{sem}||\mathbf{H}^{(1)}_{syn }||\mathbf{H}^{(1)}_{seq}]\mathbf{W}^{(1)})\] (23) \[Z=\text{Pooling}_{mean}(\mathbf{H}^{(2)}_{sem},\mathbf{H}^{(2)}_{syn}, \mathbf{H}^{(2)}_{seq}) \tag{21}\]

where \(H^{(0)}\) is the TF-IDF vector of the documents. Data augmentation with super nodes is also applied in TextGTL to strengthen the information in graph models.

### Word Nodes as a Graph

By neglecting the document nodes in the graph, a graph with only word nodes shows good performance in deriving the graph-based embedding and is used for downstream tasks. Since no document nodes are included, this method can be easily adapted as an inductive learning model.

**Vgcn-Bert(Sutskever et al., 2017)** VGCN-BERT enhances the input embedding of BERT by concatenating it with the graph embedding. It first constructs a vocabulary graph and uses PMI as the edge value. A variant of the GCN layer called
VGCN(Vocabulary GCN) is applied to derive the graph word embedding:

\[\mathbf{X}_{Graph}=\text{ReLU}(\mathbf{X}_{BERT}\mathbf{A}\mathbf{W}^{(0)})\mathbf{W}^{(1)} \tag{24}\]

where BERT embedding is used as the input. The graph word embeddings are concatenated with BERT embedding and fed into the BERT as extra information.

### Extra Topic Nodes in the Graph

Topic information of each document can also provide extra information in corpus-level graph neural networks. Several models also include topic nodes in the graph.

#### 3.4.1. Single Layer Topic nodes: HGAT, STGCN

**HGAT**[(64)] HGAT (Heterogeneous GAT) applies LDA[(12)] to extract topic information for each document, top \(P\) topics with the largest probabilities are selected as connected with the document. Instead of using the words directly, to utilize the external knowledge HGAT applies the entity linking tool TAGME1 to identify the entities in the document and connects them. The semantic similarity between entities using pretrained Word2vec with threshold is used to define the connectedness between entity nodes. Since the graph is a heterogeneous graph, a HIN (heterogeneous information network) model is implemented which propagates solely on each sub-graphs depending on the type of node. An HGAT model is applied by considering type-level attention and node-level attention. For a given node, the type-level attention learns the weights of different types of neighbouring nodes while node-level attention captures the importance of different neighbouring nodes when ignoring the type. By using the dual attention mechanism, HGAT can capture the information of type and node at the same time.

Footnote 1: [https://sobigdata.dscience.org/group/tagme/](https://sobigdata.dscience.org/group/tagme/)

**STGCN**[(130)] In terms of short text classification, STGCN (Short-Text GCN) applies BTM to get topic information to avoid the data sparsity problem from LDA. The graph is constructed following TextGCN while extra topic nodes are included. The edge values of word-topic and document-topic are from BTM and a classical two-layer GCN is applied. The word embeddings learned from STGCN are concatenated with BERT embeddings and a bi-LSTM model is applied for final prediction.

#### 3.4.2. Multi-layer Topic Nodes: DHTG

**DHTG**[(120)] To capture different levels of information, DHTG (Dynamic Hierarchical Topic Graph) introduces hierarchical topic-level nodes in the graph from fine-grain to coarse. Poisson gamma belief network (PGBN)[(145)] is used as a probabilistic deep topic model. The first-layer topics are from the combination of words, while deeper layers are generated by previous layers' topics with the weights of PGBN, and the weights serve as the edge values of each layer of topics. For the topics on the same layer, the cosine similarity is chosen as the edge value. A two-layer GCN is applied and the model is learned jointly with PGBN, which makes the edge of the topics dynamic.

### Critical Analysis

Compared with sequential models like CNN and LSTM, corpus-level GNN is able to capture the global corpus structure information with word nodes as bridges between document nodes and shows great performance without using external resources like pretrained embedding or pretrained model. However, the improvement in performance is marginal when pretrained embedding is included. Another issue is that most corpus-level GNN is transductive learning which is not
applicable in the real world. Meanwhile, constructing the whole corpus into a graph requires large memory space especially when the dataset is large.

A detailed comparison of corpus-level GNN is displayed in Table 2.

## 4. Document-level GNN for Text Classification

By constructing the graph based on each document, a graph classification model can be used as a text classification model. Since each document is represented by one graph and new graphs can be built for test documents, the model can easily work in an inductive way.

### Local Word Consecutive Graph

The simplest way to convert a document into a graph with words as nodes is by connecting the consecutive words within a sliding window.

#### 4.1.1. Simple consecutive graph models: Text-Level-GNN, MPAD, TextING.

**Text-Level-GNN[(37)]** Text-Level-GNN applies a small sliding window and constructs the graph with a small number of nodes and edges in each graph, which saves memory and computation time. The edge value is trainable and shared across the graphs when connecting the same two words, which also brings global information.

Unlike corpus-level graph models, Text-Level-GNN applies a message passing mechanism (MPM)[(30)] instead of GCN for graph learning. For each node, the neighbour information is aggregated using max-pooling with trainable edge values as the AGGREGATE function and then the weighted sum is used as the COMBINE function. To get the representation of each graph, sum-pooling and an MLP classifier are applied as the READOUT function. The propagation shows as:

\[\mathbf{h}_{i}^{(l+1)}=(1-\alpha)(max_{\mathbf{n}\in\mathcal{N}_{i}}e_{\mathbf{n}}\mathbf{h }_{n}^{(I)})+\alpha\mathbf{h}_{i}^{(I)} \tag{26}\] \[\mathbf{z}_{i}=\text{softmax}(\mathbf{W}\Sigma_{i}\mathbf{h}_{i}+\mathbf{b}) \tag{25}\]

where \(\mathbf{h}_{i}^{(I)}\) is \(i\)th word node presentation of layer \(l\), \(e_{\mathbf{n}i}\) is edge weight from node \(n\) to node \(i\). A two-layer MPM is applied, and the input of each graph is pretrained GloVe vectors.

**MPAD[(86)]** MPAD (Message Passing Attention Networks) connects words within a sliding window of size 2 but also includes an additional master node connecting all nodes in the graph. The edge only shows the connectedness of each pair of word nodes and is fixed. A variant of Gated Graph Neural Networks is applied where the AGGREGATE function is the weighted sum and the COMBINE function is GRU[(18)]. Self-attention is applied in the READOUT function.

To learn the high-level information, the master node is directly concatenated with the READOUT output, working as a skip connection mechanism. To get the final representation, each layer's READOUT results are concatenated to capture multi-granularity information. Pretrained Word2vec is used as the initialization of word nodes input.

**TextING[(143)]** To simplify MPAD, TextING ignores the master node in the document-level graphs, which makes the graph sparser. Compared with Text-Level-GNN, TextING remains fixed edges. A similar AGGREGATE and COMBINE function are applied under the concept of e Gated Graph Neural Networks(GGNN)[(58)] with the weighted sum and GRU. However, for the READOUT function, soft attention is used and both max-pooling and mean-pooling are applied to make sure that "every word plays a role in the text and the keywords should contribute more explicitly".

#### 4.1.2. Advanced graph models: MLGNN, TextSSL, DADGNN.
**MLGNN[61]**  MLGNN (Multi-level GNN) builds the same graph as TextING but introduces three levels of MPM: bottom-level, middle-level and top-level. In the bottom-level MPM, the same method with Text-Level-GNN is applied with pretrained Word2vec as input embedding but the edge is non-trainable. In the middle level, a larger window size is adopted and Graph Attention Networks(GAT)[115] is applied to learn long distant word nodes information. In the top-level MPM, all word nodes are connected and multi-head self-attention[114] is applied. By applying three different levels of MPM, MLGNN learns multi-granularity information well.

**DADGNN[70]**  DADGNN (Deep Attention Diffusion GNN) constructs the same graph as TextING but uses attention diffusion to overcome the oversmoothing issue. Pretrained word embedding is used as the input of each node and an MLP layer is applied. Then, the graph attention matrix is calculated based on the attention to the hidden states of each node. The diffusion matrix is calculated as

\[T=\Sigma_{n=0}^{\infty}\epsilon_{n}\mathbf{A}^{n} \tag{27}\]

where \(A\) is the graph attention matrix and \(\epsilon\) is the learnable coefficients. \(A^{n}\) plays a role of connecting \(n\)-hop neighbours and Liu et al. [70] uses \(n\in[4,7]\) in practice. A multi-head diffusion matrix is applied for layer propagation.

**TextSSL[95]**  To solve the word ambiguity problem and show the word synonymity and dynamic contextual dependency, TextSSL (Sparse Structure Learning) learns the graph using intra-sentence neighbours and inter-sentence neighbours simultaneously. The local syntactic neighbour is defined as the consecutive words and trainable edges across graphs are also included by using Gumbel-softmax. By applying sparse structure learning, TextSSL manages to select edges with dynamic contextual dependencies.

### Global Word Co-occurrence Graph

Similar to the TextGCN graph, document-level graphs can also use PMI as the word-word edge values,

#### 4.2.1. Only global word co-occurrence: DAGNN

**DAGNN[125]**  To address the long-distance dependency, hierarchical information and cross-domain learning challenges in domain-adversarial text classification tasks, Wu et al. [125] propose DAGNN (Domain-Adversarial Graph Neural Network). Each document is represented by a graph with content words as nodes and PMI values as edge values, which can capture long-distance dependency information. Pretrained FastText is chosen as the input word embeddings to handle the out-of-vocabulary issue and a GCN model with skip connection is used to address the oversmoothing problem. The propagation is formulated as:

\[\mathbf{H}^{(l+1)}=(1-\alpha)\mathbf{\tilde{A}}\mathbf{H}^{(l)}+\alpha\mathbf{H}^{(0)} \tag{28}\]

To learn the hierarchical information of documents, DiffPool[136] is applied to assign each document into a set of clusters. Finally, adversarial training is used to minimize the loss on source tasks and maximize the differentiation between source and target tasks.

#### 4.2.2. Combine with Extra Edges: ReGNN, GFN

**ReGNN[56]**  To capture both global and local information, ReGNN (Recursive Graphical Neural Network) uses PMI together with consecutive words as the word edges. And graph propagation function is the same as GGNN while additive attention[7] is applied in aggregation. Pretrained GloVe is the input embedding of each word node.

**GFN[20]**  GFN (Graph Fusion Network) builds four types of graphs using the word co-occurrence statistics, PMI, the similarity of pretrained embedding and Euclidean distance of pretrained embedding. Although four corpus-level
graphs are built, the graph learning actually happens on subgraphs of each document, making the method a document-level GNN. For each subgraph, each type of graph is learned separately using the graph convolutional method and then a fusion method of concatenation is used. After an MLP layer, average pooling is applied to get the document representation.

### Other word graphs

Some other ways of connecting words in a document have been explored.

**HyperGAT[(25)]** Ding et al. [(25)] proposes HyperGAT (Hypergraph Attention Networks) which builds hypergraphs for each document to capture high-level interaction between words. Two types of hyperedges are included: sequential hyperedges connecting all words in a sentence and semantic hyperedges connecting top-K words after getting the topic of each word using LDA. Like traditional hypergraph propagations, HyperGAT follows the same two steps of updating but with an attention mechanism to highlight the key information: Node-level attention is applied to learn hyperedges representations and edge-level attention is used for updating node representations.

**IGCN[(110)]** Contextual dependency helps in understanding a document and the graph neural network is no exception. IGCN constructs the graph with the dependency graph to show the connectedness of each pair of words in a document. Then, the word representation learned from Bi-LSTM using POS embedding and word embedding is used to calculate the similarity between each pair of nodes. Attention is used for the output to find the important relevant semantic features.

**GTNT[(80)]** Words with higher TF-IDF values should connect to more word nodes, with this in mind, GTNT(Graph Transformer Networks based Text representation) uses sorted TF-IDF value to determine the degree of each node and applies the Havel-Hakimi algorithm[(32)] to determine the edges between word nodes. A variant of GAT is applied during model learning. Despite the fact that GAT's attention score is mutual for two nodes, GTNT uses relevant importance to adjust the attention score from one node to another. Pretrained Word2vec is applied as the input of each node.

### Critical Analysis

Most document-level GNNs connect consecutive words as edges in the graph and apply a graph neural network model, which makes them similar to CNN where the receptive field enlarges when graph models go deeper. Also, the major differences among document-level GNNs are the details of graph models, e.g. different pooling methods, and different attention calculations, which diminishes the impact of the contribution of these works. Compared with corpus-level GNN, document-level GNN adopts more complex graph models and also suffers from the out-of-memory issue when the number of words in a document is large.

A detailed comparison of document-level GNN is displayed in Table 2.

## 5. Datasets and Metrics

### Datasets

There are many popular text classification benchmark datasets, while this paper mainly focuses on the datasets used by GNN-based text classification applications. Based on the purpose of applications, we divided the commonly adopted datasets into three types including _Topic Classification_, _Sentiment Analysis_ and _Other_. Most of these text classification datasets contain a single target label of each text body. The key information of each dataset is listed in Table 3.
#### 5.1.1. Topic Classification.

Topic classification models aim to classify input text bodies from diverse sources into predefined categories. News categorization is a typical topic classification task to obtain key information from news and classify them into corresponding topics. The input text bodies normally are paragraphs or whole documents especially for news categorization, while there are still some short text classification datasets from certain domains such as micro-blogs, bibliography, etc. Some typical datasets are listed:

* _Ohsumed_[(40)] is acquired from the MEDLINE database and further processed by [(133)] via selecting certain documents (abstracts) and filtering out the documents belonging to multiple categories. Those documents are classified into 23 cardiovascular diseases. The statistics of [(133)] processed Ohsumed dataset is represented in Table 3, which is directly employed by other related works.
* _R8 / R52_ are two subsets of the Reuters 21587 dataset 2 which contain 8 and 52 news topics from Reuters financial news services, respectively. Footnote 2: For the original Reuters 21587 dataset, please refer to this link [http://www.daviddlewis.com/resources/testcollections/reuters21578](http://www.daviddlewis.com/resources/testcollections/reuters21578)
* _20NG_ is another widely used news categorization dataset that contains 20 newsgroups. It was originally collected by [(50)], but the procedures are not explicitly described.

\begin{table}
\begin{tabular}{c|l l l l l} \hline \hline
**Graph** & **Model** & **External Resource** & **Edge Construction** & **Node Initialization** & **Learning** \\ \hline \multirow{8}{*}{**Corpus-level**} & TextGCN[(133)] & N/A & pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \cline{2-6}  & SGC[(123)] & N/A & pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \cline{2-6}  & SGC[(148)] & N/A & pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \cline{2-6}  & NMGC[(53)] & N/A & pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \cline{2-6}  & TG-transformer[(137)] & GloVe & pmi, tf\({}^{-}\)idf & GloVe & transductive \\ \cline{2-6}  & BERTGCN[(63)] & BERT & pmi, tf\({}^{-}\)idf & doc: 0 word: BERT emb & transductive \\ \cline{2-6}  & TensorGCN[(63)] & GloVe, CoreNLP & emb sim, dep graph, pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \cline{2-6}  & MI-GCN[(118)] & N/A & emb sim, tf\({}^{-}\)idf & Trained Word2vec/doc2vec & transductive \\ \cline{2-6}  & HetGCN[(158)] & N/A & pmi, tf\({}^{-}\)idf & one-hot & inductive \\ \cline{2-6}  & InducT-GCN[(119)] & N/A & pmi, tf\({}^{-}\)idf & one-hot, tf\({}^{-}\)idf vectors & inductive \\ \cline{2-6}  & TVGAE[(127)] & N/A & pmi & one-hot & inductive \\ \cline{2-6}  & VGCN-BERT[(73)] & BERT & pmi & BERT emb & transductive \\ \cline{2-6}  & klm-GCN[(5)] & GloVe & emb sim & GloVe & transductive \\ \cline{2-6}  & TextGTL[(54)] & CoreNLP & dep graph, pmi & tf\({}^{-}\)idf vectors & transductive \\ \cline{2-6}  & HGAT[(64)] & TAGME, Word2vec & LDA, entity link, emb sim & tf\({}^{-}\)idf, LDA, Word2vec & transductive \\ \cline{2-6}  & STGCN[(134)] & BERT & pmi, tf\({}^{-}\)idf, BTM & BERT emb & transductive \\ \cline{2-6}  & DHTG[(120)] & N/A & PGRN, pmi, tf\({}^{-}\)idf & one-hot & transductive \\ \hline \multirow{8}{*}{**Doc-level**} & Text-Level-GNN[(37)] & GloVe & consecutive words & GloVe & inductive \\ \cline{2-6}  & MPAD[(86)] & Word2vec & consecutive words & Word2vec & inductive \\ \cline{2-6}  & TextING[(143)] & GloVe & consecutive words & GloVe & inductive \\ \cline{2-6}  & MLGNN[(61)] & Word2vec & consecutive words & Word2vec & inductive \\ \cline{2-6}  & DADGNN[(70)] & Word2vec/GloVe & consecutive words & Word2vec/GloVe & inductive \\ \cline{2-6}  & TextSL[(95)] & GloVe & consecutive words & GloVe & inductive \\ \cline{2-6}  & DAGNN[(125)] & GloVe & pmi & GloVe & inductive \\ \cline{2-6}  & ReGNN[(56)] & GloVe & consecutive words, pmi & GloVe & inductive \\ \cline{2-6}  & GFN[(20)] & GloVe & pmi, emb sim & GloVe & inductive \\ \cline{2-6}  & HyperGAT[(25)] & N/A & LDA, consecutive words & one-hot & inductive \\ \cline{2-6}  & RGCN[(110)] & spatCy & dep graph & LSTM emb & inductive \\ \cline{2-6}  & GTNT[(80)] & Word2vec/GloVe & tf\({}^{-}\)idf sorted value & Word2vec/GloVe & inductive \\ \hline \hline \end{tabular}
\end{table}
Table 2. Models Detailed Comparison on whether using external resources, how to construct the edge and node input, and whether transductive learning or inductive learning. GloVe and Word2vec are pretrained if not specified. \({}^{*}\)emb sim\({}^{*}\) is short for \({}^{*}\)embedding similarity\({}^{*}\), \({}^{*}\)dep graph\({}^{*}\) is short \({}^{*}\)dependency graph\({}^{*}\).
* _AG News_[141] is a large-scale news categorization dataset compared with other commonly used datasets which are constructed by selecting the top-4 largest categories from the AG corpus. Each news topic contains 30,000 samples for training and 1900 samples for testing.
* _Database systems and Logic Programming (DBLP)_ is a topic classification dataset to classify the computer science paper titles into six various topics [80]. Different from paragraph or document based topic classification dataset, DBLP aims to categorise scientific paper titles into corresponding categories, the average input sentence length is much lower than others.
* _Dbpedia_[52] is a large-scale multilingual knowledge base that contains 14 non-overlapping categories. Each category contains 40000 samples for training and 5000 samples for testing.
* _WebKB_[19] is a long corpus web page topic classification dataset.
* _TREC_[57] is a question topic classification dataset to categorise one question sentence into 6 question categories.

#### 5.1.2. Sentiment Analysis.

The purpose of sentiment analysis is to analyse and mine the opinion of the textual content which could be treated as a binary or multi-class classification problem. The sources of existing sentiment analysis tasks come from movie reviews, product reviews or user comments, social media posts, etc. Most sentiment analysis datasets target to predict the people's opinions of one or two input sentences of which the average length of each input text body is around 25 tokens.

\begin{table}
\begin{tabular}{c|l l l l l l l l} \hline \hline
* _Movie Review (MR)_[(89)] is a binary sentiment classification dataset for movie review which contains positive and negative data equally distributed. Each review only contains one sentence.
* _Stanford Sentiment Treebank (SST)_[(106)] is an upgraded version of MR which contains two subsets SST-1 and SST-2. SST-1 provides five fine-grained labels while SST-2 is a binary sentiment classification dataset.
* _Internet Movie DataBase (IMDB)_[(74)] is also an equally distributed binary classification dataset for sentiment analysis. Different from other short text classification dataset, the average number of words of each review is around 221.
* _Yelp 2014_[(109)] is a large scale binary category based sentiment analysis dataset for longer user reviews collected from Yelp.com.

Certain binary sentiment classification benchmark datasets are also used by GNN-based text classifiers. Most of them are gathered from shorter user reviews or comments (normally one or two sentences) from different websites including Amazon Alexa Reviews (_AAR_), Twitter US Airline (_TUA_), Youtube comments (_SenTube-A_ and _SenTube-T_) [(113)].

#### 5.1.3. Other Datasets

There are some datasets targeting other tasks including hate detection, grammaticality checking, etc. For example, _ArangoHate_[(4)] is a hate detection dataset, a sub-task of intend detection, which contains 2920 hateful documents and 4086 normal documents by resampling the merged datasets from [(21)] and [(121)]. In addition, [(26)] proposes another large scale hate language detection dataset, namely _FountaHate_ to classify the tweets into four categories including 53,851, 14,030, 27,150, and 4,965 samples of normal, spam, hateful and abusive, respectively. Since there is no officially provided training and testing splitting radio for above datasets, the numbers represented in Table 3 are following the ratios (train/development/test is 85:5:10) defined by [(73)].

#### 5.1.4. Dataset Summary

Since an obvious limitation of corpus-level GNN models has high memory consumption limitation [(37; 25; 137)], the datasets with a smaller number of documents and vocabulary sizes such as Ohsumed, R8/R52, 20NG or MR are widely used to ensure feasibly build and evaluate corpus-level graphs. For the document-level GNN based models, some larger size datasets like AG-News can be adopted without considering the memory consumption problem. From Table 3, we could find most of the related works mainly focus on the GNN applied in topic classification and sentiment analysis which means the role of GNNs in other text classification tasks such as spam detection, intent detection, abstractive question answering need to be further exploited. Another observed trend is short text classification are gained less attention compared with long document classification tasks. In this case, GNN in short text classification may be an.

### Evaluation Methods

#### 5.2.1. Performance Metrics

In terms of evaluating and comparing the performance of proposed models with other baselines, accuracy and F1 are most commonly used metrics to conduct overall performance analysis, ablation studies and breakdown analysis. We use \(TP\), \(FP\), \(TN\) and \(FN\) to represent the number of true positive, false positive, true negative and false negative samples. \(N\) is the total number of samples.

* _Accuracy_ and _Error Rate_: are basic evaluation metrics adopted by many GNN-based text classifiers such as [(54; 67; 120; 133; 137)]. Most of the related papers run all baselines and their models 10 times or 5 times to show
the \(mean\pm standard\)\(deviation\) of accuracy for reporting more convincing results. It can be defined as: \[Accuracy=\frac{(TF+TN)}{N},\] (29) \[ErrorRate=1-Accuracy=\frac{(FP+FN)}{N}.\] (30)
* _Precision_, _Recall_ and _F1_: are metrics for measuring the performance especially for imbalanced datasets. Precision is used to measure the results relevancy, while recall is utilised to measure how many truly relevant results acquired. Through calculating the harmonic average of Precision and Recall could get F1. Those three measurements can be defined as: \[Precision=\frac{TP}{(TP+FP)},\] (31) \[Recall=\frac{TP}{(TP+FN)},\] (32) \[F1=\frac{2\times Precision\times Recall}{(Precision+Recall)},\] (33) Few papers only utilise recall or precision to evaluate the performance [80]. However, precision and recall are more commonly used together with F1 or Accuracy to evaluate and analyse the performance from different perspectives e.g. [56, 64, 73, 127]. In addition, based on different application scenarios, different F1 averaging methods are adopted by those papers to measure overall F1 score of multi-class (Number of Classes is \(C\)) classification tasks including:
* _Macro-F1_ applies the same weights to all categories to get overall \(F1_{macro}\) by taking the arithmetic mean. \[F1_{macro}=\frac{1}{C}\Sigma_{i=1}^{C}F1_{i},\] (34)
* _Micro-F1_ is calculated by considering the overall \(P_{micro}\) and \(R_{micro}\). It can be defined as: \[F1_{micro}=\frac{2\times P_{micro}\times R_{micro}}{(P_{micro}+R_{micro})}\] (35) where: \[P_{micro}=\frac{\Sigma_{i\in C}TP_{i}}{\Sigma_{i\in C}TP_{i}+FP_{i}},R_{micro }=\frac{\Sigma_{i\in C}TP_{i}}{\Sigma_{i\in C}TP_{i}+FN_{i}},\] (36)
* _Weighted-F1_ is the weighted mean of F1 of each category where the weight \(W_{i}\) is related to the number of occurrences of the corresponding \(i\)th class, which can be defined as: \[F1_{macro}=\Sigma_{i=1}^{C}F1_{i}\times W_{i},\] (37)

#### 5.2.2. Other Evaluation Aspects.

Since two limitations of GNN-based models are time and memory consumption, therefore, except the commonly used qualitative performance comparison, representing and comparing the GPU or CPU memory consumption and the training time efficiency of proposed models are also adopted by many related studies to demonstrate the practicality in real-world applications. In addition, based on the novelties of various models, specific evaluation methods are conducted to demonstrate the proposed contributions.

* _Memory Consumption_: [25, 37, 70] list the memory consumption of different models for comprehensively evaluating the proposed models in computational efficiency aspect.
* _Time Measurement_: [90; 98] perform performance training time comparison between their proposed models and baselines on different benchmarks. Due to the doubts about the efficiency of applying GNNs for text classification, it is an effective way to demonstrate they could well balance performance and time efficiency.
* _Parameter Sensitivity_ is commonly conducted by GNNs studies to investigate the effect of different hyperparameters e.g. varying sliding window sizes, embedding dimensions of proposed models to represent the model sensitivity via line chart such as [70; 25; 64].
* _Number of Labelled Documents_ is a widely adopted evaluation method by GNN-based text classification models [25; 54; 80; 120; 133; 64] which mainly analyse the performance trend by using different proportions of training data to test whether the proposed model can work well under the limited labelled training data.
* _Vocabulary Size_ is similar to the number of labelled documents but it investigates the effects of using different sizes of vocabulary during the GNN training stage adopted by [120].

#### 5.2.3. Metrics Summary.

For general text classification tasks, Accuracy, Precision, Recall and varying F1 are commonly used evaluation metrics for comparing with other baselines. However, for GNN based models, only representing the model performance cannot effectively represent the multi-aspects of proposed models. In this case, there are many papers conducting external processes to evaluate and analyse the GNN based classifier from multiple views including time and memory consumption, model sensitivity and dataset quantity.

## 6. Performance

While different GNN text classification models may be evaluated on different datasets, there are some datasets that are commonly used across many of these models, including **20NG**, **R8**, **R52**, **Ohsumed** and **MR**. The accuracy of various models assessed on these five datasets is presented in Table 4. Some of the results are reported with ten times average accuracy and standard derivation while some only report the average accuracy. Several conclusions can be drawn:

\begin{table}
\begin{tabular}{c|l l l l l l l} \hline \hline \multirow{2}{*}{**Type**} & \multirow{2}{*}{**Method**} & \multicolumn{2}{l}{**External**} & \multirow{2}{*}{**20NG**} & \multirow{2}{*}{**R8**} & \multirow{2}{*}{**R52**} & \multirow{2}{*}{**Ohsumed**} & \multirow{2}{*}{**MR**} \\  & & **Resource** & & & & & & \\ \hline \multirow{11}{*}{**Corpus-level**} & TextGCN [133] & N/A & 86.34 \(\pm\) 0.09 & 97.07 \(\pm\) 00.10 & 93.56 \(\pm\) 0.18 & 68.36 \(\pm\) 0.56 & 76.74 \(\pm\) 0.20 \\ \cline{2-7}  & SGC [123] & N/A & 88.5 \(\pm\) 0.1 & 97.2 \(\pm\) 0.1 & 94.0 \(\pm\) 0.2 & 68.5 \(\pm\) 0.3 & 75.9 \(\pm\) 0.3 \\ \cline{2-7}  & S2GC [148] & N/A & 88.6\(\pm\) 0.1 & 97.4 \(\pm\) 0.1 & 94.5 \(\pm\) 0.2 & 68.5 \(\pm\) 0.1 & 76.7 \(\pm\) 0.0 \\ \cline{2-7}  & TG-transformer [137] & GloVe & - & 98.1\(\pm\)0.1 & 95.2\(\pm\)0.2 & 70.4\(\pm\)0.4 & - \\ \cline{2-7}  & DHTG [120] & N/A & 87.13 \(\pm\) 0.07 & 97.33 \(\pm\) 0.06 & 93.93 \(\pm\) 0.10 & 68.80 \(\pm\) 0.33 & 77.21 \(\pm\) 0.11 \\ \cline{2-7}  & TensorGCN [68] & GloVe, CoreNLP & 87.74 \(\pm\) 0.05 & 98.04 \(\pm\) 0.08 & 95.05 \(\pm\) 0.11 & 70.11 \(\pm\) 0.24 & 77.91 \(\pm\) 0.07 \\ \cline{2-7}  & STGCN [134] & BERT & - & 98.5 & - & - & 82.5 \\ \cline{2-7}  & NMGC [53] & N/A & 86.61 \(\pm\) 0.06 & 97.31 \(\pm\) 0.09 & 94.35 \(\pm\) 0.06 & 69.21 \(\pm\) 0.17 & 76.21 \(\pm\) 0.25 \\ \cline{2-7}  & BertGCN [63] & BERT & 89.3 & 98.1 & 96.6 & 72.8 & 86 \\ \cline{2-7}  & RobertaGCN [63] & RoBERTa & 89.5 & 98.2 & 96.1 & 72.8 & 89.7 \\ \cline{2-7}  & T-VGAE [127] & N/A & 88.08 \(\pm\) 0.06 & 97.68 \(\pm\) 0.14 & 95.05 \(\pm\) 0.10 & 70.02 \(\pm\) 0.14 & 78.03 \(\pm\) 0.11 \\ \hline \multirow{11}{*}{**Doc-level**} & ReGNN [56] & GloVe & - & 97.93 \(\pm\) 0.31 & 95.17 \(\pm\) 0.17 & 67.93 \(\pm\) 0.33 & 78.71 \(\pm\) 0.56 \\ \cline{2-7}  & Text-Level-GNN [37] & GloVe & 84.16 \(\pm\) 0.25\({}^{*}\) & 97.8 \(\pm\) 0.2 & 94.6 \(\pm\) 0.3 & 69.4 \(\pm\) 0.6 & 75.47 \(\pm\) 0.06\({}^{*}\) \\ \cline{1-1} \cline{2-7}  & TextING [143] & GloVe & - & 98.13 \(\pm\) 0.12 & 95.68 \(\pm\) 0.35 & 70.84 \(\pm\) 0.52 & 80.19 \(\pm\) 0.31 \\ \cline{1-1} \cline{2-7}  & HyperGAT [25] & N/A & 86.62 \(\pm\) 0.16 & 97.97 \(\pm\) 0.23 & 94.98 \(\pm\) 0.27 & 69.90 \(\pm\) 0.34 & 78.32 \(\pm\) 0.27 \\ \cline{1-1} \cline{2-7}  & TextSSL [95] & GloVe & 85.26 \(\pm\) 0.28 & 97.81 \(\pm\) 0.14 & 95.48 \(\pm\) 0.26 & 70.59 \(\pm\) 0.38 & 79.74 \(\pm\) 0.19 \\ \hline \hline \end{tabular}
\end{table}
Table 4. Performance Table. - indicates unavailability. \({}^{*}\) refers to replication from HyperGAT [25].
* Models that use external resources usually achieve better performance than those that do not, especially models with BERT and RoBERTa(Rosen et al., 2019; Wang et al., 2020).
* Under the same setting, such as using GloVe as the external resource, Corpus-level GNN models (e.g. TG-Transformer(Wang et al., 2020), TensorGCN(Wang et al., 2020)) typically outperform Document-level GNN models (e.g. TextING(Wang et al., 2020), TextSSL(Wang et al., 2020)). This is because Corpus-level GNN models can work in a transductive way and make use of the test input, whereas Document-level GNN models can only use the training data.
* The advantage of Corpus-level GNN models over Document-level GNN models only applies to topic classification datasets and not to sentiment analysis datasets such as **MR**. This is because sentiment analysis involves analyzing the order of words in a text, which is something that most Corpus-level GNN models cannot do.

## 7. Challenges and Future Work

### Model Performance

With the development of pre-trained models(Wang et al., 2020; Wang et al., 2020), and prompt learning methods(Wang et al., 2020; Wang et al., 2020) achieve great performance on text classification. Applying GNNs in text classification without this pre-training style will not be able to achieve such good performance. For both corpus-level and document-level GNN text classification models, researching how to combine GNN models with these pretrained models to improve the pretrained model performance can be the future work. Meanwhile, more advanced graph models can be explored, e.g. more heterogeneous graph models on word and document graphs to improve the model performance.

### Graph Construction

Most GNN text classification methods use a single, static-value edge to construct graphs based on document statistics. This approach applies to both corpus-level GNN and document-level GNN. However, to better explore the complex relationship between words and documents, more dynamic hyperedges can be utilized. Dynamic edges in GNNs can be learned from various sources, such as the graph structure, document semantic information, or other models. And hyperedges can be built for a more expressive representation of the complex relationships between nodes in the graph.

### Application

While corpus-level GNN text classification models have demonstrated good performance without using external resources, these models are mostly transductive. To apply them in real-world settings, an inductive learning approach should be explored. Although some inductive corpus-level GNNs have been introduced, the large amount of space required to construct the graph and the inconvenience of incremental training still present barriers to deployment. Improving the scalability of online training and testing for inductive corpus-level GNNs represents a promising area for future work.

## 8. Conclusion

This survey article introduces how Graph Neural Networks have been applied to text classification in two different ways: corpus-level GNN and document-level GNN, with a detailed structural figure. Details of these models have been introduced and discussed, along with the datasets commonly used by these methods. Compared with traditional machine learning and sequential deep learning models, graph neural networks can explore the relationship between words and documents in the global structure (corpus-level GNN) or the local document (document-level GNN), giving a good
performance. A detailed performance comparison is applied to investigate the influence of external resources, model learning methods, and types of different datasets. Furthermore, we propose the challenges for GNN text classification models and potential future work.
