Connecting the Dots: What Graph-Based Text Representations Work Best for
  Text Classification Using Graph Neural Networks?
Given the success of Graph Neural Networks (GNNs) for structure-aware machine
learning, many studies have explored their use for text classification, but
mostly in specific domains with limited data characteristics. Moreover, some
strategies prior to GNNs relied on graph mining and classical machine learning,
making it difficult to assess their effectiveness in modern settings. This work
extensively investigates graph representation methods for text classification,
identifying practical implications and open challenges. We compare different
graph construction schemes using a variety of GNN architectures and setups
across five datasets, encompassing short and long documents as well as
unbalanced scenarios in diverse domains. Two Transformer-based large language
models are also included to complement the study. The results show that i)
although the effectiveness of graphs depends on the textual input features and
domain, simple graph constructions perform better the longer the documents are,
ii) graph representations are especially beneficial for longer documents,
outperforming Transformer-based models, iii) graph methods are particularly
efficient at solving the task.
Margarita Bugueño, Gerard de Melo
Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification using Graph Neural Networks?

###### Abstract

Given the success of Graph Neural Networks (GNNs) for structure-aware machine learning, numerous studies have explored their application to text classification, as an alternative to traditional feature representation models. However, most studies considered just a specific domain and validated on data with particular characteristics. This work presents an extensive empirical investigation of graph-based text representation methods proposed for text classification, identifying practical implications and open challenges in the field. We compare several GNN architectures as well as BERT across five datasets, encompassing short and also long documents. The results show that: i) graph performance is highly related to the textual input features and domain, ii) despite its outstanding performance, BERT has difficulties converging when dealing with short texts, iii) graph methods are particularly beneficial for longer documents.

## 1 Introduction

The predominant forms of text representation based on vectors make it non-trivial to recover much information about the structure of a given text. Graph-based text representation strategies instead explicitly seek to capture relationships among the different text elements (nodes) by the definition of associations between pairs of them (edges). While such ideas have a long history (Hassan and Banea, 2006, _inter alia_), in recent years we have witnessed the rise of Graph Neural Network (GNN) models. These make it particularly appealing to convert even unstructured data into graphs to enable new forms of analysis and extract significant patterns by capturing dependencies between graph nodes via message passing.

For text classification, several different proposals of graph-based text representation schemes have been put forth, but the majority were introduced as a solution for a particular domain-specific task and validated on a limited range of datasets with a limited set of model architectures. Some of these were proposed before the introduction of GNNs and hence validated using graph mining techniques or classical machine learning models, making it difficult to assess their merits in modern settings.

This work presents a thorough empirical investigation of previously proposed graph-based text representation methods for text classification. We analyze their performance with several GNN-based architectures across five datasets. Unlike previous work (Galke and Scherp, 2022), our study considers diverse datasets both with short and long documents (more than 512 tokens). While longer documents can be particularly challenging, our study finds that GNN methods hold particularly promise for longer documents. Furthermore, we provide a particularly extensive empirical comparison of graph-based text representation methods and shed light on what are the most successful choices of GNN architectures for learning from them.

The main contributions of the study are:

1. We evaluate the merits of previously proposed graph-based text representations across a variety of GNN architectures and datasets.
2. We study the effect of four different node feature initialization schemes.
3. We identify the practical implications of the previously proposed graph-based text representations, shedding light on open challenges.

## 2 Prior Work on Graphs in NLP

A notable early approach for graph-based document representation is to consider co-occurrence within a fixed-size sliding window (Mihalcea and Tarau, 2004). Words are modeled as nodes, and two nodes are connected if the corresponding terms co-occur within a window of at most \(N\) words. Mihalcea and Tarau (2004) used such co-occurrence graphs for \(N\in\{2,\ldots,10\}\) as a graph-based ranking model for keyword extraction. They found
lower choices of \(N\) to be preferable, as the connection between words further apart is often weaker. Hassan and Banea (2006) used the same approach for \(N=2\) along with TextRank to replace term frequency weights, and then conducted text classification with classic machine learning models. In most of their experiments, this scheme outperformed using TF-IDF vectors. Rousseau et al. (2015) also used a fixed-size sliding window graph (calling it _graph-of-words_). They cast text classification as a graph classification problem by applying graph mining to obtain subgraph features to train a classifier.

_Sequence graphs_ are another simple scheme with edges reflecting the original order of words in the text (Castillo et al., 2015). The authors used the number of times the corresponding two words appear consecutively in the text as edge weights.

Other graph construction methods have been proposed. Mihalcea and Tarau (2004) highlighted that depending on the application at hand, multiple text units and characteristics can be considered as vertices, such as words, collocations, entire sentences, etc. They invoked application-specific criteria to define edges, such as lexical or semantic relations. To this end, they also proposed a similarity-weighted graph for sentence extraction. Every node represents an entire sentence, while edges are defined by measuring their content overlap as the number of shared tokens. Although this scheme can be applied in other tasks (text classification or summarization), it normally results in a fairly densely connected graph, making it difficult to extract local patterns and differentiate the content of the text.

Given that traditional work in linguistics and computational linguistics often considers tree and graph-structured formalisms as the principal way of analyzing individual sentences, these may also serve as building blocks for document-level representations (Arora et al., 2009; Joshi and Rose, 2009, _inter alia_). For instance, a neural parsing model (Dozat and Manning, 2016; Yuan et al., 2021) can infer word dependencies to obtain syntactic dependency trees. However, the overall graph representation becomes rather sparse, as nodes share edges with only a limited number of other units.

Text Graph Convolutional Network (TextGCN; Yao et al. 2019) is a recent high-impact approach, indeed also one of the first approaches to include a Graph Convolutional Neural Network (GCN) as a classification method. TextGCN proposes a heterogeneous graph with nodes for words and documents, and edges for associations between pairs of words and document-word associations representing the terms contained in a specific document. In addition, TextGCN defines edge weights as Point-wise Mutual Information (PMI) similarity for word pairs and TF-IDF for word-in-document edges. This approach casts document classification as a node classification problem. Liu et al. (2020) proposes a text graph tensor to learn word and document embeddings using semantic, syntactic, and sequential contextual information. However, these methods are computationally inefficient and fail to capture higher-order interactions between words.

Alternatively, TextLevelGCN (Huang et al., 2019) creates one graph per input text. The proposal defines every word as a node, which can be duplicated if a word appears more than once in a text. Edges are defined for word nodes within a sliding window using PMI edge weights. Despite promising results, the experiments were limited to very short documents.

GraphIE (Qian et al., 2019) proposes a different scheme based on co-reference, integrating a GCN with an RNN encoder-decoder architecture for sequence tagging and information extraction tasks. Nodes can be defined as words or entire sentences, while associations are defined as co-reference and identical mention edges, to account for non-local and non-sequential ties. Despite enhancing the accuracy in the target tasks, the strategy requires prior domain knowledge to establish the edge types.

Recently, some studies have brought back the classic co-occurrence graph construction methods but this time using a different message passing function based on Gated Recurrent Units (Li et al., 2015; Cho et al., 2014) for updating the node feature vectors (Zhang et al., 2020). MPAD (Nikolentzos et al., 2020) included an extra master node connected to every other node in the graph. The network is densely connected, and the structural information is vague during message passing.

Since Zhang et al. (2020) outperform TextlevelGCN despite using the same graph construction, it is clear that the graph construction method and the way patterns are extracted from it are closely related. Hence, an in-depth study analyzing multiple factors in a controlled environment is necessary.

In terms of broader empirical comparisons, one previous study also conducted an empirical analysis
of GNN approaches for text classification (Galke and Scherp, 2022). Nevertheless, despite the comprehensive evaluation, the authors limited their analysis to standard data collections with only short texts. In contrast, our study considers a broader range of domains and more challenging datasets, including much longer documents and unbalanced classification scenarios.

## 3 Comparing Graph-Based Text Representations

To study the merits of the previously proposed graph-based text representation strategies, we conducted comprehensive experiments on five text classification datasets widely known in the area. In each of these tasks, we compare different graph construction schemes using a variety of GCN models to separate the effect of the graph construction strategy from that of the message-passing technique in the model architecture. For reference, we also evaluated the BERT model as a baseline.

### Methods

#### 3.1.1 Graph-Based Text Representation

Among the studied techniques, there are some graph construction methods that follow an intuitive construction process. They are based solely on simple relationships between pairs of nodes and only consider basic co-occurrence statistics if needed. Thus, they do not require a deep understanding of the semantic structure. In the following, we refer to these sorts of networks as **Intuitive Graphs**. Figure 1 illustrates how they work.

**Window-based:** Following Hassan and Banea (2006), given an input text, if a term has not been previously seen, then a node is added to the graph to represent it, and an undirected edge is induced between two nodes simply if they are two consecutive terms in the text.

**Window-based extended:** As for the above window-based construction, except that the window size is extended to three. Then, the tokens contained in the window define edges among each other. With this, each word in the input text will ultimately be tied to the two previous terms and the two subsequent ones.

**Sequence-weighted:** This strategy (Castillo et al., 2015) defines a directed graph with nodes for words and edges that represent that the two corresponding lexical units appear together in the text sequence and follow the order in which they appear. Additional edge weights capture the number of times that two vertices appear together in the text to represent the relationship's importance in the text.

**Sequence simplified:** Inspired by the above sequential graphs, a simplified version is considered that omits the edge weights. Thus, the effect of the edge importance function over the pure graph structure can be studied in isolation.

A more sophisticated graph-based text representation strategy requiring a deeper understanding is also considered. Contrary to the previous intuitive schemes, it requires an elaborate process for defining the graph components.

**TextLevelGCN:** Every word appearing in a text is treated as a node, and edges are defined between adjacent words in a fixed-size window. Unlike Intuitive Graphs, TextLevelGCN (Huang et al., 2019) considers each word token occurrence as a separate node, i.e., it allows multiple nodes if the corresponding term occurs more than once in the text. Therefore, even for polysemous words, the specific in-context meaning can be determined by the influence of weighted information from its neighbors, improving the expressiveness of the graph. The authors further employed PMI as an edge weighting function for the word associations as in Yao et al. (2019).

#### 3.1.2 Traditional Text Representation

Finally, BERT (Devlin et al., 2018) is also included as a baseline to complement our comparative study. This enables us to better understand how the results of the graph methods relate to those of mainstream pretrained large language models.

**BERT:** A BERT pre-trained model is included by freezing the encoder architecture and stacking a final dense layer for conducting the corresponding text classification task.

**FF BERT:** Fully fine-tuning BERT for each task was also considered without any freezing. This way, the full expressiveness of the model is involved in learning downstream patterns for solving the corresponding task.

### Datasets

The literature review reveals that many graph-based text representation methods have been evaluated on different datasets. Most of the time, the proposals were each introduced for a specific task domain and validated on text with very restricted characteristics such as a limited vocabulary and an average
document length not greater than 221 words [16, 17]. Hence, it is unclear how these approaches can generalize to new data and be applied to long documents in a more general scenario.

We consider five publicly available datasets for text classification. These are labeled by experts and validated by the community. The datasets are summarized in Table 1.

**App Reviews**[1] - A collection of 288,065 English user reviews of Android applications from 23 different app categories, with the goal of fine-grained sentiment analysis in an imbalanced setting, where 60.5% of the total samples correspond to 4-star reviews. Each example includes the name of the software application package, the comment, the date when the user posted the evaluation, and the rating provided.

**DBpedia**[16] - For topic classification, the DBpedia ontology classification dataset was constructed by picking 14 non-overlapping classes from DBpedia 2014 [10]. For each category, the authors randomly chose 40,000 Wikipedia articles as training samples and 5,000 samples for testing. Every article contains the title, content, and class label. Although the original DBpedia is a multilingual knowledge base, this dataset only contains English data.

**IMDB**[16] - English language movie reviews from the Internet Movie Database for binary sentiment classification, with 25,000 reviews for training and 25,000 for testing, with balanced numbers of positive and negative reviews.

**BBC News**[1] - A publicly available1 dataset consisting of 2,225 English documents from the BBC News website. The articles correspond to stories from 2004-5 in the areas of business, entertainment, politics, sport, and technology. The dataset exhibits minor class imbalance, with sports being the majority class with 511 articles, while entertainment is the smallest one with 386 samples.

Footnote 1: [http://derekgreene.com/bbc/](http://derekgreene.com/bbc/)

**Hyperpartisan News Detection (HND)**[17] - A dataset2 for binary news classification. Although it comprises two parts, _byarticle_ and _hypublisher_, this study only uses the first one. The dataset has 645 English samples labeled through crowdsourcing, with 238 (37%) labeled as hyperpartisan and 407 (63%) as not hyperpartisan.

Footnote 2: [https://zenodo.org/HNDrecord](https://zenodo.org/HNDrecord)

### Experimental Setup

#### 3.3.1 Data Preparation

A fixed-size data partition was taken from each dataset to conduct a fair comparative analysis among the methods. Thus, a training and test split was defined, consisting of 7,000 and 3,000 samples, respectively. For those datasets that did not have

\begin{table}
\begin{tabular}{l|r|r|r} \hline
**Dataset** & **Avg. Length** & **NoC** & **IRate** \\ \hline App Reviews & 14 & 5 & 1:8 \\ \hline DBpedia & 51 & 14 & 1:1 \\ \hline IMDB & 283 & 2 & 1:1 \\ \hline BBC News & 438 & 5 & 4:5 \\ \hline HND & 912 & 2 & 1:2 \\ \hline \end{tabular}
\end{table}
Table 1: **Statistics of text classification datasets.** It includes the average length of documents, the number of classes (NoC), and the imbalance rate between the minority and majority classes (IRate).

Figure 1: **Intuitive Graph methods.** Given the input text “_Start working! The sooner you start working, the sooner you will have money_”, four co-occurrence graph representations are shown. These include a window-based graph [16], an extended version by setting the window size to \(3\) (new edges are shown as dashed), a sequence-weighted graph [17], and a simplified version that omits edge weights.
that many examples, i.e., BBC News and HND, 80% of the samples were used for training and the remaining 20% for testing. For all datasets, we randomly reserve \(10\%\) of the samples from the training set for building the validation set.

Since each graph node would represent a word of the input text, specific standardization rules are needed. Accordingly, the text normalization procedure encompassed lowercase conversion, punctuation mark and stop word removal, as well as removing any other non-ASCII character.

Note that our TextLevelGCN experiments are built on the implementation3 published by the corresponding authors, on which certain additional operations are defined for data preprocessing. These include the removal of every token whose length is less than three characters, the definition of a maximum length allowed for the documents to be processed to 350 terms, elimination of words whose frequency is less than 5, the application of a lemmatizer, as well as the definition of expansion rules for the removal of English contractions.

Footnote 3: [https://github.com/mojave-pku/TextLevelGCN](https://github.com/mojave-pku/TextLevelGCN)

#### 3.3.2 Model Setups

Graph Neural Networks.The architectural design for Intuitive Graph experiments considered varying the number of hidden layers from 1 to 4 and varying the dimensionality of the node representation vector in \(\{16,32,64\}\). We applied Dropout after every convolutional layer with a retention probability of 0.8 and used average pooling for node-level aggregation. The final representation was fed into a softmax classifier.

The experiments were conducted employing three different types of graph convolutional neural layers: (i) the traditional one (GCN; Kipf and Welling 2016), (ii) using a graph isomorphism operator (GIN; Xu et al. 2018), which has shown improved structural discriminative power compared to GCNs, and (iii) including a graph attentional operator (GAT; Velickovic et al. 2017) with 4 attention heads. We followed the implementations provided in PyTorch Geometric (see Appendix C).

For TextLevelGCN, we used default parameter settings as in the original implementation, varying the window size (n-gram parameter) from 1 to 4.

All GNNs used a batch size of 64 samples and were trained for a maximum of 100 epochs using Adam optimization (Kingma and Ba, 2014) with an initial learning rate of \(10^{-4}\). The training was stopped if the validation macro-averaged F1 score did not improve for ten consecutive epochs. Only for HND, the patience was set at 20.

Four different node vector initialization strategies were also compared. We considered GloVe Wiki-Gigaword 300-dim. embeddings (Pennington et al., 2014), Word2Vec Google News 300-dim. embeddings (Mikolov et al., 2013), static BERT pre-trained embeddings encoding each token in isolation, and a contextualized version of BERT embeddings. The latter involves encoding the entire input text using BERT and using the corresponding embedding from the 12th layer.

BERT baselines.We used BERT-base uncased as the pre-trained model. A Dropout layer right after it was included with a retention probability of 80%, and a final dense layer was stacked for conducting the text classification.

During training, we froze the BERT architecture and trained the classifier using Adam optimization (Kingma and Ba, 2014) with an initial learning rate of \(10^{-4}\). As for GNNs, we set a batch size of 64 samples, 100 epochs as a maximum, early stopping based on macro-averaged F1-scores with a patience of 10. In turn, for fully fine-tuning BERT, the batch size and learning rate were decreased to 32 and \(10^{-6}\), respectively. The maximum number of epochs was 10, and the patience was 5.

The objective function of each model was to minimize the cross-entropy loss. Accuracy and macro-averaged F1-score metrics were evaluated, as some datasets exhibit class imbalance. The experiments were run on an NVIDIA Quadro RTX A6000 with \(48\)GB VRAM. For reproducibility, we release our code on [https://github.com/](https://github.com/)<anonymized>/<anonymized>.

## 4 Results and Discussion

The results reported below show the best-performing architectures for each dataset. For a full comparison of the results, see Appendix A.

Each table reports the accuracy and the F1 score on the corresponding test partition at the macro level, as averages over 10 runs. For each intuitive graph, the best results are reported in bold. A star mark is used to indicate the best architecture across the entire dataset. The comparison with baselines is presented in Table 3.
### What Convolutional Layer Works Best?

GIN obtained the best results for the App Reviews and HND datasets (Table 2), showing that structural information is highly relevant for the respective tasks. Regarding the other datasets, the attention heads made it possible to efficiently identify the most influential nodes in the corresponding graphs and thus obtain the best results. Likewise, it is observed that the GAT models are stable to variations in parameters such as the number of convolutional layers and the number of hidden units.

### Do High-Dimensional Embeddings Yield Better Results?

There is no clear relationship between the initialization strategy and the architecture of the GNN model. However, a noteworthy finding is that the best results were most of the time reported by non-BERT initializations. Hence, well-known pre-trained word embeddings with a much lower

\begin{table}

\end{table}
Table 2: **Summary of key experimental results for Intuitive Graphs.** Accuracy and F1-macro are shown.
dimensionality yield better results than BERT embeddings. This is the case for the App Reviews and IMDB datasets using Word2Vec, and BBC News using GloVe.

### How Well do Graphs Adapt to Different Tasks?

For App Reviews, Table 2 reveals that \(\text{Window}_{\text{ext}}\) has better results than when using \(N=2\). In turn, a simplified sequence graph outperforms its weighted counterpart and the window-based constructions with an F1 score of 0.357. As these are extremely short documents, a simple model is powerful enough to address the task.

Although no significant differences are observed between the window-based graphs for DBpedia, the best accuracy is reported when \(N=2\), using only one neural layer and 16 hidden units. Thus, very few parameters are sufficient to attain competitive results. Likewise, this setting outperforms all GNNs trained on sequence graphs, demonstrating that the interaction between the left-right contexts for each graph node is beneficial for the task. Similarly, the window graph surpasses its extended version on IMDB. However, comparing the sequence graphs, we observe that the simplified construction achieves even better results. With just one neural layer, \(\text{Sequence}_{\text{simp}}\) achieves 87.9% accuracy.

For BBC News, Table 2 shows that as the number of convolutional layers and the number of hidden units increase, the performance of the models consistently improves, with \(\text{Window}_{\text{ext}}\) being the best graph construction at \(98\%\) accuracy. This representation establishes associations between different text parts by defining more extended contexts than using a window size of \(2\).

Since news articles tend to reiterate key facts in the news multiple times, exact co-occurrences of word pairs appear to be more frequent than in other domains. Accordingly, sequence graphs with edge weights yield better results than simplified ones. Despite this, they do not surpass the results of \(\text{Window}_{\text{ext}}\).

The contrary is observed for HND. The window-based graph achieves better results than its extended version, and \(\text{Sequence}_{\text{simp}}\) significantly outperforms the weighted version, which fails to learn the task. This is because, although these are extremely long documents, many of the tokens correspond to HTML, PHP markers, and other artifacts from the source. Therefore, much of the input is insignificant for solving the task. Given that a window-based construction produces more strongly connected graphs as \(N\) grows, \(\text{Window}_{\text{ext}}\) obtains worse results. However, directed graphs are preferable, as they restrict the flow of information between neighbors in a single direction, relieving message passing between insignificant nodes. This only occurs for the simplified version, since including the number of co-occurrences of said tokens would increase the effect of noise tokens.

The results suggest that the construction of text graphs, although simple in terms of methodology, is fairly sensitive to the information to be represented. Despite being considered trivial in many NLP tasks, preprocessing plays a crucial role in this field. Since GNNs usually incorporate convolutional layers to spread messages between neighbors, they require valuable nodes and edges, and a good vector initialization to solve the target task.

### General Prediction Quality

The BERT results show that the textual information in the documents is sufficient to outperform random prediction on the DBpedia, IMDB, and BBC News datasets. This demonstrates the low difficulty involved in these tasks. On the contrary, the most challenging ones correspond to those having document lengths longer than 512 or showing class imbalance (App Reviews and HND datasets).

Despite not outperforming FF BERT, TextLevelGCN is the best graph-based model for App Reviews. It shows that the classification task benefits from edge weights, but they should be soft values for a smoother learning curve. Otherwise, it is preferable to omit them. On all other datasets, TextLevelGCN underperforms Intuitive Graphs even when processing short documents. Moreover, despite its simplicity, \(\text{Sequence}_{\text{simp}}\) outperforms TextLevelGCN.

We see that the results of graph-based models are below FF BERT for short document classification. However, for long documents such as BBC News, \(\text{Window}_{\text{ext}}\) obtains the best-reported accuracy. The same is observed for HND, where Intuitive Graphs dominate as the best way to represent longer documents. In this task, FF BERT obtains significantly better validation performance than during testing, i.e., it overfits and obtains considerably lower results than TextLevelGCN. Although the latter defines a much smaller maximum length for text documents (350 tokens), TextLevelGCN has a higher
accuracy than FF BERT. Therefore, graph-based document representations provide clear advantages when processing long texts.

### Discussion

The results show that graph-based document representation holds substantial promise as a way of providing structural information to deep neural networks. Graph-based learning models are powerful and allow the extraction of complex patterns from text. However, they are particularly task-sensitive and depend on the lexical features of the documents to be represented. Thus, special care must be taken to properly define the components of the structure (nodes, edges, and the similarity function as edge label). Despite this, the most simplistic graph constructions (such as \(\text{Sequence}_{\text{simp}}\)) can address text classification fairly well, proving competitive even in challenging scenarios such as with data imbalance and noisy documents (i.e., a large presence of insignificant tokens). This is possible because even when a junk token occurs numerous times in the text, it will be represented as a unique node. By omitting edge weights, the learning model is prevented from focusing on these issues and thus learns the task efficiently and effectively. This phenomenon can be seen in Figure 2, where the Intuitive Graphs methods require much less time and computing resources than competitors for training.

Since FF BERT is a highly complex model with respect to the number of learnable parameters, a higher execution time than graph-based models is expected. However, it is observed that for shorter documents, as in App Reviews and DBpedia, FF BERT took even longer than for medium-length documents. This suggests that this model requires several iterations to converge on these particular tasks. Beyond this, note the abrupt decrease in the execution time of the models on the BBC and HND datasets is because these datasets have a small number of samples. Therefore, the total runtime is much shorter compared to the others. See Appendix B for more details on the runtime.

## 5 Conclusion

We present an empirical analysis of graph-based text representation strategies for text classification

\begin{table}
\begin{tabular}{l l|c|c|c|c} \cline{3-6}  & & \multicolumn{2}{c|}{Validation} & \multicolumn{2}{c}{Testing} \\ \hline
**Dataset** & **Model** & **Acc** & **F1-ma** & **Acc** & **F1-ma** \\ \hline \multirow{4}{*}{App Reviews} & BERT & 0.615 \(\pm\) 0.005 & 0.180 \(\pm\) 0.014 & 0.613 \(\pm\) 0.004 & 0.184 \(\pm\) 0.012 \\  & FF BERT & 0.636 \(\pm\) 0.013 & **0.374 \(\pm\) 0.012** & 0.620 \(\pm\) 0.012 & **0.369 \(\pm\) 0.011** \\  & TextLevelGCN & 0.612 \(\pm\) 0.006 & 0.336 \(\pm\) 0.013 & **0.645 \(\pm\) 0.012** & 0.358 \(\pm\) 0.010 \\  & \(\text{Sequence}_{\text{simp}}\) & **0.640 \(\pm\) 0.010** & 0.334 \(\pm\) 0.016 & 0.637 \(\pm\) 0.007 & 0.357 \(\pm\) 0.013 \\ \hline \multirow{4}{*}{DBpedia} & BERT & 0.725 \(\pm\) 0.016 & 0.681 \(\pm\) 0.021 & 0.746 \(\pm\) 0.019 & 0.741 \(\pm\) 0.021 \\  & FF BERT & **0.984 \(\pm\) 0.003** & **0.977 \(\pm\) 0.004** & **0.983 \(\pm\) 0.001** & **0.983 \(\pm\) 0.001** \\  & TextLevelGCN & 0.964 \(\pm\) 0.003 & 0.964 \(\pm\) 0.003 & 0.961 \(\pm\) 0.001 & 0.960 \(\pm\) 0.002 \\  & Window & 0.968 \(\pm\) 0.001 & 0.967 \(\pm\) 0.001 & 0.975 \(\pm\) 0.001 & 0.974 \(\pm\) 0.001 \\ \hline \multirow{4}{*}{IMDB} & BERT & 0.682 \(\pm\) 0.025 & 0.669 \(\pm\) 0.034 & 0.686 \(\pm\) 0.029 & 0.677 \(\pm\) 0.036 \\  & FF BERT & **0.869 \(\pm\) 0.009** & **0.866 \(\pm\) 0.009** & **0.884 \(\pm\) 0.007** & **0.884 \(\pm\) 0.008** \\  & TextLevelGCN & 0.864 \(\pm\) 0.005 & 0.864 \(\pm\) 0.005 & 0.868 \(\pm\) 0.002 & 0.868 \(\pm\) 0.003 \\  & \(\text{Sequence}_{\text{simp}}\) & 0.863 \(\pm\) 0.005 & 0.861 \(\pm\) 0.005 & 0.879 \(\pm\) 0.001 & 0.879 \(\pm\) 0.001 \\ \hline \multirow{4}{*}{BBC News} & BERT & 0.594 \(\pm\) 0.054 & 0.553 \(\pm\) 0.089 & 0.525 \(\pm\) 0.056 & 0.493 \(\pm\) 0.073 \\  & FF BERT & 0.967 \(\pm\) 0.011 & 0.960 \(\pm\) 0.013 & 0.978 \(\pm\) 0.003 & 0.977 \(\pm\) 0.003 \\  & TextLevelGCN & 0.962 \(\pm\) 0.005 & 0.963 \(\pm\) 0.004 & 0.973 \(\pm\) 0.004 & 0.973 \(\pm\) 0.004 \\  & Window\({}_{\text{ext}}\) & **0.977 \(\pm\) 0.009** & **0.976 \(\pm\) 0.009** & **0.980 \(\pm\) 0.003** & **0.980 \(\pm\) 0.003** \\ \hline \multirow{4}{*}{HND} & BERT & 0.559 \(\pm\) 0.162 & 0.346 \(\pm\) 0.076 & 0.522 \(\pm\) 0.059 & 0.345 \(\pm\) 0.027 \\  & FF BERT & **0.785 \(\pm\) 0.023** & 0.674 \(\pm\) 0.034 & 0.726 \(\pm\) 0.029 & 0.706 \(\pm\) 0.045 \\ \cline{1-1}  & TextLevelGCN & 0.759 \(\pm\) 0.035 & **0.721 \(\pm\) 0.030** & 0.757 \(\pm\) 0.026 & 0.734 \(\pm\) 0.035 \\ \cline{1-1}  & \(\text{Sequence}_{\text{simp}}\) & 0.735 \(\pm\) 0.015 & 0.687 \(\pm\) 0.017 & **0.791 \(\pm\) 0.011** & **0.785 \(\pm\) 0.011** \\ \hline \end{tabular}
\end{table}
Table 3: **General effectiveness.** The results of the best Intuitive Graph, TextLevelGCN, and BERT baselines are reported for the validation and testing partition.

Figure 2: **Execution time.** Average execution time and shaded standard deviation for each classifier and dataset.
by comprehensively analyzing their effectiveness across several GNN architectures and setups. The experiments consider a heterogeneous set of five different datasets, encompassing both short and long documents. The results show that the strength of graph-based models is closely tied to the textual features and the source domain of documents. Thus, the choice of nodes and edges is found to be crucial. Despite this, sequence graphs are shown to be a strong option, reaching competitive results across all considered tasks, especially for longer documents, and exceeding those of BERT. Along with this, we observed that pre-trained static word embeddings, instead of BERT vectors, allow reaching outstanding results on some tasks.

Although predefined graph construction methods have obtained good results lately, approaches for learning the graph structure may be desirable. This would eliminate the need for picking a design manually, being less domain-dependent. However, this would come at the price of diminished explainability. Given the capacities of current deep learning models, a graph generation strategy is prone to turning into a black-box model.

In future work, we will expand this study with other graph representation methods for further NLP tasks, such as summarization and question answering, since they highly rely on context understanding for solving the task.

### Limitations

While this study successfully shows the impact and potential of graphs for document representation, there are some points to keep in mind.

First, despite all the judgments and conclusions presented being supported by the results of the experiments, they were based on graph neural network models trained on particular sub-partitions, as stated in Section 3.3.1, so as to allow a fairer comparison between models. However, this means that the results reported here are not directly comparable with those reported in the literature. To assess how the models are positioned with regard to the State-of-the-Art in the different tasks, it is advisable to train on the original training partitions and thus learn from all the available data.

It is also important to note that our study analyzes multiple text representation strategies on text classification only. Although this is one of the most important classes of NLP tasks, we cannot ensure that such graph approaches show the same behavior in other tasks. Therefore, tackling other types of problems that require a deep level of understanding of the local and global context of the text is an important direction for future work.

Furthermore, all the experiments were run on English data. As English has comparatively simple grammar and well-known rules for conjugations and plurals, it is possible that graph-based models may not be as effective in other languages. Analyzing this aspect would be particularly interesting for low-resource languages.

Finally, the comparison with BERT does not seem fair, since it requires a cut-off for documents with more than 512 tokens and thus is unable to access the entirety of the text. We included BERT to analyze its effects and compare the results with TextLevelGCN, which also cuts text to an even smaller length than BERT. In future work, modifications of the BERT and TextLevelGCN models, where both access all tokens contained in the document, could be investigated.

## Ethics Statement

This work studies fundamental questions that can be invoked in a multitude of different application contexts. Different applications entail different ethical considerations that need to be accounted for before deploying graph-based representations. For instance, applying a trained hyperpartisan news detection model in an automated manner bears the risk of false positives, where legitimate articles get flagged merely for a choice of words that happens to share some resemblance with words occurring in hyperpartisan posts. For sentiment classification, Mohammad (2022) provides an extensive discussion of important concerns. Hence, ethical risks need to be considered depending on the relevant target use case.

## Acknowledgments

This study was possible due to the funding of the Data Science and Engineering (DSE) Research School program at Hasso Plattner Institute.
