# GraphFormers: GNN-nested Transformers for Representation Learning on Textual
Graph

Junhan Yang${}^{\textbf{{\tiny\char 169}}}$, Zheng Liu${}^{\textbf{{\tiny\char
168}}}$, Shitao Xiao${}^{\textbf{{\tiny\char 171}}}$, Chaozhuo
Li${}^{\textbf{{\tiny\char 168}}}$, Defu Lian${}^{\textbf{{\tiny\char 169}}}$,
Sanjay Agrawal${}^{\textbf{{\tiny\char 170}}}$, Amit
Singh${}^{\textbf{{\tiny\char 170}}}$, Guangzhong Sun${}^{\textbf{{\tiny\char
169}}}$, Xing Xie${}^{\textbf{{\tiny\char 168}}}$
♣ Microsoft Research Asia, Beijing, China
♥ Microsoft India Development Center , Bengaluru, India
♠ University of Science and Technology of China, Hefei, China
♠ Beijing University of Posts and Telecommunications, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work was done by 2021.01 during Junhan Yang and
Shitao Xiao’s internship in MSRAProject owner and co-first author

###### Abstract

The representation learning on textual graph is to generate low-dimensional
embeddings for the nodes based on the individual textual features and the
neighbourhood information. Recent breakthroughs on pretrained language models
and graph neural networks push forward the development of corresponding
techniques. The existing works mainly rely on the cascaded model architecture:
the textual features of nodes are independently encoded by language models at
first; the textual embeddings are aggregated by graph neural networks
afterwards. However, the above architecture is limited due to the independent
modeling of textual features. In this work, we propose GraphFormers, where
layerwise GNN components are nested alongside the transformer blocks of
language models. With the proposed architecture, the text encoding and the
graph aggregation are fused into an iterative workflow, making each node’s
semantic accurately comprehended from the global perspective. In addition, a
progressive learning strategy is introduced, where the model is successively
trained on manipulated data and original data to reinforce its capability of
integrating information on graph. Extensive evaluations are conducted on three
large-scale benchmark datasets, where GraphFormers outperform the SOTA
baselines with comparable running efficiency.

##  1 Introduction

The textual graph is a widely existed data format, where each node is
annotated with its textual feature. The representation learning on textual
graph is to generate low-dimensional node embeddings based on the individual
textual features and the information from the neighbourhood. In recent years,
the breakthroughs in pretrained language models and graph neural networks
contribute to the development of corresponding techniques. Particularly, with
pretrained language models, such as BERT (Devlin et al., 2018) and RoBERTa
(Liu et al., 2019a), the underlying semantics of texts can be captured more
precisely; at the same time, with graph neural networks, like GraphSage
(Hamilton et al., 2017a) and GAT (Veličković et al., 2018), neighbours can be
effectively aggregated for more informative node embeddings. It is necessary
to combine both techniques for better textual graph representation. As
suggested by GraphSage (Hamilton et al., 2017a) and PinSage (Ying et al.,
2018), the textual feature can be independently modeled by text encoders and
further aggregated by rear-mounted GNNs for the final node embeddings. Such a
representation paradigm has been widely adopted by subsequent works on various
scenarios (Zhu et al., 2021; Li et al., 2021; Hu et al., 2020; Liu et al.,
2019b; Zhou et al., 2019), where GNNs are combined with powerful PLM-based
text encoders.

Figure 1: Model architecture comparison (a center node C is connected with two
neighbours N1, N2). (A) Cascaded Transformers-GNN: text embeddings are
independently generated by language models and aggregated by rear-mounted
GNNs. (B) GNN-nested Transformers: the text encoding and graph aggregation are
iteratively performed with the layerwise GNNs and Transformers (TRM).

The above way of combination is called the “Cascaded Transformers-GNN”
architecture (Figure 1 A), as the language models (built upon Transformers)
are deployed ahead of the GNN component. With the above architecture, the text
encoding and the graph aggregation are performed in two consecutive steps,
where there is no information exchange between the nodes when text embeddings
are generated. However, the above workflow is defective considering that the
linked nodes are correlated, whose underlying semantics can be mutually
enhanced. For example, given a node “notes on transformers” and its neighbour
“tutorials on machine translation”; by making reference to the whole context,
the “transformers” here can be interpreted as a machine learning model, rather
than an electric device.

Our Work. We propose “GNN-nested Transformers” (GraphFormers), which are
highlighted for the fusion of GNNs and language models (Figure 1 B). In
GraphFormers, the GNN components are nested alongside the transformer layers
(TRM) of language models, where the text encoding and graph aggregation are
fused as an iteratively workflow. In each iteration, the linked nodes will
exchange information with each other in the layerwise GNN component; thus,
each node will be augmented by its neighbourhood information. The transformer
component will work on the augmented node features, where increasingly
informative node representations can be generated for the next iteration.
Compared with the cascaded architecture, GraphFormers achieve more sufficient
utilization of the cross-node information on graph, which significantly
benefit the representation quality. Given that the layerwise GNN components
merely involve simple and effective multi-head attention, GraphFormers
preserve comparable running costs as the existing cascaded Transformers-GNN
models.

On top of the proposed model architecture, we further improve GraphFormers’
representation quality and practicability as follows. Firstly, the training of
GraphFormers is likely to be shortcut: in many cases, the center node itself
can be “sufficiently informative”, where the training tasks can be
accomplished without leveraging the neighbourhood information. As such,
GraphFormers may end up with insufficiently trained GNNs. Inspired by recent
success of curriculum learning (Bengio et al., 2009), we propose to train the
model progressively: the first round of training is performed with manipulated
data, where the nodes are randomly polluted; thus, it becomes harder to make
prediction merely rely on the center nodes, and the model will be forced to
leverage the whole input nodes. The second round of training gets back to the
unpolluted data, where the model will be fit into the targeted distribution.
Another concern about GraphFormers is that all the linked nodes are mutually
dependent in the representation process: once a new node is presented, all the
neighbours, regardless of whether they have been processed before, need to be
encoded from scratch. As a result, a great deal of unnecessary computations
will be incurred. We introduce unidirectional graph attention to alleviate
this problem: only the center node is required to make reference to the
neighbours, while the neighbour nodes remain independently encoded. By this
means, the existing neighbours’ encoding results can be cached and reused,
which significantly saves the computation cost.

Extensive evaluations are conducted with three million-scale textual graph
datasets: DBLP, Wiki and Product Ads. According to our experiment results,
GraphFormers significantly outperform the SOTA cascaded Transformers-GNN
baselines with comparable running efficiency.

##  2 Related Work

The textual graph representation is an important research topic in multiple
areas, such as natural language processing, information retrieval and graph
learning (Yang et al., 2015; Wang et al., 2016b, a; Yasunaga et al., 2017;
Wang et al., 2019a; Xu et al., 2019). To learn high-quality representation for
textual graph, techniques on natural language understanding and graph
representation need to be jointly leveraged. In recent years, breakthroughs on
pretrained language models (PLM) and graph neural networks (GNN) significantly
advance the development of corresponding techniques.

PLM. The PLMs are proposed to learn universal language models with neural
networks trained on large-scale corpus. The early works were based on shallow
networks, e.g, word embeddings learned by Skip-Gram (Mikolov et al., 2013) and
GloVe (Pennington et al., 2014). In recent years, the backbone networks are
being quickly scaled up: from EMLo (Peters et al., 2018), GPT (Radford et al.,
2018), to BERT (Devlin et al., 2018), XLNet (Yang et al., 2019), T5 (Raffel et
al., 2019), GPT-3 (Brown et al., 2020). The large-scale models, which get
fully trained with massive data, demonstrate superior performances on general
NLP tasks. One of the most critical usages of PLMs is text representation,
where the underlying semantics of texts are captured by low-dimensional
embeddings. Such embeddings achieve competitive results on downstream tasks,
like text retrieval and classification (Reimers and Gurevych, 2019; Luan et
al., 2020; Gao et al., 2021; Su et al., 2021).

GNN. Graph neural networks are recognized as powerful tools of modeling graph
data (Hamilton et al., 2017b; Zhou et al., 2020). Such methods (e.g., GCN
(Kipf and Welling, 2016), GAT (Veličković et al., 2018), GraphSage (Hamilton
et al., 2017a)) learn effective message passing mechanisms such that
information between the nodes can get aggregated for expressive graph
representations.

Graph neural networks may also incorporate node attributes, like texts; and
it’s quite straightforward to leverage GNNs and PLMs for textual graph
representation following the “cascaded architecture” suggested by GraphSage
(Hamilton et al., 2017a): the node features are independently encoded at
first; then, the node embeddings are aggregated via GNNs to generate the final
representations. Such a representation paradigm is widely adopted by
subsequent works (Zhu et al., 2021; Li et al., 2021; Hu et al., 2020; Liu et
al., 2019b; Zhou et al., 2019). However, the above approaches treat the text
encoding and graph aggregation as two consecutive steps, where the node-level
features are independently processed. Our work is different from these
approaches as the text encoding and graph aggregation are fused as an
iterative workflow based on the “GNN-nested Transformers”.

##  3 GraphFormers

In this work, we deal with textual graph data, where each node $x$ is a text.
The node $x$ together with its neighbours $N_{x}$ are denoted as $G_{x}$. Our
model learns the embedding for node $x$ based on its own textual feature and
the information of its neighbourhood $N_{x}$. The generated embeddings are
expected to capture the relationship between the nodes, i.e., to accurately
predict whether two nodes $x_{q}$ and $x_{k}$ are connected based on the
embedding similarity.

###  3.1 GNN-nested Transformers

The encoding process of GraphFormers is indicated as follows. The input nodes
(the center node and its neighbours) are tokenized into sequences of tokens,
with special tokens [CLS] padded in the front, whose states are used for node
representation. The input sequences are mapped into the initial embedding
sequences $\\{\mathbf{H}^{0}_{g}\\}_{G}$ based on the summation of word
embeddings and position embeddings. The embedding sequences are encoded by
multiple layers of GNN-nested Transformers (shown as Figure 2), where the
graph aggregation and text encoding are iterative performed.

$\bullet$ Graph Aggregation in GNN. Each node is enhanced by its neighbourhood
information based on the layerwise graph aggregation. For each node in the
$l$-th layer, the first token-level embedding (corre-sponding to [CLS]) is
taken as the node-level embedding:
$\mathbf{z}^{l}_{g}\leftarrow\mathbf{H}^{l}_{g}[0]$. The node-level embeddings
are gathered from all the nodes and passed to the layerwise GNN for graph
aggregation. We leverage Multi-Head Attention (MHA) to encode the node-level
embeddings $\mathbf{Z}^{l}_{G}$ ($\\{\mathbf{z}^{l}_{g}\\}_{G}$), similar as
GAT (Veličković et al., 2018). For each attention head, the scaled dot-product
attention is performed as:

$\begin{gathered}\mathbf{\hat{Z}}^{l}_{G}=\mathrm{MHA}(\mathbf{Z}^{l}_{G});\\\
\mathrm{MHA}(\mathbf{Z}^{l}_{G})=\mathrm{Concat}(\mathbf{head}_{1},...,\mathbf{head}_{h});\\\
\mathbf{head}_{j}=\mathrm{softmax}(\frac{\mathbf{QK^{T}}}{\sqrt{d}}+\mathbf{b})\mathbf{V};\\\
\mathbf{Q}=\mathbf{Z}^{l}_{G}\mathbf{W}_{j}^{Q};~{}\mathbf{K}=\mathbf{Z}^{l}_{G}\mathbf{W}_{j}^{K};~{}\mathbf{V}=\mathbf{Z}^{l}_{G}\mathbf{W}_{j}^{V};\end{gathered}$
(1)

In the above equations, $\mathbf{W}_{j}^{Q}$, $\mathbf{W}_{j}^{K}$, and
$\mathbf{W}_{j}^{V}$ are the projection matrices of MHA, corresponding to the
$j$-th attention head. A learnable position bias $\mathbf{b}$ is added to the
dot-product result; the positions differentiate the relationship between the
nodes; i.e., “center-to-center” ($x$ to $x$), “center-to-neighbour” ($x$ to
$N_{x}$), and “neighbour-to-neighbour” ($N_{x}$ to $N_{x}$), respectively.

Each of the embeddings $\mathbf{\hat{z}}^{l}_{g}$
($\mathbf{\hat{z}}^{l}_{g}\in\mathbf{\hat{Z}}^{l}_{G}$) is dispatched to its
original node and concatenated ($\oplus$) with the token-level embeddings,
which gives rise to the graph-augmented token-level embeddings:

$\widehat{\mathbf{H}}^{l}_{g}\leftarrow\mathrm{Concat}(\mathbf{\hat{z}}^{l}_{g},\mathbf{H}^{l}_{g}).$
(2)

In this place, the GNN-processed node-level embeddings
$\mathbf{\hat{Z}}^{l}_{G}$ can be interpreted as “messagers”, with which the
neighbourhood information can be introduced to each of the nodes.

Figure 2: GNN-nested Transformers (using the $l$-th layer for illustration).
The graph aggregation is performed in the first place: the node-level
embeddings $\\{\mathbf{z}^{l}_{g}\\}_{G}$ are gathered from all the nodes and
processed by the GNN component (the leftmost rectangle). The GNN processed
node-level embeddings $\\{\mathbf{\hat{z}}^{l}_{g}\\}_{G}$ are dispatched to
their original nodes, which forms the graph-augmented token-level embeddings.
The graph-augmented token-level embeddings are further encoded by Transformer.

$\bullet$ Text Encoding in Transformer. The graph-augmented token-level
embeddings $\widehat{\mathbf{H}}^{l}_{g}$ are processed by the transformer
component (Vaswani et al., 2017), where the following computations are
performed:

$\begin{gathered}\widehat{\mathbf{H}}^{l}_{g}=\mathrm{LN}(\mathbf{{H}}^{l}_{g}+\mathrm{MHA}^{asy}(\widehat{\mathbf{H}}^{l}_{g}));\\\
\mathbf{H}^{l+1}_{g}=\mathrm{LN}(\widehat{\mathbf{H}}^{l}_{g}+\mathrm{MLP}(\widehat{\mathbf{H}}^{l}_{g})).\end{gathered}$
(3)

In the above equations, $\mathrm{MLP}$ is the Multi-Layer Projection unit, and
$\mathrm{LN}$ is the Layer-Norm unit. We use asymmetric Multi-Head Attention
($\mathrm{MHA}^{asy}$), where $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$ are
computed as:

$\begin{gathered}\mathbf{Q}=\mathbf{{H}}^{l}_{g}\mathbf{W}_{j}^{Q};~{}\mathbf{K}=\widehat{\mathbf{H}}^{l}_{g}\mathbf{W}_{j}^{K};~{}\mathbf{V}=\widehat{\mathbf{H}}^{l}_{g}\mathbf{W}_{j}^{V}.\end{gathered}$
(4)

Therefore, the output sequence $\mathbf{H}^{l+1}_{g}$ will be of the same
length as the input sequence $\mathbf{{H}}^{l}_{g}$. The encoding result will
be used as the input token-level embeddings for the next layer. The node-level
embedding at the last layer $\mathbf{z}^{L}_{x}$ (i.e.,
$\mathbf{{H}}^{L}_{g}[0]$) will be used as the final node representation.

$\bullet$ Workflow. We summarize GraphFormers’ encoding workflow as Algorithm
1. The initial token-level embeddings $\\{\mathbf{H}^{0}_{g}\\}_{G}$ are
independently encoded by the first Transformer layer $\mathrm{TRM}^{0}$. For a
$L$-layer GraphFormers, the graph aggregation and text encoding are
iteratively performed for the subsequent $L$-1 steps (from 1 to $L-1$). In
each step, the node-level embeddings $\mathbf{Z}^{l}_{G}$ are gathered and
processed by the layerwise GNN component. The output node-level embeddings
$\mathbf{\hat{Z}}^{l}_{G}$ are dispatched to their original nodes, which
generates the graph-augmented token-level embeddings
$\widehat{\mathbf{H}}^{l}_{g}$. The graph-augmented token-level embedding are
further processed by the Transformer component. Finally, The node-level
embedding (for the center node $x$) in the last layer $\mathbf{z}^{L}_{x}$ is
taken as our representation result.

$\bullet$ Encoding Complexity. Given an input of $M$ nodes, each one has $P$
tokens; the time complexity of each layer’s encoding operation is
$O(M^{2}+MP^{2})$: the graph aggregation takes $O(M^{2})$, because $M$ node-
level embeddings are gathered for multi-head attention; the text encoding
takes $O(MP^{2})$, as each of the $M$ node calls for the multi-head attention
of $P$ tokens. Compared with Transformers, the GNN’s computation cost is much
smaller, mainly because of two reasons: 1) $M^{2}\ll MP^{2}$ in general, 2)
operations like MLP are not needed in graph aggregation. Therefore, the
working efficiency of GraphFormers is close to the cascaded GNN-Transformers
as the extra computation cost of layerwise graph aggregation is relatively
small. Such a property is also empirically verified in our experiment.

Input: The input graphs $G$ (consist of the center node $x$ and its
neighbours).

Output: The embedding for the center node $\mathbf{h}_{x}$.

1 begin

2 for  _each text $g\in G$_ do

3 $\mathbf{H}^{1}_{g}\leftarrow\mathrm{TRM}^{0}(\mathbf{H}^{0}_{g})$; // Get
the initial token-level embeddings

4

5 for  _$l=1,...,L-1$_ do

6 $\mathbf{Z}^{l}_{G}\leftarrow\\{\mathbf{z}^{l}_{g}|g{\in}G\\}$; // Gather
node-level embeddings to GNN

7 $\mathbf{\hat{Z}}^{l}_{G}\leftarrow\mathrm{GNN}(\mathbf{{Z}}^{l}_{G})$; //
Graph aggregation in GNN

8 for  _each text $g\in G$_ do

9
$\widehat{\mathbf{H}}^{l}_{g}\leftarrow\mathrm{Concat}(\mathbf{\hat{z}}^{l}_{g},\mathbf{H}^{l}_{g})$;
// Get graph-augmented token-level embeddings

10
$\mathbf{H}^{l+1}_{g}\leftarrow\mathrm{TRM}^{l}(\widehat{\mathbf{H}}^{l}_{g})$;
// Text encoding in Transformer

11

12

13 Return $\mathbf{h}_{x}\leftarrow\mathbf{z}_{x}^{L}$;

14

15

Algorithm 1 GraphFormers’ Workflow

###  3.2 Model Simplification: Unidirectional Graph Aggregation

One concern about GraphFormers is that the input nodes are mutually dependent
on each other during the encoding process. As a result, to generate the
embedding for a node, all the related nodes in its neighbourhood need to be
encoded from scratch, regardless of whether they have been processed before.
Such a property is unfavorable in practice as a great deal of unnecessary
computation cost might be incurred (i.e., a node will be repetitively encoded
every time it serves as a neighbour node). We leverage a simple but effective
simplification, the unidirectional graph aggregation, to address this problem.
Particularly, only the center node $x$ is required to make reference to the
neighbourhood; while the rest of nodes $N_{x}$ remain independently encoded
all by their own textual features:

$\mathbf{H}^{l+1}_{g}=\begin{cases}\mathrm{TRM}^{l}(\widehat{\mathbf{H}^{l}_{x}}),~{}g=x;\\\
\mathrm{TRM}^{l}(\mathbf{H}^{l}_{g}),~{}\forall g\in N_{x}.\end{cases}$ (5)

Because the encoding of the neighbour nodes is independent of the center node,
the intermediate encoding results $\\{\mathbf{z}^{1...L}_{g}\\}_{N_{x}}$ can
be cached in storage111The encoding results can be kept in low-cost devices,
whose storage capacity can be regarded as infinite. and reused in subsequent
computations when they are needed. As a result, the nodes can be prevented
from being encoded repetitively, which saves a great deal of unnecessary
computation cost. We empirically verify that GraphFormers maintain similar
performances when the above simplification is introduced.

###  3.3 Model Training: Two-Stage Progressive Learning

$\bullet$ Training Objective. We take advantage of link prediction as our
training task. Given a pair of nodes $q$ and $k$, the model is learned to
predict whether they are connected based on their embedding similarity.
Particularly, the following classification loss is minimized for a positive
pair of $q$ and $k$:222We remove the naive cases where $q$ and $k$ are
included by each other’s neighbour set, $N_{q}$ and $N_{k}$.

$\begin{gathered}\mathcal{L}=-\log\frac{\exp(\langle\mathbf{h}_{q},\mathbf{h}_{k}\rangle)}{\exp(\langle\mathbf{h}_{q},\mathbf{h}_{k}\rangle)+\sum_{r\in
R}\exp(\langle\mathbf{h}_{q},\mathbf{h}_{r}\rangle)}.\end{gathered}$ (6)

In the above equation, $\mathbf{h}_{q}$ and $\mathbf{h}_{k}$ are the node
embeddings; $\langle\cdot\rangle$ denotes the computation of inner product;
$R$ stands for the negative samples. In our implementation, we leverage “in-
batch negative samples” (Karpukhin et al., 2020; Luan et al., 2020) for the
reduction of encoding cost: a positive sample in one training instance will be
used as a negative sample in the rest of the training instances within the
same mini-batch.

$\bullet$ Two-stage Training. In GraphFormers, the information from the center
node and neighbour nodes are not treated equally, which may undermine the
model’s training effect. Particularly, the center node’s information can be
directly utilized, while the neighbourhood information needs to be introduced
via three steps: 1) encoded as node-level embeddings, 2) making graph
aggregation with the center node, and 3) introduced to center node’s graph
augmented token-level embeddings. The message passing pathway can shortcut
when the center nodes are “sufficiently informative”, i.e., two nodes are
sufficiently similar with each other in terms of their own textual features,
such that their connection can be predicted without considering the
neighbours. Given the existence of such cases, GraphFormers may end up with
well-trained Transformers but insufficiently trained GNNs.

To alleviate the above problem, we introduce a warm-up training task, where
the link prediction is made based on the polluted input nodes. Particularly,
for each input node $g$, a subset of its tokens $g_{m}$ will be randomly
masked333We use the common MLM strategy, where 15% of the input tokens are
masked: 80% of them are replaced by [MASK], the rest ones are replaced
randomly or kept as the original tokens with the same probabilities.. As a
result, the classification loss becomes:

$\begin{gathered}\mathcal{L}^{\prime}=-\log\frac{\exp(\langle\mathbf{h}_{\tilde{q}},\mathbf{h}_{\tilde{k}}\rangle)}{\exp(\langle\mathbf{h}_{\tilde{q}},\mathbf{h}_{\tilde{k}}\rangle)+\sum_{r\in
R}\exp(\langle\mathbf{h}_{\tilde{q}},\mathbf{h}_{\tilde{r}}\rangle)},\end{gathered}$
(7)

where $\mathbf{h}_{\tilde{q}}$, $\mathbf{h}_{\tilde{k}}$,
$\mathbf{h}_{\tilde{r}}$ are the embeddings generated from the polluted nodes.
The masked tokens reduce the informativeness of each individual node;
therefore, the model is forced to leverage of the whole input nodes to make
the right prediction.

Finally, the model training is organized as a two-stage progressive learning
process. In the first stage, the model is trained to minimize
$\mathcal{L}^{\prime}$ based on the polluted nodes until its convergence,
which reinforce the model’s capability of integrating information on graph. In
the second stage, the model is continually trained to minimize $\mathcal{L}$
based on the original data until the convergence, which makes the model fit
into the target distribution.

##  4 Experimental Studies

###  4.1 Data and Settings

We make use of the following three real-world textual graph datasets for our
experimental studies.

Table 1: Specifications of the experimental datasets: the number of items, the
number of neighbour nodes on average, and the number of training, validation,
testing cases.
---
|  Product |  DBLP |  Wiki
#Item |  5,643,688 |  4,894,081 |  4,818,679
#N |  4.71 |  9.31 |  8.86
#Train |  22,146,934 |  3,009,506 |  7,145,834
#Valid |  30,000 |  60,000 |  66,167
#Test |  306,742 |  100,000 |  100,000

$\bullet$ DBLP444https://originalstatic.aminer.cn/misc/dblp.v12.7z, which
contains the paper citation graph from DBLP up to 2020-04-09. Two papers are
linked if one is cited by the other one. The paper’s title is used as the
textual feature.

$\bullet$ Wikidata5M555https://deepgraphlearning.github.io/project/wikidata5m
(Wiki) (Wang et al., 2019b), which contains the entity graph from Wikipedia.
The first sentence in each entity’s introduction is taken as its textual
feature.

$\bullet$ Product Graph (Product), an even larger dataset of online products
collected by a world-wide search engine. In this dataset, the users’ web
browsing behaviors are tracked for the targeted product webpages (e.g., Amazon
webpages of Nike shoes). The user’s continuously browsed webpages within a
short period of time (e.g., 30 minutes) is called a “session”. The products
within a common session are connected in the graph (which is a common way of
graph construction in e-commerce scenarios (Ying et al., 2018; Wang et al.,
2018)). Each product has its unique textual description, which specifies
information like the product name, brand, and saler, etc.

The textual features of all the datasets are in English. We make use of
uncased WordPiece (Wu et al., 2016) to tokenize the input text. In our
experiment, each text is associated with 5 uniformly sampled neighbours
(without replacement); for texts with neighbourhood smaller than 5, all the
neighbours will be utilized. We summarized the specifications of all the
datasets with Table 1. The experiment results are evaluated in terms of link
prediction accuracy, i.e., to predict whether a query node and key node are
connected given the textual features of themselves and their neighbours. In
each testing instance, one query is provided with 300 keys: 1 positive plus
299 randomly sampled negative cases. We leverage three common metrics to
measure the prediction accuracy: Precision@1, NDCG, and MRR. Without
specifications, we will take the unidirectional-simplified GraphFormers
trained with the two-stage progressive learning as our default model. More
details about the implementations and the training/testing configurations are
summarized in an Appendix file. It is submitted together with our source code
within the supplementary materials.

###  4.2 Baselines

We focus on the comparison between GNN-nested Transformers and Cascaded
Transformers-GNN. To make sure the difference between both architectures can
be truthfully reflected from the evaluation results, GraphFormers and the
Cascaded Transformers-GNN baselines are equipped with text encoders and graph
aggregators of the same capacities. Particularly, we use the BERT-like PLM as
our text encoder, where UniLM-base666An enhanced BERT-like PLM showing more
competitive performances than peers like RoBERTa, XLNet. (Bao et al., 2020) is
chosen as the network backbone for all related methods; the final layer’s
[CLS] token embedding is used for the text embedding.

We enumerate the following representative graph aggregators as used in GAT
(Veličković et al., 2018), GIN (Xu et al., 2018), GraphSage (Hamilton et al.,
2017a). The GAT aggregator, where the node embedding is generated as the
weighted sum of all the text embeddings. Each text embedding’s relative
importance is calculated as the attention score with the center node. The
Pooling-and-Concat aggregators, where the center node’s text embedding is
concatenated with the neighbours’ pooling result and linearly transformed for
the final representation. Depending on the form of pooling function, we have
the following options: Max and Mean, where neighbours are aggregated by max-
pooling and mean-pooling, respectively; Att, where the neighbours are summed
up based on the attention weights with the center node. By comparison, the
neighbourhood information may get more emphasized with GAT; while the center
node itself tends to be highlighted with Pooling-and-Concat.

We consider two more baselines which make use of simplified text encoders
(such as CNN) and network embeddings: TNVE (Wang et al., 2019a) and IFTN (Xu
et al., 2019). We also include the PLM only baseline, which merely leverages
the textual feature of the center node.

Table 2: Overall evaluation (GraphFormers marked in bold, the best baseline
underlined). GraphFormers outperforms all baselines, especially the ones based
on cascaded Transformers-GNN.
---
| Product | DBLP | Wiki
Methods |  P@1 |  NDCG |  MRR |  P@1 |  NDCG |  MRR |  P@1 |  NDCG |  MRR
PLM |  0.6563 |  0.7911 |  0.7344 |  0.5673 |  0.7484 |  0.6777 |  0.3466 |  0.5799 |  0.4712
TNVE |  0.4618 |  0.6204 |  0.5364 |  0.2978 |  0.5295 |  0.4163 |  0.1786 |  0.4274 |  0.2933
IFTN |  0.5233 |  0.6740 |  0.5982 |  0.3691 |  0.5798 |  0.4773 |  0.1838 |  0.4276 |  0.2945
PLM+GAT |  0.7540 |  0.8637 |  0.8232 |  0.6633 |  0.8204 |  0.7667 |  0.3006 |  0.5430 |  0.4270
PLM+Max |  0.7570 |  0.8678 |  0.8280 |  0.6934 |  0.8386 |  0.7900 |  0.3712 |  0.6071 |  0.5022
PLM+Mean |  0.7550 |  0.8671 |  0.8271 |  0.6896 |  0.8359 |  0.7866 |  0.3664 |  0.6037 |  0.4980
PLM+Att |  0.7513 |  0.8652 |  0.8246 |  0.6910 |  0.8366 |  0.7875 |  0.3709 |  0.6067 |  0.5018
GraphFormers |  0.7786 |  0.8793 |  0.8430 |  0.7267 |  0.8565 |  0.8133 |  0.3952 |  0.6230 |  0.5220
Table 3: Impact of neighbour size (#N).
---
| GraphFormers | PLM+Max
#N |  P@1 |  NDCG |  MRR |  P@1 |  NDCG |  MRR
1 |  0.6485 |  0.8087 |  0.7522 |  0.6249 |  0.7946 |  0.7342
2 |  0.6841 |  0.8308 |  0.7804 |  0.6538 |  0.8137 |  0.7583
3 |  0.6980 |  0.8396 |  0.7916 |  0.6728 |  0.8256 |  0.7734
4 |  0.7126 |  0.8485 |  0.8029 |  0.6823 |  0.8319 |  0.7814
5 |  0.7267 |  0.8565 |  0.8133 |  0.6934 |  0.8386 |  0.7900

###  4.3 Overall Evaluation

The overall evaluation results are reported in Table 2. It’s observed that
GraphFormers consistently outperform all the baselines, especially the ones
based on the cascaded Transformers-GNN, with notable advantages. Particularly,
it achieves 2.9%, 4.8%, 6.5% relative improvements over the most competitive
baselines (underlined) on each of the experimental datasets. Such an
observation indicates that the relationship between the nodes can be captured
more accurately based on the node embeddings generated by GraphFormers, which
verifies the effectiveness of our proposed method.

We also observe the following underlying factors that may influence the
representation quality.

Firstly, the effective utilization of neighbourhood information is critical.
With the joint consideration of the center node and neighbour nodes, the
PLM+GNNs methods, including GraphFormers and the cascaded Transformers-GNN
baselines, significantly outperform the PLM only baseline in most of the time.
We further analyze the impact of neighbourhood size as Table 3, with a
fraction of neighbour nodes randomly sampled for each center node (using DBLP
for illustration). It can be observed that both GraphFormers and PLM+Max (the
most competitive baseline) achieve higher prediction accuracy than the PLM
only method (P@1:0.5673, NDCG:0.7484, MRR:0.6777, as reported in Table 2),
even with fewer neighbour nodes included. With the increasing number of
neighbour nodes, the advantages become gradually enlarged. However, the
marginal gain is varnishing, as the relative improvement becomes smaller when
more neighbours are included. In all the testing cases, GraphFormers maintain
consistent advantages over PLM+Max, which reaffirms the effectiveness of our
proposed methods.

Secondly, the capacity of the text encoder is crucial for textual graph
representation. All the pretrained language model based methods (GraphFormers,
Cascaded Transformers-GNN baselines, PLM-only baseline) significantly
outperform the baselines with simplified text encoders (TNVE, IFTN).

Thirdly, the representation quality is also sensitive to the form of graph
aggregator. In Product, the cascaded Transformers-GNN baselines’ performances
are quite close to each other. In DBLP, PLM+(Max, Mean, Att) outperforms
PLM+GAT. In Wiki, not only PLM+(Max, Mean, Att) but also PLM-only baseline
outperform PLM+GAT. Such phenomenons could be attribute to the type of graph:
whether it is homogeneous or heterogeneous. Particularly, both Product and
DBLP can be regarded as homogeneous graphs as the nodes are connected based on
the same relationships; i.e., co-view relationship in Product, and citation
relationship in DBLP. In both homogeneous graphs, the connected nodes may have
quite similar semantics (the co-viewed products usually serve similar user
intents, and the citation relationships usually indicate similar research
topics); thus, the incorporated neighbour nodes will probably provide
complementary information for the link prediction between the center nodes.
However, Wiki is a heterogeneous graph, where the connections between entities
may have highly different semantics. As a result, the incorporation of
neighbour nodes may not contribute to the link prediction task, especially
when the incorporated neighbours and the prediction target are connected to
the center nodes with totally different relationships. Considering that GAT
tends to focus more on the neighbourhood, its performance can be vulnerable in
such unfavorable situations. These findings suggest that the neighbourhood
information should be properly handled in case that the information of the
center node is wiped out.

Finally, we may conclude different methods’ utility in textual graph
representation: simplified text encoders $\prec$ PLMs $\prec$ Cascaded
Transformers-GNN $\prec$ GNNs-nested Transformers. Such findings are
consistent with our expectation that the precise modeling of individual
textual feature and the effective integration of neighbourhood information
will jointly contribute to high-quality textual graph representation.
GraphFormers enjoy the high expressiveness of PLMs and leverage layerwise
nested-GNNs to facilitate graph aggregation, which contributes to both of the
above perspectives.

Table 4: Ablation Studies (The top ablated methods are marked in bold;
“$\uparrow$”/“$\downarrow$”: the performance is increased/decreased compared
with the default setting). “-Progressive”: two-stage progressive learning
disabled; “-Simplified”: unidirectional simplification disabled; “-Shared
GNNs”: GNNs parameters are not shared across the layers; “-Position”: GNNs
learnable position bias disabled.
---
| Product | DBLP | Wiki
Methods |  P@1 |  NDCG |  MRR |  P@1 |  NDCG |  MRR |  P@1 |  NDCG |  MRR
GraphFormers |  0.7786 |  0.8793 |  0.8430 |  0.7267 |  0.8565 |  0.8133 |  0.3952 |  0.6230 |  0.5220
PLM+Max |  0.7570 |  0.8678 |  0.8280 |  0.6934 |  0.8386 |  0.7900 |  0.3712 |  0.6071 |  0.5022
\- Progressive |  0.7688 |  0.8751 |  0.8373 |  0.7096 |  0.8468 |  0.8007 |  0.3834 |  0.6155 |  0.5127
\- Simplified |  0.7795 $\uparrow$ |  0.8798 $\uparrow$ |  0.8436 $\uparrow$ |  0.7225 |  0.8542 |  0.8102 |  0.3923 |  0.6209 |  0.5195
\- Shared GNNs |  0.7788 |  0.8795 |  0.8433 |  0.7256 |  0.8558 |  0.8123 |  0.3945 $\downarrow$ |  0.6221 $\downarrow$ |  0.5211 $\downarrow$
\- Position |  0.7788 |  0.8795 |  0.8434 |  0.7276 $\uparrow$ |  0.8570 $\uparrow$ |  0.8139 $\uparrow$ |  0.3942 |  0.6222 |  0.5211

###  4.4 Ablation Studies

The ablation studies (as Table 4) are performed to clarify the following
issues: 1) the impact of two-stage progressive learning, and 2) the impact of
unidirectional-simplified GraphFormers.

Firstly, the two-stage progressive learning substantially improves
GraphFormers’ representation quality. Without such a training strategy
("-Progressive": training directly on the original data), the model’s
performance is decreased by 0.98%, 1.71%, and 1.18% in each of the datasets,
respectively.

Secondly, the performances between simplified and non-simplified
(“-Simplified”) GraphFormers are comparable. In fact, the necessity of graph
aggregation is not equivalent for the center node and the neighbour nodes:
since the center node is the one for representation, it is much more important
to ensure that the center node may extract complementary information from its
neighbours. The unidirectional-simplified GraphFormers maintain such a
property; thus, there is little impact on the final performances. Such a
finding affirms that we may safely leverage the simplified model to save the
cost of repetitively encoding the existing neighbours.

We make two additional ablation studies. “-Shared GNNs”: the GNNs parameters
sharing is disabled, where each layer maintains its own graph aggregator (by
default, the layerwise GNN components in GraphFormers share the same set of
parameters). “-Position”: the learnable position bias ($\mathbf{b}$ in Eq. 1)
is disabled in GNNs. We find that model’s performance is little affected from
the above changes.

###  4.5 Efficiency Analysis

Table 5: Time and memory costs per mini-batch for PLM+Max and GraphFormers,
with neighbour size increased from 3 to 200. GraphFormers achieve similar
efficiency and scalability as PLM+Max.
---
#N |  3 |  5 |  10 |  20 |  50 |  100 |  200
Time: PLM+Max |  60.29 ms |  93.41 ms |  161.40 ms |  295.92 ms |  684.16 ms |  1357.93 ms |  2706.35 ms
Time: GraphFormers |  63.95 ms |  97.19 ms |  170.16 ms |  306.12 ms |  714.32 ms |  1411.09 ms |  2801.67 ms
Mem: PLM+Max |  1.33 GiB |  1.39 GiB |  1.55 GiB |  1.82 GiB |  2.67 GiB |  4.09 GiB |  6.92 GiB
Mem: GraphFormers |  1.33 GiB |  1.39 GiB |  1.55 GiB |  1.83 GiB |  2.70 GiB |  4.28 GiB |  7.33 GiB

We compare the time efficiency between GNN-nested Transformers (GraphFormers)
and Cascaded Transformers+GNN (using PLM+Max for comparison). The evaluation
is made with a Nvidia P100 GPU. Each mini-batch contains 32 encoding
instances; each instance contains one center and #N neighbour nodes; the token
length of each node is 16. We report the average time and memory (GPU RAM)
costs per mini-batch as Table 5.

Firstly, the time and memory costs of both methods grow linearly with the
increment of neighbours. (There are overheads of time and memory costs. The
time cost overhead may come from CPU processing; while the memory cost
overhead is mainly due to the model parameters (Rajbhandari et al., 2020)). We
may approximately remove the overheads by deducting the time and memory costs
where #N=3). Such a finding is consistent with our theoretical analysis in
Section 3.1.

Secondly, the overall time and memory costs of GraphFormers are quite close to
PLM+Max. When the number of neighbour nodes is small, the differences between
both methods are almost ignorable. The differences become slightly larger when
more neighbour nodes are included, because the layerwise graph aggregations in
GraphFormers get increasingly time consuming. However, the differences are
still relatively small: merely around 3.5% of the overall running costs when
#N is increased to 200 (“#N=200” is already more than enough for most of the
real world scenarios).

Based on the above observations, we may conclude that GraphFormers are more
accurate, meanwhile equally efficient and scalable as the conventional
cascaded Transformer+GNNs.

###  4.6 Online A/B Test on Bing Search

GraphFormers has been deployed as one of the major ads retrieval algorithms on
Bing Search, and it achieves highly competitive performance against the
previous production system (the combination of a wide spectrum of semantic
representation algorithms, including large-scale PLMs and cascaded PLMs-GNNs).
Particularly, the primary objective of Ads service is to maximize the revenue
meanwhile increasing the user clicks. Therefore, the following three metrics
are taken as the major performance indicators:
RPM777https://support.google.com/adsense/answer/190515?hl=en (revenue per
thousand impressions), CY (click yield), and
CPC888https://support.google.com/google-ads/answer/116495?hl=en (cost per
click) . During our large-scale online A/B test, GraphFormers significantly
improves the overall RPM, CY, CPC by 1.87%, 0.96% and 0.91%, respectively. A
11-day performance snapshot is demonstrated as Figure 3; it can be observed
that in most of the time, all three metrics are significantly improved thanks
to the utilization of GraphFormers (the daily performance are measured based
on millions of impressions, thus having strong statistic significance).

Figure 3: Online A/B Test: the relative improvements of RPM, CY and CPC
against the last version of production system in Bing Search (green: positive;
blue: negative). In most of the time, all three performance indicators are
significantly improved thanks to the utilization of GraphFormers.

##  5 Conclusion

In this paper, we propose a novel model architecture GraphFormers for textual
graph representation. By having GNNs nested alongside each transformer layer
of the pretrained language model, the underlying semantic of each textual node
can be precisely captured and effectively integrated for high-quality textual
graph representation. On top of the fundamental architecture, we introduce the
two-stage progressive training strategy to further strengthen GraphFormers’
representation quality; we also simplify the model with the unidirectional
graph aggregation, which eliminates the unnecessary computation cost. The
experimental studies on three large-scale textual graph datasets verify the
effectiveness of our proposed methods, where GraphFormers notably outperform
the existing cascaded Transformer-GNNs methods with comparable running
efficiency and scalability.

## References

  * Bao et al. (2020) Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, et al. 2020.  Unilmv2: Pseudo-masked language models for unified language model pre-training.  In _International Conference on Machine Learning_ , pages 642–652. PMLR.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Hamilton et al. (2017a) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a.  Inductive representation learning on large graphs.  In _Advances in neural information processing systems_ , pages 1024–1034.
  * Hamilton et al. (2017b) William L Hamilton, Rex Ying, and Jure Leskovec. 2017b.  Representation learning on graphs: Methods and applications.  _arXiv preprint arXiv:1709.05584_.
  * Hu et al. (2020) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020.  Gpt-gnn: Generative pre-training of graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1857–1867.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Li et al. (2021) Chaozhuo Li, Bochen Pang, Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. 2021.  Adsgnn: Behavior-graph augmented relevance modeling in sponsored search.  _arXiv preprint arXiv:2104.12080_.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2019b.  Fine-grained fact verification with kernel graph attention network.  _arXiv preprint arXiv:1910.09796_.
  * Luan et al. (2020) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020.  Sparse, dense, and attentional representations for text retrieval.  _arXiv preprint arXiv:2005.00181_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _2014 EMNLP_ , pages 1532–1543.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rajbhandari et al. (2020) Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020.  Zero: Memory optimizations toward training trillion parameter models.  In _SC20: International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16. IEEE.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021.  Whitening sentence representations for better semantics and faster retrieval.  _arXiv preprint arXiv:2103.15316_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks.  _International Conference on Learning Representations (ICLR)_.
  * Wang et al. (2016a) Chenguang Wang, Yangqiu Song, Haoran Li, Ming Zhang, and Jiawei Han. 2016a.  Text classification with heterogeneous information network kernels.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30.
  * Wang et al. (2018) Jizhe Wang, Pipei Huang, Huan Zhao, Zhibo Zhang, Binqiang Zhao, and Dik Lun Lee. 2018.  Billion-scale commodity embedding for e-commerce recommendation in alibaba.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 839–848.
  * Wang et al. (2016b) Suhang Wang, Jiliang Tang, Charu Aggarwal, and Huan Liu. 2016b.  Linked document embedding for classification.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 115–124.
  * Wang et al. (2019a) Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liqun Chen, Xinyuan Zhang, Ruiyi Zhang, Qian Yang, Ricardo Henao, and Lawrence Carin. 2019a.  Improving textual network learning with variational homophilic embeddings.  _arXiv preprint arXiv:1909.13456_.
  * Wang et al. (2019b) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2019b.  Kepler: A unified model for knowledge embedding and pre-trained language representation.  _arXiv preprint arXiv:1911.06136_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xu et al. (2018) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2018.  How powerful are graph neural networks?  _arXiv preprint arXiv:1810.00826_.
  * Xu et al. (2019) Zenan Xu, Qinliang Su, Xiaojun Quan, and Weijia Zhang. 2019.  A deep neural information fusion architecture for textual network embeddings.  _arXiv preprint arXiv:1908.11057_.
  * Yang et al. (2015) Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun, and Edward Y Chang. 2015.  Network representation learning with rich text information.  In _IJCAI_ , volume 2015, pages 2111–2117.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _arXiv preprint arXiv:1906.08237_.
  * Yasunaga et al. (2017) Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radev. 2017.  Graph-based neural multi-document summarization.  _arXiv preprint arXiv:1706.06681_.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018.  Graph convolutional neural networks for web-scale recommender systems.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 974–983.
  * Zhou et al. (2020) Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020.  Graph neural networks: A review of methods and applications.  _AI Open_ , 1:57–81.
  * Zhou et al. (2019) Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019.  Gear: Graph-based evidence aggregating and reasoning for fact verification.  _arXiv preprint arXiv:1908.01843_.
  * Zhu et al. (2021) Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Liangjie Zhang, Tianqi Yan, Ruofei Zhang, and Huasha Zhao. 2021.  Textgnn: Improving text encoder via graph neural network in sponsored search.  _arXiv preprint arXiv:2101.06323_.


copyrightbox

# Cross-Domain Learning for
Classifying Propaganda in Online Contents

Liqiang Wang1,2, Xiaoyu Shen1, Gerard de Melo3, Gerhard Weikum1
Max Planck Institute for Informatic, Saarbrücken, Germany1
Shandong University, Jinan, China2
Hasso Plattner Institute, University of Potsdam, Potsdam, Germany3
{lwang, xshen<EMAIL_ADDRESS><EMAIL_ADDRESS>
Now at Amazon, work done before joining

###### Abstract

As news and social media exhibit an increasing amount of manipulative
polarized content, detecting such propaganda has received attention as a new
task for content analysis. Prior work has focused on supervised learning with
training data from the same domain. However, as propaganda can be subtle and
keeps evolving, manual identification and proper labeling are very demanding.
As a consequence, training data is a major bottleneck.

In this paper, we tackle this bottleneck and present an approach to leverage
cross-domain learning, based on labeled documents and sentences from news and
tweets, as well as political speeches with a clear difference in their degrees
of being propagandistic. We devise informative features and build various
classifiers for propaganda labeling, using cross-domain learning. Our
experiments demonstrate the usefulness of this approach, and identify
difficulties and limitations in various configurations of sources and targets
for the transfer step. We further analyze the influence of various features,
and characterize salient indicators of propaganda.

##  1 Introduction

###  1.1 Motivation and Problem

Propaganda can be loosely defined as “misleading information that is spread
deliberately to deceive and manipulate its recipients” (see, e.g., Jowett and
O’Donnell, 2018 and www.britannica.com/topic/propaganda). Various factors of
propaganda have been studied in the humanities, including emotionality of
language, biased selection of information and deviation from facts,
manipulation of cognition, and more Ellul and Kellen (1973); Silverstein
(1987); Jowett and O’Donnell (2018). However, there is no consensus on the
decisive factors that tell whether a given article or speech is propagandistic
or not.

In the modern digital world, the influence of propaganda on society has
drastically increased. Hence, there is also a major increase in computer
science, computational linguistics and computational sociology research on
analyzing, characterizing and, ultimately, automatically detecting propaganda
Da San Martino et al. (2020).

To a first degree, one may think of propaganda as a variation of fake news,
and some works investigate propaganda as a refined type of disinformation
(see, e.g., Rashkin et al., 2017; Wang et al., 2019; Shu et al., 2017). While
false claims can be an element of propaganda, we think that fake news is
merely the tip of the iceberg, and that the persuasive and manipulative nature
of propagandistic contents requires deeper approaches. Classifiers for
propaganda detection need to better capture how propaganda is expressed in
subtle ways by language style and rhetoric or even demagogic wording. This
holds for news as well as social media posts and speeches. In all these cases,
correct information may be presented in incomplete form or placed in distorted
contexts, along with manipulative phrases, in order to mislead the audience.

Prior work has mostly looked into news articles (e.g., Saleh et al., 2019;
Barrón-Cedeño et al., 2019; Da San Martino et al., 2019) and tweets, and has
typically focused on strongly polarized topics like the 2016 US election and
the related Russian Internet Research Agency (IRA) affair, the UK Brexit
discussion, or political extremism. All these approaches consider propaganda
detection as a classification task assuming sufficient amounts of labeled in-
domain training data. For example, in the “Hack the News” datathon
challenge111https://www.datasciencesociety.net/hack-news-datathon/, a large
number of news articles Barrón-Cedeño et al. (2019) and sentences Da San
Martino et al. (2019) from such articles were annotated by distant supervision
and human judgment, respectively, to train a variety of machine learning
methods. The resulting F1 scores on the leaderboard of this benchmark are
amazingly high, around 90%. This may give the impression that propaganda
detection is a solved problem. However, most of the positively labeled samples
are simple cases of “loaded language” with strong linguistic cues independent
of the topic. Moreover, the learned classifiers benefit from ample training
data, which is all but self-guaranteed in general.

In this paper, we question these prior assumptions, hypothesizing that
propagandistic sources and speakers are sophisticated and creative and will
find new forms of deception evading the trained classifiers. The overall
approach is still text classification; the novelty of our approach lies in
cross-domain learning, where domains denote different kinds of sources, such
as news articles vs. social media posts vs. public speeches. We acknowledge
that there is often a shortage of perfectly fitting labeled data, and instead
tap into alternative sources that require a transfer step. Specifically, we
consider speeches and tweets, in addition to news articles, at both article
and sentence levels.

###  1.2 Approach and Contribution

Our goal is to build more general propaganda detectors, which can leverage
different kinds of data sources. In particular, we tap on political speeches
of notorious propagandists, such as Joseph Goebbels (the Nazi’s Reich Minister
of Propaganda). As it is very difficult (and often subjective) to label
speeches and their sentences in a binary manner, we pursue a pairwise ordinal
approach where training data merely ranks samples of a strongly propagandistic
speaker against those of a relatively temperate speaker. We investigate to
what extent models learned from such data can be transferred to classifying
news and tweets, and we also study the inverse direction of learning from news
and tweets to cope with speeches.

Figure 1 illustrates our framework towards generalizable propaganda detection
that overcomes the bottleneck of directly applicable training labels and
instead leverages cross-domain learning.

Figure 1: Framework for propaganda detection and analysis

The salient contributions of this paper are as follows.

  1. 1.

We introduce a framework for cross-domain learning of propaganda
classification. This comprises data collection, feature selection, training
procedures, different learning methods, and analysis.

  2. 2.

We devise a pairwise ranking LSTM model, called $\rm{LSTMR}$, that uses a
judiciously designed loss function to enhance cross-domain performance.

  3. 3.

We present experiments and analyses that provide insights into the advantages
as well as limitations of cross-domain learning for propaganda classification.

  4. 4.

All our datasets and code have been made publicly available to support further
research on understanding and countering advanced forms of
propaganda.222https://github.com/leereak/propaganda-detection

##  2 Related Work

Propaganda Detection. We summarize the state-of-the-art according to the type
of underlying data sources: the news, tweets or data to be fact-checked. Saleh
et al. (2019) detect propagandistic content using a variety of well-engineered
features. The study is based on a dataset of news articles annotated for the
SemEval 2019 task on “Hyperpartisan News Detection”. Barrón-Cedeño et al.
(2019) generated training and test data by collecting news from blacklisted
sites as propaganda, while considering the Gigaword News corpus as trusted
news. Further research related to propaganda detection on news articles
includes work on multi-class labeling Da San Martino et al. (2019) and on the
related topic of hyperpartisanship Potthast et al. (2018).

For Twitter as a data source, several datasets are widely used, most notably
the tweets on the Russia-based IRA Miller (2019); Farkas and Bastos (2018);
Badawy et al. (2019), contents spread by trolls or bots Caldarelli et al.
(2019); Williamson III and Scrofani (2019), and extremist tweets Nizzoli et
al. (2019); Johnston and Weiss (2017). Most of these studies rely on
classifiers developed for bot detection to filter the propagandistic contents.
Deep learning methods are also used to detect propaganda Johnston and Weiss
(2017); Nizzoli et al. (2019). As more emphasis is on the analytics, the above
detection techniques are still in the formative stage, by applying basic
approaches like logistic regression, support vector machines, LSTMs, etc.
These methods motivate us with regard to our feature engineering.

As mentioned above, some works take propaganda as a fine-grained type of fake
news Rashkin et al. (2017). Some approaches Rashkin et al. (2017); Wang et al.
(2019); Shu et al. (2017, 2019) considered fact-checked statements from
sources such as PolitiFact, Snopes, BuzzFeed, etc. This data encompasses
trusted content, satire, hoaxes, as well as propaganda, but the approaches do
not develop propaganda-specific techniques. In our view, sophisticated forms
of propaganda are quite different from hoaxes or plainly wrong statements, and
call for custom-tailored approaches.

Propaganda Analysis. Analyses of propagandistic content have mostly looked
into quantifying its influence and spreading across networks. Timothy (2017)
studied how propaganda influences public opinion by means of simulations. Some
studies Caldarelli et al. (2019); Williamson III and Scrofani (2019) analyzed
the role of bots in spreading propagandistic posts. Gorrell et al. (2019)
examined strategies of troll accounts. Farkas and Bastos (2018) reported
findings on propaganda in IRA tweets. The observations suggest that techniques
are customized to the targeted political agenda. As for feature analysis, the
work of Barrón-Cedeño et al. (2019) and Potthast et al. (2018) investigated
the effectiveness of linguistic and stylistic features. Bisgin et al. (2019)
analyzed extremism propaganda contents in terms of entities, topics and
targets.

Fake news analysis techniques Resende et al. (2019); Volkova and Jang (2018);
Shen et al. (2017); Wang et al. (2019); Zhang et al. (2018) can as well serve
as inspiration to better understand propaganda. In particular, Budak (2019)
studied the prevalence and focus of fake news based on related tweets, news
and interviews. Yang et al. (2019) used visualization techniques for analyzing
fake news to make the detector more explainable.

##  3 Datasets

For our study, we compile five datasets from three domains with the aim of
exploring cross-domain characteristics and performance of propaganda
detection.

Speeches. We collected transcripts of speeches from four politicians,
organized as ordered pairs. Trump333https://factba.se and
Obama444http://obamaspeeches.com are considered as contemporary speakers,
largely talking about the same or related topics. We consider the former as
more propagandistic than the latter. We use Joseph Goebbels (the Nazis’
Minister of Propaganda, human-translated to English by the data
provider)555https://research.calvin.edu/german-propaganda-archive/goebmain.htm
and Winston Churchill (the Prime Minister of the
UK)666https://winstonchurchill.org/resources/speeches/ as prominent figures
from the World War II era, with the former being more propagandistic than the
latter. We realize that all four of these politicians have given some
propagandistic speeches. Our assumption is that, collectively and relatively,
two of the speakers exhibit substantially less propaganda than the other two.
The data is organized at two different levels of granularity: articles
($\rm{SPE_{A}}$) and sentences ($\rm{SPE_{S}}$).

News. The news dataset comes from the Hack the News Datathon1. We combined and
reorganized different datasets Barrón-Cedeño et al. (2019); Da San Martino et
al. (2019) from this source, to construct an article-level news corpus
($\rm{NEW_{A}}$) and a sentence-level corpus ($\rm{NEW_{S}}$), both
comprehensively annotated with binary labels: propagandistic or normal. Note
that the articles for the sentence-level corpus are completely disjoint from
the ones in the article-level corpus. So learning on one and testing on the
other entails a challenging cross-domain transfer as well.

Tweets. We combine two pre-existing collections of tweets to construct this
$\rm{TWE}$ dataset. We consider the Twitter IRA corpus Edgett (2017) with time
period of 2016 as propagandistic, and the “twitter7” data from SNAP Yang and
Leskovec (2011) in 2009 as regular. As “twitter7” encompasses around 476
million tweets, we under-sampled 8,963 tweets to maintain a balance with the
IRA data. The tweets are randomly sampled from the June 2009 collection within
the dataset. As with speeches, the data is cast into ordered pairs rather than
using absolute labels as ground truth.

As some of the data initially comes with a strong label skew (with way more
negative than positive samples), we apply under-sampling to construct corpora
with balanced positive and negative samples. Table 1 summarizes our five
datasets.

Table 1: Dataset sizes. Subscripts $A$ and $S$ indicate article and sentence
granularity, respectively.

| Speeches | News | Tweets
---|---|---|---
Dataset | $\rm{SPE_{A}}$ | $\rm{SPE_{S}}$ | $\rm{NEW_{A}}$ | $\rm{NEW_{S}}$ | $\rm{TWE}$
Size | 288 | 24,934 | 7,798 | 7,876 | 17,926

##  4 Methods

In this section, we first introduce commonly used methods for propaganda
detection, and subsequently introduce our proposed pairwise ranking model that
aims at enhancing the effectiveness in the cross-domain setting.

We investigate three widely used methods as propaganda classifiers: logistic
regression (LR) Hosmer Jr et al. (2013), support vector machines (SVM) Cortes
and Vapnik (1995) and bidirectional long short-term memory (LSTM) Graves and
Schmidhuber (2005) neural networks. In addition, we devise an enhanced form of
LSTM-based network that incorporates pairwise ranking information and is
equipped with a specifically designed loss function. We denote this method as
LSTMR, and present it in the Subsection 4.3.

###  4.1 Feature-based Models

$\rm{LR}$ and $\rm{SVM}$ are feature-driven learners. We consider various
informative features on the language characteristics of articles and
sentences:

  * $\bullet$

word counts (WC),

  * $\bullet$

word level TF-IDF scores (TI-W),

  * $\bullet$

N-gram level TF-IDF scores for $N$=2 or 3 (TI-G),

  * $\bullet$

occurrence statistics of word categories, such as first-person pronouns or
negative-emotion words, from the widely used Linguistic Inquiry and Word Count
(LIWC) dictionary Pennebaker et al. (2015),

  * $\bullet$

the combination of all the features (ALL).

###  4.2 LSTM Baseline Model

Unlike LR and SVM, the LSTM-based methods do not rely on feature modeling.
Instead they can use pre-trained word embeddings to capture the text structure
and semantic features. We adopt a basic LSTM classifier as a baseline. The
structure of the LSTM baseline model is illustrated in Figure 2. On top of the
bi-LSTM layers, the networks have a dense layer and a final sigmoid function
to learn scores and yield classifier labels. The cross-entropy loss function
is applied for the $\rm{LSTM}$ classifier.

###  4.3 Pairwise Ranking Model

Model Architecture. Figure 2 depicts our pairwise propaganda ranking model
$\rm{LSTMR}$. The motivation for this design is that training labels may be
subjective and noisy, especially but not only in a cross-domain setting where
training and test data do not come from the same distribution. Traditional
supervised learning models strictly distinguish the samples according to their
labels, which leads to models over-fitting the training set, while their
cross-domain generalizability diminishes. Supervised learning typically
interprets the labels of data samples as a strict ground truth. In certain
settings, this can lead to over-fitting to the training data distribution
while hampering the ability for cross-domain learning. The $\rm{LSTMR}$ model
aims to mitigate this over-fitting and enhance cross-domain applicability by
relaxing constraints from strict labeling to rankings. Unavoidably, the in-
domain performance may decrease. As Figure 2 shows, the model operates in two
phases:

  1. 1.

The first phase is to train the model based on the ranked pairs of data
samples, where one is considered more propagandistic than the other. The bi-
LSTM and dense layers serve to learn numeric scores for each of the two data
points in each pair. Then the two scores are constrained by a rule that the
more propagandistic sample is supposed to obtain a higher score than the other
sample that it is paired up with. We can think of the former as a positive
sample and the latter as a negative sample, in terms of a binary classifier.
This stage results in an improved scoring function, with awareness of the
constraints.

  2. 2.

The second phase addresses the scoring of previously unseen data points with
the trained $\rm{LSTMR}$ model receiving a single input text instead of a
pair. The label is then determined according to the constraints-based scoring
function.

Figure 2: Pairwise propaganda ranking model $\rm{LSTMR}$

Pairwise Constraints. The constraints are instantiated using the following
loss functions, where $\mathbf{y}$ is the ground-truth label ($0$ and $1$
corresponding to the less propagandistic and the more propagandistic sample,
respectively), $\hat{\mathbf{y}}$ is the predicted score, and
$\mathbb{I}(\cdot)$ is the indicator function with output 1(constraints are
satisfied) or 0:

  * •

Logistic loss (LOG) Pasumarthi et al. (2019):

$\displaystyle\ell(\mathbf{y},\hat{\mathbf{y}})$
$\displaystyle=\sum_{j=1}^{n}\sum_{k=1}^{n}\mathbb{I}(y_{j}>y_{k})\cdot$ (1)
$\displaystyle\log(1+\exp(\hat{y}_{k}-\hat{y}_{j}))$

  * •

Linear loss (LIN) Hanselowski et al. (2018):

$\displaystyle\ell(\mathbf{y},\hat{\mathbf{y}})=\sum_{j=1}^{n}\sum_{k=1}^{n}\mathbb{I}(y_{j}>y_{k})\,(1+\hat{y}_{k}-\hat{y}_{j})$
(2)

  * •

Threshold loss (THR), where $\theta$ is the threshold to control the intensity
of the constraints:

$\displaystyle\ell(\mathbf{y},\hat{\mathbf{y}})$
$\displaystyle=\sum_{j=1}^{n}\sum_{k=1}^{n}\mathbb{I}(y_{j}>y_{k})\cdot$ (3)
$\displaystyle\mathbb{I}\left(\frac{1+\hat{y}_{k}-\hat{y}_{j}}{2}>\theta\right)\frac{1+\hat{y}_{k}-\hat{y}_{j}}{2}$

  * •

Counting loss (COU) to count the discordant pairs:

$\displaystyle\ell(\mathbf{y},\hat{\mathbf{y}})=\sum_{j=1}^{n}\sum_{k=1}^{n}\mathbb{I}(y_{j}>y_{k})\mathbb{I}(\hat{y}_{k}>\hat{y}_{j})$
(4)

Among the above loss functions, the threshold loss is designed to further
relax the constraints by explicitly disregarding those pairs with confidence
below the threshold. We compare the model performance using different loss
functions in Section 6. In the subsequent evaluation experiments we adopt
$\rm{THR}$ loss function for $\rm{LSTMR}$ by reason of its better performance.

Sampling Method. We apply different methods for constructing and sampling the
ranked pairs. We consider data from all three source types: news, tweets and
speeches. Whenever we have 0/1-labeled data points, we randomly combine a
positive sample (1) and a negative sample (0) into an ordered pair for
training the $\rm{LSTMR}$ model. This is the case for news and tweets, as
these datasets have been manually annotated. When we have merely relatively
ordered data points where some are more propagandistic than others, we sample
such pairs. This is the case for the speeches, where we assume that Goebbels
is more propagandistic than Churchill and the same ranking holds for Trump
versus Obama.

For news and tweets, we need to cope with skewed label distributions: way more
negative than positive samples. As we consider cross-domain learning where the
eventual test set has an a-priori unknown distribution that could be fairly
different from the prior label distribution at training time, we generally re-
balance all datasets. We consider two strategies to this end. The first is to
under-sample the entire dataset, and the second is to over-sample the smaller
class. Note that the over-sampling is per data point, so we still construct
new pairs of ranked samples (i.e., there are no duplicate pairs).

Table 2: F1 score (‰) for in-domain classification with different methods,
features and datasets. $PN$ stands for proper noun.

PN | Method | $\rm{SPE_{A}}$ | $\rm{SPE_{S}}$ | $\rm{TWE}$ | $\rm{NEW_{A}}$ | $\rm{NEW_{S}}$
---|---|---|---|---|---|---
WC | TI-W | TI-G | LIWC | ALL | WC | TI-W | TI-G | LIWC | ALL | WC | TI-W | TI-G | LIWC | ALL | WC | TI-W | TI-G | LIWC | ALL | WC | TI-W | TI-G | LIWC | ALL
YES | LR | 964 | 971 | 964 | 903 | 964 | 844 | 845 | 795 | 629 | 865 | 813 | 810 | 651 | 674 | 817 | 914 | 926 | 895 | 810 | 930 | 665 | 674 | 621 | 609 | 677
SVM | 960 | 982 | 960 | 900 | 982 | 837 | 838 | 783 | 630 | 846 | 795 | 794 | 638 | 676 | 795 | 907 | 915 | 875 | 812 | 921 | 647 | 645 | 598 | 608 | 657
LSTM | 973 | 747 | 807 | 866 | 715
LSTMR | 908 | 751 | 789 | 856 | 718
NO | LR | 960 | 971 | 960 | 892 | 964 | 823 | 820 | 778 | 621 | 843 | 755 | 757 | 665 | 662 | 762 | 895 | 905 | 877 | 811 | 915 | 646 | 660 | 615 | 600 | 659
SVM | 953 | 982 | 960 | 900 | 975 | 816 | 814 | 766 | 621 | 820 | 741 | 739 | 652 | 661 | 735 | 882 | 893 | 853 | 813 | 907 | 632 | 639 | 594 | 595 | 640

##  5 Experimental Evaluation

Table 3: Precision (P), recall (R) and F1 (F) score (‰) on cross-domain
classification with the features. Results are highlighted when $\rm{LSTMR}$ is
better than $\rm{LSTM}$.

| Test | $\rm{SPE_{A}}$ | $\rm{SPE_{S}}$ | $\rm{TWE}$ | $\rm{NEW_{A}}$ | $\rm{NEW_{S}}$
---|---|---|---|---|---|---
Train | Method | P | R | F | P | R | F | P | R | F | P | R | F | P | R | F
$\rm{SPE_{A}}$ | LR | - | - | - | 949 | 084 | 154 | 451 | 009 | 017 | 900 | 005 | 009 | 696 | 012 | 024
SVM | - | - | - | 887 | 181 | 301 | 532 | 056 | 102 | 771 | 014 | 027 | 553 | 055 | 101
LSTM | - | - | - | 639 | 113 | 191 | 312 | 118 | 172 | 279 | 190 | 226 | 450 | 110 | 177
LSTMR | - | - | - | 632 | 064 | 115 | 354 | 067 | 113 | 300 | 233 | 263 | 430 | 098 | 159
$\rm{SPE_{S}}$ | LR | 1000 | 667 | 800 | - | - | - | 508 | 648 | 570 | 333 | 005 | 010 | 459 | 367 | 408
SVM | 1000 | 778 | 875 | - | - | - | 507 | 630 | 562 | 440 | 017 | 033 | 472 | 400 | 433
LSTM | 886 | 486 | 628 | - | - | - | 484 | 889 | 627 | 452 | 018 | 035 | 485 | 657 | 559
LSTMR | 980 | 667 | 793 | - | - | - | 446 | 587 | 507 | 475 | 034 | 064 | 490 | 471 | 480
$\rm{TWE}$ | LR | 727 | 056 | 103 | 537 | 671 | 596 | - | - | - | 665 | 192 | 298 | 524 | 681 | 592
SVM | 583 | 049 | 090 | 524 | 619 | 568 | - | - | - | 650 | 246 | 357 | 533 | 658 | 589
LSTM | 267 | 306 | 285 | 517 | 981 | 677 | - | - | - | 537 | 770 | 633 | 496 | 961 | 654
LSTMR | 356 | 326 | 341 | 500 | 754 | 601 | - | - | - | 646 | 669 | 658 | 534 | 855 | 657
$\rm{NEW_{A}}$ | LR | 510 | 889 | 648 | 481 | 556 | 516 | 571 | 454 | 506 | - | - | - | 564 | 685 | 619
SVM | 500 | 757 | 602 | 487 | 572 | 526 | 554 | 527 | 540 | - | - | - | 557 | 702 | 621
LSTM | 516 | 986 | 678 | 362 | 264 | 305 | 532 | 197 | 287 | - | - | - | 616 | 566 | 590
LSTMR | 515 | 972 | 673 | 489 | 913 | 637 | 532 | 831 | 649 | - | - | - | 530 | 951 | 680
$\rm{NEW_{S}}$ | LR | 503 | 1000 | 670 | 460 | 474 | 467 | 571 | 393 | 466 | 514 | 980 | 674 | - | - | -
SVM | 507 | 1000 | 673 | 474 | 486 | 480 | 546 | 431 | 481 | 534 | 964 | 688 | - | - | -
LSTM | 505 | 1000 | 671 | 470 | 750 | 578 | 559 | 632 | 593 | 503 | 998 | 669 | - | - | -
LSTMR | 511 | 1000 | 676 | 476 | 762 | 586 | 570 | 723 | 637 | 511 | 994 | 675 | - | - | -

Table 4: F1 score (‰) for cross-domain classification with proper nouns
removed.

| Test | $\rm{SPE_{A}}$ | $\rm{SPE_{S}}$ | $\rm{TWE}$ | $\rm{NEW_{A}}$ | $\rm{NEW_{S}}$
---|---|---|---|---|---|---
Train | Method | P | R | F | P | R | F | P | R | F | P | R | F | P | R | F
$\rm{SPE_{A}}$ | LR | - | - | - | 967 | 061 | 115 | 491 | 006 | 013 | 938 | 004 | 008 | 688 | 008 | 017
SVM | - | - | - | 925 | 098 | 178 | 546 | 026 | 050 | 778 | 007 | 014 | 535 | 023 | 045
$\rm{SPE_{S}}$ | LR | 980 | 347 | 513 | - | - | - | 494 | 581 | 534 | 207 | 002 | 003 | 449 | 347 | 391
SVM | 987 | 535 | 694 | - | - | - | 497 | 540 | 518 | 370 | 008 | 015 | 477 | 371 | 417
$\rm{TWE}$ | LR | 462 | 042 | 076 | 537 | 704 | 609 | - | - | - | 564 | 215 | 311 | 520 | 709 | 600
SVM | 333 | 028 | 051 | 525 | 641 | 577 | - | - | - | 566 | 252 | 349 | 525 | 670 | 589
$\rm{NEW_{A}}$ | LR | 518 | 785 | 624 | 486 | 422 | 452 | 569 | 341 | 427 | - | - | - | 576 | 560 | 568
SVM | 486 | 611 | 542 | 490 | 449 | 469 | 532 | 408 | 462 | - | - | - | 564 | 582 | 573
$\rm{NEW_{S}}$ | LR | 503 | 1000 | 670 | 476 | 491 | 483 | 539 | 379 | 445 | 511 | 982 | 672 | - | - | -
SVM | 505 | 1000 | 671 | 494 | 506 | 500 | 522 | 432 | 473 | 534 | 961 | 687 | - | - | -

Setup. The feature dimensions of $\rm{WC}$, $\rm{TI}$-$\rm{W}$,
$\rm{TI}$-$\rm{G}$ are truncated to their respective top 5,000, and we use
LIWC 2015 Pennebaker et al. (2015) with 76 categories including first person
singular, negation, sexual, swear, etc. Optionally, we eliminate proper nouns
(PN), as tagged using the NLTK toolkit Bird et al. (2009). This is to reduce
the influence of particular names and their respective topics (e.g., “Iran”).
The rationale is that texts could be about the very same topic but differ in
their degree of propaganda.

For $\rm{LSTM}$ and $\rm{LSTMR}$, we use the pre-trained 100-dimensional
embeddings from GloVe (Pennington et al., 2014), 128 bi-LSTM cells and
200$\times$50$\times$1 dense layers. The embeddings are frozen during
training. For $\rm{LSTMR}$, the $\rm{THR}$ loss with a threshold of $0.4$ and
complete sampling (neither under- nor over-sampling) is adopted. Note that
$\rm{LSTMR}$ is a ranking model rather than a classifier. As the ranking
scores are comparable and range from 0 to 1, we empirically consider $\geq
0.5$ as a threshold to classify test samples as positive or negative.

For in-domain learning, where training and test data are from the same source,
we adopt 5-fold cross validation. Hyper-parameters are tuned by this setting,
and then kept fixed for cross-domain experiments. Note that in the cross-
domain case, cross-validation is impossible as the target-domain data is
treated completely unseen. We report on precision (P), recall (R) and F1 (F)
scores, with propaganda being the positive class.

In-Domain Performance. The results are shown in Table 2. Overall, we achieve
acceptable in-domain performance on most datasets (over 80%) except
$\rm{NEW_{S}}$ (around 65%). Results for article-level classification
($\rm{SPE_{A}}$ and $\rm{NEW_{A}}$) are better than for the sparser, and
perhaps noisier, sentence-level data ($\rm{SPE_{S}}$, $\rm{TWE}$ and
$\rm{NEW_{S}}$). We observe a larger drop for $\rm{NEW_{S}}$ than
$\rm{SPE_{S}}$ regarding the $\rm{LR}$ and $\rm{SVM}$ methods but not for the
$\rm{LSTM}$ and $\rm{LSTMR}$ methods. This suggests a greater robustness of
semantic features compared to linguistic features.

For the $\rm{LR}$ and $\rm{SVM}$ methods, the $\rm{WC}$, $\rm{TI}$-$\rm{W}$
and $\rm{TI}$-$\rm{G}$ features work better than the $\rm{LIWC}$ features.
However, the combination of all four feature groups yields the best results.
When omitting proper nouns, performance drops, which indicates that the models
also learn some topic features instead of propaganda itself. However, the
$\rm{LIWC}$ representations are not affected much when excluding proper nouns
due to their focus on word categories.

The $\rm{LSTM}$ and $\rm{LSTMR}$ methods perform slightly worse than $\rm{LR}$
and $\rm{SVM}$ on all datasets except $\rm{NEW_{S}}$. Due to the influence of
the pairwise ranking and the loss function, $\rm{LSTMR}$ performs slightly
worse than $\rm{LSTM}$ on some datasets. As a trade-off, it gets better cross-
domain performance (see Table 3). Overall, feature-based learning works best
for the standard case of in-domain classification.

Cross-Domain Performance. The cross-domain results are given in Table 3. As
expected, cross-domain classification is much more challenging than in-domain
classification. Still, many cross-domain settings show reasonably good
performance. The best cross-domain results are observed for the classifier
trained on $\rm{NEW_{A}}$ and tested on $\rm{NEW_{S}}$. The most balanced
precision–recall performance is obtained by the classifier trained on
$\rm{TWE}$ and tested on $\rm{NEW_{S}}$. Overall, $\rm{LSTMR}$ compares
favorably against other methods, especially on $\rm{TWE}$, $\rm{NEW_{A}}$ and
$\rm{NEW_{S}}$.

Combining Datasets for Training. As a final configuration, we combine several
datasets, excluding $\rm{NEW_{S}}$, into a single training set and test on
$\rm{NEW_{S}}$. The results are given in Table 5. As we see, the larger amount
of training data does not help the classifiers to improve their cross-domain
performance. On the contrary, the F1 scores slightly drop when more data is
combined. The likely reason is the high variability in the underlying features
and data characteristics that comes from such highly heterogeneous sources.
These settings require further research.

Table 5: Cross-domain performance (‰) of classifiers trained on combined
datasets and tested on $\rm{NEW_{S}}$.

Method | $\rm{SPE_{A}}$ | +$\rm{SPE_{S}}$ | ++$\rm{TWE}$ | +++$\rm{NEW_{A}}$
---|---|---|---|---
P | R | F | P | R | F | P | R | F | P | R | F
LR | 696 | 012 | 024 | 447 | 327 | 377 | 503 | 446 | 473 | 509 | 580 | 542
SVM | 553 | 055 | 101 | 485 | 363 | 415 | 508 | 475 | 491 | 507 | 554 | 529
LSTM | 450 | 110 | 177 | 478 | 594 | 530 | 496 | 966 | 655 | 462 | 639 | 536
LSTMR | 430 | 098 | 159 | 477 | 416 | 445 | 504 | 361 | 421 | 508 | 458 | 482

##  6 Analysis

Datasets and Granularity. The best cross-domain results are obtained for
training on news and applying the learned models to speeches or tweets. This
suggests that high-quality training labels are still crucial. Performance for
articles is better than for sentences, underlining the difficulty of dealing
with very short texts without placing them into their full context. From a
model perspective, this makes features such as TF-IDF sparser without abundant
signals to aid in the detection.

Proper Nouns. Table 4 shows cross-domain results when excluding proper nouns.
We observe a notable drop in performance. This suggests that focusing on
language alone is not sufficient for detecting propaganda. The entities to
which news, tweets or speeches refer are important as topical context, and
cannot be easily left out. This finding seems to contradict recent
experimental work on fact checking Suntwal et al. (2019), where de-
lexicalization was found useful. A major reason is that our cross-domain
setting not just switches between data sources, but involves a transfer to a
different style of source, like news vs. speeches. Moreover, our negative
finding underlines the additional complexity of propaganda detection, calling
for more research on the role of proper nouns.

Feature Groups. We conduct experiments with classifiers learned with single
feature groups, using $\rm{NEW_{S}}$ for training. Table 6 gives the cross-
domain results. Comparing these against the $\rm{ALL}$ feature performance in
Tables 2 and 4, we observe that the combination of all four features has a
positive effect for in-domain classification but does not work so well for
cross-domain learning. However, there is no single feature group that
dominates all others. This suggests that different datasets exhibit different
kinds of linguistic cues, underlining again the big challenge in cross-domain
classification.

Table 6: Cross-domain performance (F1 score, ‰) with training on
$\rm{NEW_{S}}$, using different features. ‘1’, ‘2’, ‘3’ and ‘4’ stand for
$\rm{WC}$, $\rm{TI}$-$\rm{W}$, $\rm{TI}$-$\rm{G}$ and $\rm{LIWC}$.

Method | $\rm{SPE_{A}}$ | $\rm{SPE_{S}}$ | $\rm{TWE}$ | $\rm{NEW_{S}}$
---|---|---|---|---
1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4
LR | 673 | 673 | 671 | 387 | 397 | 458 | 452 | 491 | 422 | 487 | 418 | 435 | 672 | 676 | 623 | 603
SVM | 674 | 673 | 678 | 390 | 428 | 467 | 458 | 493 | 459 | 498 | 416 | 443 | 687 | 688 | 662 | 596

Informative Language Features. To analyze to what extent word-level features
can yield interpretable cues that characterize propaganda in different kinds
of sources, we trained the $\rm{LR}$ classifer on the $\rm{NEW_{S}}$ data with
$\rm{TI}$-$\rm{W}$ and $\rm{LIWC}$ features, excluding proper nouns. We
observed the following most distinctive words, characteristic for propaganda:

  * “advantage”, “absolutely”, “neo”, “political”, “attacks”, “american”, “shocking”, “influential”, “impossible”, “lies”, “devastating”, “hell”, “administration”, “autonomous”, “ridiculous”

The most salient $\rm{LIWC}$ features (word categories) were:

  * _affect_ , _neg-emo_ , _anger_ , _swear_ , _certain_

This suggests that exaggerations (e.g., “absolutely”) and negative emotions
(e.g., “lies” or “devastating”) play a key role in manipulating the audience.
As for $\rm{LIWC}$ features, words that express negative emotions are typical
for propaganda, as well as being strongly self-confident.

Figure 3: F1 of $\rm{LSTMR}$ trained on $\rm{NEW_{S}}$ and tested on other
datasets, with different loss functions.

LSTMR Loss Functions. For our $\rm{LSTMR}$ model, the selection of the loss
function has a notable influence on performance. The previous experiments all
used $\rm{THR}$ as loss function. Training on $\rm{NEW_{S}}$ and with cross-
domain test data, Figure 3 compares the effectiveness of all four loss
functions that we studied. We notice that the specifically designed $\rm{THR}$
loss function outperforms all alternatives, especially on the $\rm{SPE_{S}}$
and $\rm{TWE}$ datasets. This observation is in line with our design goal that
$\rm{THR}$ helps the model training to discount ranked pairs of low
confidence. We sacrifice in-domain performance to some extent, but improve the
cross-domain classifier.

Figure 4: F1 of $\rm{LSTMR}$ trained on $\rm{NEW_{S}}$ and tested on other
data, with varying rate of over-sampling.  Figure 5: F1 of $\rm{LSTMR}$
trained on $\rm{NEW_{S}}$ and tested on other data, with varying percentage of
under-sampling.

Sampling Strategies. The training of $\rm{LSTMR}$ requires ranked pairs of
samples. Over-sampling constructs these pairs by repeatedly drawing from the
positive and negative classes. The effect of the amount of over-sampling is
plotted in Figure 4, for the setting with training on $\rm{NEW_{S}}$ and
cross-domain testing on the other datasets. The results show the limitations
of this sampling strategy: there is hardly any increase in performance and
even a drop for some cases. This calls for further research towards overcoming
training bottlenecks.

For the under-sampling strategy, Figure 5 shows results (for the same train-
test setting) with varying fractions of training samples. On the
$\rm{SPE_{A}}$ data, the performance increases with more training samples, as
expected. However, on other test datasets, performance remains unchanged or
drops or fluctuates. This points out the limitations of cross-domain
classification. Our approach counters the training bottleneck to a good
degree, but there is quite some room for improvement and the need for further
research.

##  7 Conclusion and Future Work

Although propaganda has become a pervasive challenge in online media, previous
work has mostly treated it as a variation of fake news, or considered
unrealistic settings where the test distribution precisely matches the
training data distribution.

In this paper, we present a first and preliminary analysis of the problem of
propaganda detection in cross-domain learning settings. This encompasses
several novel aspects, ranging from data collection methods, feature
computation, designing different classifiers, and the corresponding analysis.
We tap into a previously unexplored content source: speeches by politicans who
are known for different levels of propaganda, using them as collective and
relative signals. On the methodology side, we devise a pairwise ranking method
with customized loss functions to improve the classification. The experimental
results demonstrate the effectiveness of this method. Furthermore, we conduct
a series of experiments to explore the most salient factors for cross-domain
generalizability of propaganda detection learning. The observations and
analysis reveal insightful patterns and lessons for building more general
propaganda detectors.

As our datasets are still fairly small, our findings are of preliminary nature
and our methodology is subject to ongoing research. We believe that cross-
domain learning is a crucial asset for the important topic of propaganda
detection, and hope that our initial results are useful for further research
along these lines.

## Acknowledgments

The work is partly supported by “The Fundamental Research Funds of Shandong
University”. Gerhard Weikum’s work is partly supported by the ERC Synergy
Grant 610150 (imPACT). Any opinions, findings and conclusions, or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the funding agencies.

## References

  * Badawy et al. (2019) Adam Badawy, Aseel Addawood, Kristina Lerman, and Emilio Ferrara. 2019.  Characterizing the 2016 russian ira influence campaign.  _Social Network Analysis and Mining_ , 9(1):31.
  * Barrón-Cedeño et al. (2019) Alberto Barrón-Cedeño, Israa Jaradat, Giovanni Da San Martino, and Preslav Nakov. 2019.  Proppy: Organizing the news based on their propagandistic content.  _Information Processing & Management_, 56(5):1849–1864.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  O’Reilly Media, Inc.
  * Bisgin et al. (2019) Halil Bisgin, Hasan Arslan, and Yusuf Korkmaz. 2019.  Analyzing the dabiq magazine: The language and the propaganda structure of isis.  In _International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation_ , pages 1–11. Springer.
  * Budak (2019) Ceren Budak. 2019.  What happened? The spread of fake news publisher content during the 2016 US presidential election.  In _The World Wide Web Conference_ , pages 139–150. ACM.
  * Caldarelli et al. (2019) Guido Caldarelli, Rocco De Nicola, Fabio Del Vigna, Marinella Petrocchi, and Fabio Saracco. 2019.  The role of bot squads in the political propaganda on twitter.  _arXiv preprint arXiv:1905.12687_.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ , 20(3):273–297.
  * Da San Martino et al. (2020) Giovanni Da San Martino, Stefano Cresci, Alberto Barrón-Cedeño, Seunghak Yu, Roberto Di Pietro, and Preslav Nakov. 2020.  A survey on computational propaganda detection.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 4826–4832. IJCAI.
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-grained analysis of propaganda in news article.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5640–5650. ACL.
  * Edgett (2017) Sean Edgett. 2017.  Testimony of Sean J. Edgett.  _United States Senate Committee on the Judiciary, Subcommittee on Crime and Terrorism_.
  * Ellul and Kellen (1973) Jacques Ellul and Konrad Kellen. 1973.  _Propaganda: The formation of men’s attitudes_.  Vintage Books New York.
  * Farkas and Bastos (2018) Johan Farkas and Marco Bastos. 2018.  Ira propaganda on twitter: Stoking antagonism and tweeting local news.  In _Proceedings of the 9th International Conference on Social Media and Society_ , pages 281–285. ACM.
  * Gorrell et al. (2019) Genevieve Gorrell, Mehmet E Bakir, Ian Roberts, Mark A Greenwood, Benedetta Iavarone, and Kalina Bontcheva. 2019.  Partisanship, propaganda and post-truth politics: Quantifying impact in online debate.  _arXiv preprint arXiv:1902.01752_.
  * Graves and Schmidhuber (2005) Alex Graves and Jürgen Schmidhuber. 2005.  Framewise phoneme classification with bidirectional lstm and other neural network architectures.  _Neural networks_ , 18(5-6):602–610.
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  Ukp-athene: Multi-sentence textual entailment for claim verification.  _arXiv preprint arXiv:1809.01479_.
  * Hosmer Jr et al. (2013) David W Hosmer Jr, Stanley Lemeshow, and Rodney X Sturdivant. 2013.  _Applied logistic regression_ , volume 398.  John Wiley & Sons.
  * Johnston and Weiss (2017) Andrew H Johnston and Gary M Weiss. 2017.  Identifying sunni extremist propaganda with deep learning.  In _2017 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 1–6. IEEE.
  * Jowett and O’Donnell (2018) Garth S. Jowett and Victoria O’Donnell. 2018.  _Propaganda & Persuasion (7th Edition)_.  Sage Publications.
  * Miller (2019) Daniel Taninecz Miller. 2019.  Topics and emotions in russian twitter propaganda.  _First Monday_ , 24(5).
  * Nizzoli et al. (2019) Leonardo Nizzoli, Marco Avvenuti, Stefano Cresci, and Maurizio Tesconi. 2019.  Extremist propaganda tweet classification with deep learning in realistic scenarios.  In _Proceedings of the 10th ACM Conference on Web Science_ , pages 203–204. ACM.
  * Pasumarthi et al. (2019) Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. 2019.  Tf-ranking: Scalable tensorflow library for learning-to-rank.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2970–2978. ACM.
  * Pennebaker et al. (2015) James W Pennebaker, Ryan L Boyd, Kayla Jordan, and Kate Blackburn. 2015.  The development and psychometric properties of liwc2015.  Technical report.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Potthast et al. (2018) Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018.  A stylometric inquiry into hyperpartisan and fake news.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 231–240.
  * Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017\.  Truth of varying shades: Analyzing language in fake news and political fact-checking.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2931–2937.
  * Resende et al. (2019) Gustavo Resende, Philipe Melo, Hugo Sousa, Johnnatan Messias, Marisa Vasconcelos, Jussara Almeida, and Fabrício Benevenuto. 2019.  (Mis) Information dissemination in WhatsApp: Gathering, analyzing and countermeasures.  In _The World Wide Web Conference_ , pages 818–828. ACM.
  * Saleh et al. (2019) Abdelrhman Saleh, Ramy Baly, Alberto Barrón-Cedeño, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019.  Team QCRI-MIT at SemEval-2019 Task 4: Propaganda analysis meets hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 1041–1046.
  * Shen et al. (2017) Xiaoyu Shen, Youssef Oualil, Clayton Greenberg, Mittul Singh, and Dietrich Klakow. 2017.  Estimation of gap between current language models and human performance.  _Proc. Interspeech 2017_ , pages 553–557.
  * Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017.  Fake news detection on social media: A data mining perspective.  _ACM SIGKDD Explorations Newsletter_ , 19(1):22–36.
  * Shu et al. (2019) Kai Shu, Suhang Wang, and Huan Liu. 2019.  Beyond news contents: The role of social context for fake news detection.  In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_ , pages 312–320. ACM.
  * Silverstein (1987) Brett Silverstein. 1987.  Toward a science of propaganda.  _Political Psychology_ , pages 49–59.
  * Suntwal et al. (2019) Sandeep Suntwal, Mithun Paul, Rebecca Sharp, and Mihai Surdeanu. 2019.  On the importance of delexicalization for fact verification.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3411–3416. ACL.
  * Timothy (2017) Jithender J Timothy. 2017.  How does propaganda influence the opinion dynamics of a population?  _arXiv preprint arXiv:1703.10138_.
  * Volkova and Jang (2018) Svitlana Volkova and Jin Yea Jang. 2018.  Misleading or falsification: Inferring deceptive strategies and types in online news and social media.  In _Companion Proceedings of the The Web Conference 2018_ , pages 575–583. ACM.
  * Wang et al. (2019) Liqiang Wang, Yafang Wang, Gerard de Melo, and Gerhard Weikum. 2019.  Understanding archetypes of fake news via fine-grained classification.  _Social Network Analysis and Mining_ , 9(1):37.
  * Williamson III and Scrofani (2019) William Williamson III and James Scrofani. 2019.  Trends in detection and characterization of propaganda bots.  In _Proceedings of the 52nd Hawaii International Conference on System Sciences_.
  * Yang et al. (2019) Fan Yang, Shiva K Pentyala, Sina Mohseni, Mengnan Du, Hao Yuan, Rhema Linder, Eric D Ragan, Shuiwang Ji, and Xia Ben Hu. 2019.  Xfake: Explainable fake news detector with visualizations.  In _The World Wide Web Conference_ , pages 3600–3604. ACM.
  * Yang and Leskovec (2011) Jaewon Yang and Jure Leskovec. 2011.  Patterns of temporal variation in online media.  In _Proceedings of the fourth ACM international conference on Web search and data mining_ , pages 177–186. ACM.
  * Zhang et al. (2018) Amy X Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B Adams, Emmanuel Vincent, Jennifer Lee, Martin Robbins, et al. 2018.  A structured response to misinformation: Defining and annotating credibility indicators in news articles.  In _Companion Proceedings of the The Web Conference 2018_ , pages 603–612. ACM.