# Using Multi-Encoder Fusion Strategies to Improve Personalized Response
Selection

Souvik Das Sougata Saha Rohini K. Srihari
{souvikda, sougatas<EMAIL_ADDRESS>
Department of Computer Science and Engineering, University at Buffalo, NY.

###### Abstract

Personalized response selection systems are generally grounded on persona.
However, a correlation exists between persona and empathy, which these systems
do not explore well. Also, when a contradictory or off-topic response is
selected, faithfulness to the conversation context plunges. This paper
attempts to address these issues by proposing a suite of fusion strategies
that capture the interaction between persona, emotion, and entailment
information of the utterances. Ablation studies on Persona-Chat dataset show
that incorporating emotion and entailment improves the accuracy of response
selection. We combine our fusion strategies and concept-flow encoding to train
a BERT-based model which outperforms the previous methods by margins larger
than 2.3% on original personas and 1.9% on revised personas in terms of hits@1
(top-1 accuracy), achieving a new state-of-the-art performance on the Persona-
Chat dataset.

##  1 Introduction

Currently, most response selection systems tend to perform well in most cases
Gu et al. (2021a); Zhang et al. (2021b); Gu et al. (2019a, 2020a). However,
these re-ranking systems have the poor capability to detect and evade
contradictory responses. Responses selected by these systems often contradict
previous utterances, and any form of contradiction may disrupt the flow of
conversation. Previous research has attempted to incorporate persona while
selecting Gu et al. (2021b); Zhang et al. (2021a) or generating Wu et al.
(2021) responses to maintain consistency. Additionally, a correlation exists
between persona with personality Leary and Allen (2011) , which influences
empathy Richendoller and Weaver III (1994). Zhong et al. presented a multi-
domain dataset collected from several empathetic Reddit threads contributing
towards persona-based empathetic conversations. Nevertheless, no one has
studied the emotion-persona interplay in data presented in a more natural
form. Figure 1 depicts situational emotion sometimes needs more preference
than the chatbot’s persona in response selection.

On the contrary, different personality traits are related to emotion
regulation difficulties Pollock et al. (2016). Due to this, a person’s
expected emotions can deviate based on his persona. Besides that, we also
observe that concepts discussed in a conversational flow play an important
role in response selection. However, no one has incorporated this in response
selection.

Figure 1: For this conversation, the selected candidate response directly
contradicts the context. Also, the bot’s persona influences the response
selection, while the situational emotions and concepts get ignored. The
underlines phrases/words denote the concepts.

Model | Emotion Inappropriate(%) | Contradictory(%) | Off-topic(%)
---|---|---|---
BERT-CRA | 7.35 | 11.88 | 12.3

Table 1: Statistics of issues reported in the test split of Persona-Chat
inferred by BERT-CRA Gu et al. (2021b) 111Insights drawn from the human
evaluation done on 500 randomly selected data-points from self-persona
original and partner-persona original sets of Persona-Chat

We can infer the significance of these problems from Table 1. So, to increase
the usability of the personalized response selection systems, all these
fundamental problems need to be addressed. We automatically annotate Persona-
Chat Zhang et al. (2018) dataset using a series of classifiers and rule-based
modules. We model emotion-persona interaction, context-response entailment,
and concept-flow using the annotations. To compare the ability of annotated
features to enhance the emotion-persona interaction, contradiction avoidance,
and adherence to the concept flow, we perform preliminary experiments by
devising independent encoders based on BERT. Our baseline model extends BERT-
CRA Gu et al. (2021b) where we introduce an additional bot-encoder to
represent the bot-utterances better. Subsequently, we propose three fusion
strategies, emotion-aware(EmA), entailment-aware(EnA), persona-entailment-
aware(P-EmA). These fusion strategies are designed based on emotion-persona
interaction or persona-entailment information. We propose a concept-flow
encoding technique that matches relevant concepts from the context and
candidate responses with these fusion strategies.

We test our proposed methods on the Persona-Chat dataset with our automatic
annotation. The results show that a model trained on a combination of our
proposed fusion strategies outperforms the current state-of-the-art model by a
margin of 2.3% in terms of top-1 accuracy hits@1.

In summary, the contributions of this paper are three-fold. (1)Automatically
annotate Persona-Chat dataset with utterance level emotion, entailment, and
concept information to provide extra supervision. (2) A suite of fusion
strategies and a concept-flow encoder which are designed and implemented into
a series of models, aiming to explore the impact of emotion, entailment, and
concept-flow in the task of response selection. (3) Experimental results
demonstrate that our proposed models outperform the existing state-of-the-art
models by significant margins on the widely used Persona-Chat response
selection benchmark.

##  2 Related Works

###  2.1 Personalized Response Selection

Chit-chat models typically trained over many dialogues with different speakers
lack a consistent personality and explicit long-term memory. These models
produce an utterance given only a recent dialogue history. Li et al. proposed
a persona-based neural conversation model to capture individual
characteristics such as background information and speaking style. Zhang et
al. (2018) has constructed Persona-Chat dataset to build personalized dialog
systems; this is by far the largest public dataset containing million-turn
dialog conditioned on persona. Many benchmarks have been established for this
dataset. For example, Mazaré et al. (2018) proposed the fine-tuned Persona-
Chat (FT-PC) model, which first pre-trained models using a large-scale corpus
based on Reddit to extract valuable dialogues conditioned on personas and then
fine-tuned these pre-trained models on the Persona-Chat dataset. Wolf et al.
(2019); Liu et al. (2020) also employed the pre-trained language model(GPT)
for building personalized dialogue agents. Gu et al. (2020c) proposed
filtering before iteratively referring (FIRE) to ground the conversation on
the given knowledge and then perform the deep and iterative matching. Gu et
al. (2021b) explored a new direction by proposing four persona fusion
strategies, thereby incorporating partner persona in response selection.

###  2.2 Faithfulness to Context

Faithfulness in conversational systems to conversation context or knowledge is
a very broad topic that can range from decreasing fact hallucinationChen et
al. (2021), reducing contradictory responses, staying on topic, etc. Rashkin
et al. (2021) has used additional inputs to act as stylistic controls that
encourage the model to generate responses that are faithful to a provided
evidence or knowledge. However, no one has studied the level of faithfulness
the current personalized response selection systems exhibit to the
conversation history. Thus, this paper thoroughly explores the impact of
utilizing utterance-level emotions, entailment, and concepts on the
performance of personalized response selection.

##  3 Dataset

In this work, we extend Persona-Chat Zhang et al. (2018) and augment it with a
series of annotators. The dataset consists of 8939 complete dialogues for
training, 1000 for validation, and 968 for testing. Responses are selected at
every turn of a conversation sequence, resulting in 65719 context-responses
pairs for training, 7801 for validation, and 7512 for testing. The positive
and negative response ratio is 1:19 in the training, validation, and testing
sets. There are 955 possible personas for training, 100 for validation, and
100 for testing, consisting of 3 to 5 profile sentences. A revised version of
persona descriptions is also provided by rephrasing, generalizing, or
specializing the original ones to make this task more challenging.

##  4 Automatic Dataset Annotation

We have annotated the Persona-Chat with the help of a series of automatic
annotation schemes. Since we are studying the effect of emotions in
personalized response selection, we assign emotion labels to the personas,
context-utterances, and candidate responses using an emotion classifier.
Personas and utterances were annotated using an entailment classifier to
incorporate the entailment information while selecting responses. Finally, we
follow a multi-layer keyword mining strategy to match meaningful concepts
appearing in the context and response.

###  4.1 Emotion

We trained an emotion classifier on GoEmotions dataset Demszky et al. (2020).
This dataset contains 58k English Reddit comments, labeled for 27 emotion
categories or Neutral. We fine-tuned RoBERTa using this dataset. We saved the
checkpoint with the best Macro F1 of $49.4\%$ (equal to the current state of
the art) and used this for annotating each utterance. Since emotion
classification is a challenging task and given the complexity of the affect
lexicons in the corpus, we only consider the labels which can be predicted
with more than $90\%$ confidence(i.e., probability higher than $90\%$). The
goal here is to study the effect of emotion in personalized response
selection; developing a highly accurate emotion classifier is kept outside the
scope of this work.

###  4.2 Entailment

For annotating entailment, we have used an ensemble of two models. The first
one is RoBERTa based model trained on Stanford Natural Language Inference
(SNLI) corpus MacCartney and Manning (2008) released by
AllenAI222https://github.com/allenai/allennlp-models. The second model is also
a RoBERTa based model fine-tuned on DECODE Nie et al. (2020). We take the two
models’ weighted average of both probabilities during inference. The second
model has a higher preference with $80\%$ weightage as it is trained on
conversational data. The entailment label is assigned to every persona-
response and utterance-response pair.

###  4.3 Concept Mining

We mine keywords and key phrases from the persona sentences, utterances, and
responses denoted as
$\\{pc_{i}\\}_{i=1}^{N_{pc}},\\{uc_{i}\\}_{i=1}^{N_{uc}},\\{rc_{i}\\}_{i=1}^{N_{rc}}$
respectively. We follow the techniques proposed in Tang et al. (2019) to
extract the first level of keywords. Subsequently, we expand the concept lists
by extracting key phrases using the RAKE Rose et al. (2010). We hypothesize
that concepts appearing in responses should adhere to the speaker’s persona.
So, we prune some of the response/ context keywords by calculating the average
of Point-wise Mutual Information score between persona keywords and response/
context keywords $\sum_{j=1}^{N_{pc}}PMI(pc_{j},rc_{i})/N_{pc}$ and rejecting
the concepts which are below a threshold value($\lambda$). Similarly, for
response/ concept key phrases extracted using RAKE, we only keep top $N$ key
phrases. Finally, we combine the persona and context keywords and treat them
as context keywords($uc_{i}$).

##  5 Methodology

###  5.1 Problem Definition

Given a dataset
$D=\left\\{(C_{i},uc_{i},p_{i},r_{i},rc_{i},y_{i})\right\\}_{i=1}^{N}$ is a
set of $N$ tuples consisting context $C_{i}$, the persona of the speaker or
the partner $p_{i}$, response to the context $r_{i}$, and the ground truth
$y_{i}$. A set of concepts appearing in context and a response is denoted by
$uc_{i}$ and $rc_{i}$, respectively. The context can be represented as
$C_{i}=\left\\{(U_{j},Emo_{j},Entail_{j})\right\\}_{j=1}^{L}$ where $U_{j}$ is
an utterance, $Emo_{j}$ is the dominant emotion present in $U_{j}$ and
$Entail_{j}$ is the entailment label of $U_{j}$ with respect to $r_{i}$. The
$j^{th}$ utterance $U_{j}$ is denoted by
$U_{j}=\left\\{u_{1}j,u_{2}j,...,u_{M}j\right\\}$ which consists of $M$
tokens. Each response $r_{i}$ contains single utterance,
$y_{i}\in\left\\{0,1\right\\}$, $Emo_{j}\in\left\\{0,1,...P\right\\}$ , and
$Entail_{j}\in$ { entailment,
neutral,
contradiction} where $P$ are the total number of emotion types possible in the
$D$. The task is to train a matching model for $D$, $g(C,uc,p,rc,r)$. Given a
triple of context-persona-response the goal of the matching model
$g(C,uc,p,rc,r)$ is to calculate the degree of match between $(C,uc,p)$ and
$(rc,r)$.

###  5.2 Bot Context Encoding

When two users communicate, many topics are often discussed in parallel, and
sometimes a few utterances might not be relevant for response selection. To
account for the model to be aware of the speaker change information, Gu et al.
introduced a speaker disentanglement strategy in the form of speaker
embeddings fused with the original token embeddings. This technique has proven
to improve response selection performance Gu et al. (2020b); Su et al. (2021).
However, the problem of the maximum length of positional embeddings still
exists. To circumvent this, we have created bot-context encoding, which
captures the representation of the bot’s turns in the context while ignoring
the user’s turns. The intent is to use the bot’s turns to maintain consistency
during response selection. The input sequence that is sent to BERT to encode
bot context is composed as follows:

$\displaystyle x_{si}$ $\displaystyle=$
$\displaystyle[CLS]u_{2}[EOU]u_{4}[EOU]...$ (1)
$\displaystyle...u_{n-1}[EOU][SEP]r_{i}[EOU]$

Where ${u_{1},u_{4},...u_{n-1}}$ are bot’s utterances in the context, $[EOU]$
is a special token denoting the end of an utterance.

The resultant tokens $x_{si}$ are passed through bert-base-uncased, the last
hidden states of $k$ layers i.e.
$\\{\mathbf{h_{s1}^{(l)},h_{s2}^{(l)},..h_{sT}^{(l)}}\\}$, for $l=1,2,..k$ are
used in downstream tasks.

###  5.3 Fusion Strategies

We use several fusion strategies to model the inter-dependencies of the
persona, emotion, and entailment information. We use BERT Devlin et al. (2019)
as our base sentence encoder. Like the Bi-encoder Humeau et al. (2020) we
concatenate context utterances as a single context sentence before passing it
into BERT.

####  5.3.1 Baseline

For the baseline, we have extended BERT-CRA Gu et al. (2021b) where persona
and context are concatenated to form sequence A and response form sequence B.
Then, these two sequences are concatenated using $[SEP]$ token. We made two
changes to this model; first, we added speaker embeddings with the original
token representation. Secondly, we fuse bot-context encoding as described in
the previous section with BERT-CRA encoding by doing multi-headed attention
between the hidden representation of the last $k$ layers of both encoders. The
token arrangement is as follows:

$\displaystyle x_{CRAi}$ $\displaystyle=$
$\displaystyle[CLS]p_{1}p_{2}...p_{i}[EOP]u_{1}[EOU]$ (2)
$\displaystyle...u_{i}[EOU][SEP]r_{i}[EOU]$

(a) Dual encoder pipeline consisting of combination of all the encoding
strategies.

(b) Concept-flow interaction network, the output of this network
$\mathbf{h_{concept}}$ can be concatenated with any of the BERT based dual
encoder’s output($\mathbf{h_{d}}$).

Figure 2: Overall Training Architecture. Though the BERT based dual encoder is
independently depicted but it is trained along with concept-flow
interaction(if included).

Where $p_{1}p_{2}...p_{i}$ are the personalities of the speaker, $[EOP]$ token
denotes end of personality representation, $u_{1},u_{2},..u_{i}$ are the
utterances in the context. The resultant tokens $x_{CRAi}$ are passed through
bert-base-uncased, the hidden states of last $k$ layers i.e.
$\\{\mathbf{h_{c1}^{(l)},h_{c2}^{(l)},..h_{cT}^{(l)}}\\}$, for $l=1,2,..k$ are
used in downstream tasks.

Interaction Layer :  Since we are using a multi-encoder pipeline, it is
crucial to capture the interaction between the encoders. For that, we use
multi-head attention between hidden states of speaker context encoder and
BERT-CRA. For ease of presentation, we denote the whole multi-headed attention
layer as $f_{mha}(*,*)$. Then these attention outputs are passed through an
aggregation layer, which basically concatenates then passes it through a two-
layer feed-forward network and finally mean pools across all the layers to get
$\mathbf{h_{d}}$. The output is passed through a $MLP$ to get the matching
degree with the response.

$\displaystyle\\{\mathbf{\widetilde{h}_{s1}^{(l)},\widetilde{h}_{s2}^{(l)},..\widetilde{h}_{sT}^{(l)}}\\}$
$\displaystyle=$ $\displaystyle
f_{mha}(\\{\mathbf{h_{s1}^{(l)},h_{s2}^{(l)},..h_{sT}^{(l)}}\\},$ (3)
$\displaystyle\\{\mathbf{h_{c1}^{(l)},h_{c2}^{(l)},..h_{cT}^{(l)}}\\})$
$\displaystyle\\{\mathbf{\widetilde{h}_{c1}^{(l)},\widetilde{h}_{c2}^{(l)},..\widetilde{h}_{cT}^{(l)}}\\}$
$\displaystyle=$ $\displaystyle
f_{mha}(\\{\mathbf{h_{c1}^{(l)},h_{c2}^{(l)},..h_{cT}^{(l)}}\\},$ (4)
$\displaystyle\\{\mathbf{h_{s1}^{(l)},h_{s2}^{(l)},..h_{sT}^{(l)}}\\})$
$\displaystyle\mathbf{h_{d}}$ $\displaystyle=$
$\displaystyle\mathrm{MeanPool}(\\{\mathrm{FFN}([\\{\mathbf{\widetilde{h}_{s1}^{(l)},..\widetilde{h}_{sT}^{(l)}}\\};$
(5)
$\displaystyle\\{\mathbf{\widetilde{h}_{c1}^{(l)},..\widetilde{h}_{cT}^{(l)}}\\}])\\}_{l=0}^{k})$

Figure 3: Interaction Layer

Loss Function:  The MLP layer predicts whether a context-persona $(C,p)$ pair
matches with the corresponding response $r$ based on the derived features.
Subsequently, the output from the MLP layer is passed through a softmax output
layer to return a probability distribution over all response candidates. All
the models described in this paper are trained using MLP cross-entropy loss.
Let $\Theta$ be the model parameters, then the loss function
$\mathcal{L}(D,\Theta)$ for all the models can be formulated as follows:

$\mathcal{L}(D,\Theta)=-\sum_{(C,p,r,y))\epsilon D}ylog(g(C,p,r))$ (6)

####  5.3.2  BERT-EmA Emotion Aware Fusion:

In this strategy, an emotion incorporation framework is introduced. Similar to
BERT-CRA a dual pipeline matching network is followed. The first pipeline
encodes the speaker’s and listener’s emotional and personality characteristics
in the context. While the other encodes the bot-context as described in
section 5.2.

We attach the most probable emotion tag to each utterance to incorporate
emotion features in the BERT contextual representation. The emotion-infused
context representation is then concatenated with the original persona
representation, as described in section 5.3.1. The main goal of representing
the context in this way is to understand how the emotions of each utterance
interact with the speaker’s persona. The input to the emotion encoder is as
follows:

$\displaystyle x_{EmAi}$ $\displaystyle=$
$\displaystyle[CLS]p_{1}p_{2}...p_{i}[EOP][Emo_{1}]u_{1}[EOU]$ (7)
$\displaystyle...[Emo_{i}]u_{i}[EOU][SEP]r_{i}[EOU]$

Similar to baseline, the hidden states of last $k$ layers i.e.
$\\{\mathbf{h_{e1}^{(l)},h_{e2}^{(l)},..h_{eT}^{(l)}}\\}$, for $l=1,2,..k$ are
used in downstream tasks.

####  5.3.3  BERT-EnA-P: Entailment Aware Fusion

In this fusion strategy, the intention is to model the entailment information
about each of the utterances and personas with the response. Like BERT-EmA we
follow a dual encoder pipeline, the first encodes the entailment features, and
the second encodes the bot-context. To incorporate entailment features into
BERT contextual representation, we attach entailment tags i.e.
<contradiction>, <entailment> and <neutral> at the start of every utterance
and persona. The response is concatenated with the context-entailment
representation with a $[SEP]$ token. The input to the entailment encoder is as
follows:

$\displaystyle x_{EmA-Pi}$ $\displaystyle=$
$\displaystyle[CLS][Entail_{p1}]p_{1}...[EOP]$ (8)
$\displaystyle[Entail_{1}]u_{1}[EOU]$ $\displaystyle[Entail_{2}]u_{2}[EOU]$
$\displaystyle...[Entail_{i}]u_{i}[EOU]$ $\displaystyle[SEP]r_{i}[EOU]$

The hidden states of last $k$ layers i.e.
$\\{\mathbf{h_{en1}^{(l)},h_{en2}^{(l)},..h_{enT}^{(l)}}\\}$, for $l=1,2,..k$
are used in downstream tasks.

Finally, we experiment with a combined pipeline as depicted in Figure 2(a).

###  5.4 Concept-Flow(CF) Interaction

In section 4.3, we describe how we extract relevant concepts from the context
and the response. An appropriate response often has concepts most recently
discussed in the context. So, to model that, we construct a concept-flow
interaction network, where the interaction between the context-concepts and
response-concepts are measured and used as a feature in response relevance
classification.

Let us consider $\left\\{{CC_{1},CC_{2},...,CC_{n}}\right\\}$ are concepts
extracted from context and $\left\\{{RC_{1},RC_{2},...,RC_{n}}\right\\}$ are
concepts extracted from a response. Now, we pass each of these concepts
through a transformer-based concept encoder $f_{c}$ to get two sets of concept
embeddings $\left\\{\mathbf{ec_{1},ec_{2},...,ec_{n}}\right\\}$ ,
$\mathbf{ec_{i}}\in\mathbb{R}^{d_{c}}$ and
$\left\\{\mathbf{rc_{1},rc_{2},...,rc_{n}}\right\\}$ ,
$\mathbf{rc_{i}}\in\mathbb{R}^{d_{c}}$ for context and response concepts
respectively. To learn the context flow representation for each set of
concepts, we apply a bi-directional GRU network to capture sequential
dependencies between subsequent concepts in a conversational situation.
Context-concept and response-concept representation $\mathbf{h_{i}^{cc}}$ ,
$\mathbf{h_{i}^{rc}}$ can be formulated as:

$\displaystyle\mathbf{c_{i}^{cc},h_{i}^{cc}}$ $\displaystyle=$
$\displaystyle\overleftrightarrow{GRU}(\mathbf{ec_{i},h_{i-1}^{cc}})$ (9)
$\displaystyle\mathbf{c_{i}^{rc},h_{i}^{rc}}$ $\displaystyle=$
$\displaystyle\overleftrightarrow{GRU}(\mathbf{er_{i},h_{i-1}^{rc}})$ (10)
$\displaystyle\mathbf{h_{cc}}$ $\displaystyle=$ $\displaystyle
tanh(\mathbf{\sum_{j\in 2*N_{l}}W_{j}h_{j}^{cc}})$ (11)
$\displaystyle\mathbf{h_{rc}}$ $\displaystyle=$ $\displaystyle
tanh(\mathbf{\sum_{j\in 2*N_{l}}W_{j}h_{j}^{rc}})$ (12)

Where $\mathbf{h_{i}^{cc}}\in\mathbb{R}^{2d_{c}}$ ,
$\mathbf{h_{i}^{rc}}\in\mathbb{R}^{2d_{c}}$ are the $i$ \- the hidden states
and $\mathbf{c_{i}^{cc}}\in\mathbb{R}^{2d_{c}}$ ,
$\mathbf{c_{i}^{rc}}\in\mathbb{R}^{2d_{c}}$ are the outputs of the respective
GRU encoders, $\mathbf{W_{j}}$ is a learn-able parameter and $N_{l}$ is the
number of layers in each GRUs. To model the interaction between
$\mathbf{h_{i}^{cc}}$ and $\mathbf{h_{i}^{rc}}$ we follow the same interaction
mechanism described in the earlier section. The output $\mathbf{h_{concept}}$
is concatenated with the dual encoder output $\mathbf{h_{d}}$ before passing
it through an MLP.

Model | Self Persona | Partner Persona
---|---|---
Original | Revised | Original | Revised
hits@1 | MRR | hits@1 | MRR | hits@1 | MRR | hits@1 | MRR
FT-PC Mazaré et al. (2018) | - | - | 60.7 | - | - | - | - | -
DIM Gu et al. (2019b) | 78.8 | 86.7 | 70.7 | 81.2 | 64.0 | 76.1 | 63.9 | 76.0
TransferTransfo Wolf et al. (2019) | 80.7 | - | - | - | - | - | - | -
FIRE Gu et al. (2020c) | 81.6 | - | 74.8 | - | - | - | - | -
BERT-CRA Gu et al. (2021b) | 84.3 | 90.3 | 79.4 | 86.9 | 71.2 | 80.9 | 71.8 | 81.5
Baseline | 84.4 | 90.7 | 79.4 | 87.6 | 71.2 | 81.1 | 71.4 | 81.5
BERT-EmA | 84.6 | 90.9 | 79.8 | 87.7 | 71.4 | 81.2 | 71.4 | 81.6
BERT-P-EnA | 85.3 | 91.2 | 80.5 | 87.9 | 71.7 | 81.3 | 71.3 | 81.4
BERT-EmA+BERT-P-EnA | 85.8 | 91.4 | 80.7 | 88.0 | 72.3 | 81.5 | 71.7 | 81.5
BERT-EmA+BERT-P-EnA+CF (All) | 86.6* | 91.6* | 81.3* | 88.6* | 72.6* | 81.9* | 72.4* | 81.9*

Table 2: Performance of the proposed and previous methods on the Persona-Chat
dataset under different persona configurations. The meanings of "Self
Persona", "Partner Persona", "Original", and "Revised" can be found in Section
3. "-" represents that the results were not reported in their papers. Numbers
marked with * denote that the improvement over the best performing baseline is
statistically significant (t-test with p-value < 0.05). Numbers in bold denote
the combined fusion strategy that achieves the best performance.

##  6 Experimental Setup

###  6.1 Training Details

The ratio of positive to negative samples in the training set is 1:19, so
there is a high imbalance in training data. Taking inspiration from Gu et al.
(2021b) we adopted a dynamic negative sampling strategy in which the ratio of
positive and negative responses is 1:1 in an epoch. We keep the positive
response constant and change the negative response for every epoch, generating
data for 19 epochs. We use bert-base-uncased as the base for each of our
pretraining-based fusion models. In concept mining strategy, we have taken the
top 3 concepts extracted using RAKE, $\lambda$ for PMI-based scoring was
varied from 0.3 to 0.8 with 0.1 steps, and 0.5 was found optimum. The number
of turns in the conversation history used for concept mining varied following
this set: $\\{2,3,4,5,6,7\\}$. We preserve the original parameters of bert-
base-uncased. The number of $k$-last layers in the interaction layer varied
following this set: $\\{3,4,5,6\\}$; after some initial experimentation, 4 was
found as the optimum value. The number of heads in the multi-head attention
layer was kept 8. We use a 6-layered version MiniLMWang et al. (2020) to
encode the concepts; the embedding dimension was 384. The number of layers in
the bi-directional GRUs in the concept encoder is 2. A dropout rate of 0.7 is
applied to the concept encoder hidden representation before we send it to the
interaction layer. AdamWLoshchilov and Hutter (2019) optimizer was used for
optimization. The initial learning rate was set to 2e-5 and linearly decayed
by L2 weight decay. The maximum sequence length was set to 320. The training
batch size was 12. The relevance prediction head used a single feed-forward
layer with sigmoid activation. All code was implemented using the PyTorch
framework. Also, we used 2 NVIDIA RTX A5000 GPUs to train the models. The
average training time for one epoch was 46 minutes, using all our fusion
strategies and concept encoding.

###  6.2 Evaluation Metrics

We used the same evaluation metrics as the previous work to ensure comparable
results. Each model aimed to select the best-matched response from available
candidates for the given context and persona. We calculated the recall of the
true positive replies, denoted as hits@1. In addition, the mean reciprocal
rank (MRR) was also adopted to take the rank of the correct response overall
candidates into consideration.

###  6.3 Comparison Methods

For comparison, we have only selected pretraining-based models.

  * •

FT-PC Mazaré et al. (2018): employed the “pretrain and fine-tune” framework by
first pretraining on a domain-specific corpus, dialogues of which were
extracted from Reddit, and then fine-tuning on the Persona-Chat.

  * •

DIM Gu et al. (2019b): used a dually matching network (DIM) which performs
interactive matching between responses and contexts and between responses and
personas respectively for ranking response candidates.

  * •

TransferTransfo Wolf et al. (2019): the paper fine-tunes a transformer
model(GPT) using Persona-Chat dataset on a multi-task objective which combines
several unsupervised task.

  * •

BERT-CRA Gu et al. (2021b): This work presents four context-aware persona
fusion strategies and the models are initialized and pretrained using BERT on
Persona-Chat dataset.

###  6.4 Experimental Results

Table 2 reports the evaluation results of our proposed and previous methods on
Persona-Chat under various persona configurations. We can see that
incorporating the emotion and entailment knowledge of the utterances coupled
with generic distributional semantics and external knowledge learned from
pretraining rendered improvements on both hits@1 and MRR conditioned on
various personas. Compared to FT-PC Mazaré et al. (2018), our best model
outperformed it by 20.4 % in terms of hits@1 conditioned emotion, entailment
and concepts. Compared to TransferTransfo Wolf et al. (2019), which was also
trained using pretrained transformer models, our combined model outperformed
it, which shows the effectiveness of fusion strategies and the concept-
encoder. Lastly, our combined model outperformed the BERT-CRA Gu et al.
(2021b) in all the tasks. We see a 2.3 % and 1.9 % improvement in original and
revised self-persona, and 1.4 % and 0.6 % improvement in original and revised
partner-persona in terms of hits@1. The results bolster our hypothesis that
emotion, entailment, and concepts play an important role in the task of
response selection. Also, it is to be noted that Persona-Chat is a synthetic
dataset, i.e., the data collection did not happen naturally. Therefore, the
chances that the user will display this subtle interplay of persona and
emotion is less. In addition to that, we observe the presence of contradictory
distractor responses. We see a significant performance improvement from this
information by introducing entailment-aware fusion and concept encoding.

###  6.5 Human Evaluation

Figure 4: Human evaluation results on Persona-Chat self-persona original test
split

Since a qualitative study by humans is necessary to understand the
effectiveness of the proposed methods, we further perform a human evaluation
on a portion of the data. We randomly sampled 100 inferred examples from the
test set by baseline model, combined model, combined model except for the
emotion, and combined model except for the concept flow interaction. We
combined all the samples and evaluated them using Amazon Mechanical Turk by 2
different turkers on three metrics: emotion inappropriate, contradictory, and
off-topic. The turkers needed to select if any of the three issues were
present in an example. The percentages of reported issues by both groups are
shown in Figure 4. The results reveal that all our encoding pipelines
effectively reduce contradictory responses and somewhat reduce off-topic and
emotionally inappropriate responses. The agreement between the two groups was
moderate (Krippendorff’s $\alpha=0.713$).

##  7 Analysis

###  7.1 Ablation Study for Emotion and Entailment

We perform ablation studies(shown in Table 3) to validate the effectiveness of
emotion and entailment fusion in our proposed models. We see a slight
improvement in our baseline model that uses our proposed speaker embedding.
Also, unsurprising that the effect of emotion is not significant. As the
dataset is artificially created, and emotions exhibited by the annotators are
not always true. However, some performance improvement is observed.
Conditioning persona in entailment fusion improves performance considerably as
responses may not entail the speaker’s persona.

Models | hits@1 | MRR
---|---|---
Baseline | 84.4 | 90.7
BERT-EmA($-$ Speaker Encoding) | 84.5 | 90.8
BERT-EmA | 84.6 | 90.9
BERT-EnA-P | 85.3 | 91.2

Table 3:  Ablation Study for Emotion and Entailment on self-original persona.

###  7.2 Effect of Context Turns on Concept Representation

Concept matching boosts the evaluation performance further. However, the
number of turns in the conversation history from which we mine the concepts
influences the performances. It is evident from Figure 5 that the essential
concepts in the most relevant response will be present in the recent
conversation history.

Figure 5: This graph shows how hit@1 reaches an optimum value and then
decreases with an increase in the number of turns used to mine concepts.

###  7.3 Case Study

personas |  my favorite color is blue. <ent: neutral> I enjoy reading mysteries. <ent: neutral> I have seven children. <ent: entail> I grew up on a large farm. <ent: neutral>
---|---
context |  A: hello how are you today? <emo:curiosity> <ent: neutral> B: I am well. how are you? <emo:curiosity> <ent: neutral> A: I am doing great just got back from the beach <emo:excitement> <ent: neutral> B: that is great. I live far from the beach. <emo:caring> <ent: neutral> A: I am very lucky we live beside the beach. what do you do for a living? <emo:curiosity> <ent: neutral> B: I keep busy with my seven children. <emo:excitement> <ent: neutral> A: wow that much have taken some adjusting I teach kindergarten. <emo:surprise> <ent: neutral>
| golden
---
response
do you teach mysteries to your children ? they are my favorite type of novel .
<emo:curiosity>
BERT-CRA |  that must be a lot of work but very rewarding i bet <emo:realization>
| All
---
do you teach mysteries to your children ? they are my favorite type of novel .
<emo:curiosity>

Table 4:  Case study showing concept flow.

Table 4 shows the efficacy of concept-encoding. Sometimes models fine-tuned on
pretrained transformer models like BERT-CRA tends to select more generic
responses. These models pay less attention to the persona or specific keywords
in the context while selecting responses. In this example, our proposed model
performs better than BERT-CRA as it is conditioned on the concepts.
Specifically, concepts in the correct response i.e "mysteries", "novel"
relates to "reading mysteries" concept in the persona and "your children"
relates to "teach kindergarten" in the context.

##  8 Conclusion

This work proposes a suite of novel fusion strategies and concept-flow
encoder, which leverages the utterances’ emotion, entailment, and concept
information. These features help improve the performances of our models and
provide critical insights into certain aspects of how humans communicate with
each other. Though the techniques used in this paper are simple, it highlights
the areas where response selection often falters, like detecting contraction,
deviation from the concepts, etc. This work can be further extended by using a
graphical model to improve the concept representations.

## Acknowledgements

We thank the anonymous reviewers for providing valuable feedback on our
manuscript. This work is partly supported by NSF grant number IIS-2214070. The
content in this paper is solely the responsibility of the authors and does not
necessarily represent the official views of the funding entity.

## References

  * Chen et al. (2021) Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021.  Improving faithfulness in abstractive summarization with contrast candidate generation and selection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5935–5941, Online. Association for Computational Linguistics.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  Goemotions: A dataset of fine-grained emotions.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Gu et al. (2021a) Jia-Chen Gu, Tianda Li, Zhen-Hua Ling, Quan Liu, Zhiming Su, Yu-Ping Ruan, and Xiaodan Zhu. 2021a.  Deep contextualized utterance representations for response selection and dialogue analysis.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:2443–2455.
  * Gu et al. (2020a) Jia-Chen Gu, Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. 2020a.  Speaker-aware bert for multi-turn response selection in retrieval-based chatbots.
  * Gu et al. (2020b) Jia-Chen Gu, Tianda Li, Quan Liu, Xiaodan Zhu, Zhen-Hua Ling, Zhiming Su, and Si Wei. 2020b.  Speaker-aware BERT for multi-turn response selection in retrieval-based chatbots.  _CoRR_ , abs/2004.03588.
  * Gu et al. (2019a) Jia-Chen Gu, Zhen-Hua Ling, and Quan Liu. 2019a.  Utterance-to-utterance interactive matching network for multi-turn response selection in retrieval-based chatbots.
  * Gu et al. (2019b) Jia-Chen Gu, Zhen-Hua Ling, Xiaodan Zhu, and Quan Liu. 2019b.  Dually interactive matching network for personalized response selection in retrieval-based chatbots.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1845–1854, Hong Kong, China. Association for Computational Linguistics.
  * Gu et al. (2020c) Jia-Chen Gu, Zhenhua Ling, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2020c.  Filtering before iteratively referring for knowledge-grounded response selection in retrieval-based chatbots.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1412–1422, Online. Association for Computational Linguistics.
  * Gu et al. (2021b) Jia-Chen Gu, Hui Liu, Zhen-Hua Ling, Quan Liu, Zhigang Chen, and Xiaodan Zhu. 2021b.  Partner matters! an empirical study on fusing personas for personalized response selection in retrieval-based chatbots.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 565–574, New York, NY, USA. Association for Computing Machinery.
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring.
  * Leary and Allen (2011) Mark R. Leary and Ashley Batts Allen. 2011.  Personality and persona: Personality processes in self-presentation.  _Journal of Personality_ , 79(6):1191–1218.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016.  A persona-based neural conversation model.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 994–1003, Berlin, Germany. Association for Computational Linguistics.
  * Liu et al. (2020) Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020.  You impress me: Dialogue generation via mutual persona perception.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.
  * MacCartney and Manning (2008) Bill MacCartney and Christopher D. Manning. 2008.  Modeling semantic containment and exclusion in natural language inference.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 521–528, Manchester, UK. Coling 2008 Organizing Committee.
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018\.  Training millions of personalized dialogue agents.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2775–2779, Brussels, Belgium. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2020.  I like fish, especially dolphins: Addressing contradictions in dialogue modeling.
  * Pollock et al. (2016) Noah C. Pollock, Gillian A. McCabe, Ashton C. Southard, and Virgil Zeigler-Hill. 2016.  Pathological personality traits and emotion regulation difficulties.  _Personality and Individual Differences_ , 95:168–177.
  * Rashkin et al. (2021) Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021.  Increasing faithfulness in knowledge-grounded dialogue with controllable features.
  * Richendoller and Weaver III (1994) Nadine R Richendoller and James B Weaver III. 1994.  Exploring the links between personality and empathic response style.  _Personality and individual Differences_ , 17(3):303–311.
  * Rose et al. (2010) Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. 2010.  _Automatic Keyword Extraction from Individual Documents_, chapter 1. John Wiley & Sons, Ltd.
  * Su et al. (2021) Yixuan Su, Deng Cai, Qingyu Zhou, Zibo Lin, Simon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2021.  Dialogue response selection with hierarchical curriculum learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1740–1751, Online. Association for Computational Linguistics.
  * Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric P. Xing, and Zhiting Hu. 2019.  Target-guided open-domain conversation.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019.  Transfertransfo: A transfer learning approach for neural network based conversational agents.
  * Wu et al. (2021) Yuwei Wu, Xuezhe Ma, and Diyi Yang. 2021.  Personalized response generation via generative split memory network.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1956–1970, Online. Association for Computational Linguistics.
  * Zhang et al. (2021a) Chen Zhang, Hao Wang, Feijun Jiang, and Hongzhi Yin. 2021a.  Adapting to context-aware knowledge in natural conversation for multi-turn response selection.  In _Proceedings of the Web Conference 2021_ , WWW ’21, page 1990–2001, New York, NY, USA. Association for Computing Machinery.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?
  * Zhang et al. (2021b) Zhuo Zhang, Danyang Zheng, and Ping Gong. 2021b.  Multi-turn response selection in retrieval based chatbots with hierarchical residual matching network.  _Journal of Physics: Conference Series_ , 1757(1):012023.
  * Zhong et al. (2020) Peixiang Zhong, Chen Zhang, Hao Wang, Yong Liu, and Chunyan Miao. 2020.  Towards persona-based empathetic conversational models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6556–6566, Online. Association for Computational Linguistics.

##  Appendix A Appendix

###  A.1 Human Evaluation for Initial Study

Five hundred context response pairs randomly sampled from the Persona-Chat
self-original test split inferred by BERT-CRA were evaluated by at least two
AMT workers. The following questions were asked to the workers:

  1. 1.

Is this response contain emotions that are consistent with the context? (Most
definitely/ not at all)

  2. 2.

Is this response contradicts the context? (Most definitely/ not at all)

  3. 3.

Do the topics discussed in this response appropriate to the topics discussed
in the context? (Most definitely/ not at all)

###  A.2 Final Human Evaluation

Same evaluation pattern is followed as A.1, average pay for each HIT was 0.07
$.


# Domain Adaptive Fake News Detection via Reinforcement Learning

Ahmadreza Mosallanezhad<EMAIL_ADDRESS>Arizona State UniversityTempeAZUSA ,
Mansooreh Karami<EMAIL_ADDRESS>Arizona State UniversityTempeAZUSA ,  Kai
Shu<EMAIL_ADDRESS>Illinois Institute of TechnologyChicagoILUSA ,  Michelle V.
Mancenido<EMAIL_ADDRESS>Arizona State UniversityGlendaleAZUSA and  Huan
Liu<EMAIL_ADDRESS>Arizona State UniversityTempeAZUSA

(2022)

###### Abstract.

With social media being a major force in information consumption, accelerated
propagation of fake news has presented new challenges for platforms to
distinguish between legitimate and fake news. Effective fake news detection is
a non-trivial task due to the diverse nature of news domains and expensive
annotation costs. In this work, we address the limitations of existing
automated fake news detection models by incorporating auxiliary information
(e.g., user comments and user-news interactions) into a novel reinforcement
learning-based model called REinforced Adaptive Learning Fake News Detection
(REAL-FND). REAL-FND exploits cross-domain and within-domain knowledge that
makes it robust in a target domain, despite being trained in a different
source domain. Extensive experiments on real-world datasets illustrate the
effectiveness of the proposed model, especially when limited labeled data is
available in the target domain.

neural networks, reinforcement learning, domain adaptation, disinformation

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
ACM Web Conference 2022; April 25–29, 2022; Virtual Event, Lyon,
France††booktitle: Proceedings of the ACM Web Conference 2022 (WWW ’22), April
25–29, 2022, Virtual Event, Lyon, France††price: 15.00††doi:
10.1145/3485447.3512258††isbn: 978-1-4503-9096-5/22/04††submissionid:
w12fp2892††ccs: Applied computing Document analysis††ccs: Information systems
Social networks††ccs: Computing methodologies Machine learning approaches

##  1\. Introduction

With people spending more time on social media platforms111In 2020, internet
users spent an average of 145 minutes per day on social media (by Statista
Research Department and 7, 2021)., it is not surprising that social media has
evolved into the primary source of news among subscribers, in lieu of the more
traditional news delivery systems such as early morning shows, newspapers, and
websites affiliated with media companies. For example, it was reported that 1
in 5 U.S. adults used social media as their primary source of political news
during the 2020 U.S. presidential elections (Mitchell and Jurkowitz, 2020).

(a) Source Domain - Healthcare

(b) Target Domain - Politics

Figure 1. Comparing two domains, Healthcare and Politics, in fake news
detection: (1) Different word usage, (2) Event-related words/features, (3)
Different user-news interaction.

The ease and speed of disseminating new information via social media, however,
have created networks of disinformation that propagate as fast as any social
media post. For example from the COVID pandemic era, around 800 fatalities,
5000 hospitalizations, and 60 permanent injuries were recorded as a result of
false claims that household bleach was an effective panacea for the SARS-CoV-2
virus (Islam et al., 2020; Coleman, 2020). Unlike traditional news delivery
where trained reporters and editors fact-check information, curation of news
on social media has largely been crowd-sourced, i.e., social media users
themselves are the producers and consumers of information.

In general, humans have been found to fare worse than machines at prediction
tasks (Schuster et al., 2020; Shabani and Sokhn, 2018) such as distinguishing
between fake or legitimate news. Machine learning models for automated fake
news detection have even been shown to perform better than the most seasoned
linguists (Pérez-Rosas et al., 2018). To this end, many automated fake news
detection algorithms have largely focused on improving predictive performance
for a specific news domain (e.g., political news). The primary issue with
these existing state-of-the-art detection algorithms is that while they
perform well for the domain they were trained on (e.g., politics), they
perform poorly in other domains (e.g., healthcare). The limited cross-domain
effectiveness of algorithms to detect fake news are mostly due to (1) the
reliance of content-based approaches on word usage that are specific to a
domain, (2) the model’s bias towards event-specific features, and (3) domain-
specific user-news interaction patterns (Figure 1). As one of the
contributions of this work, we will empirically demonstrate that the
advertised performance of SOTA methods is not robust across domains.

Additionally, due to the high cost and specialized expertise required for data
annotation, limited training data is available for effectively training an
automated model across domains. This calls for using auxiliary information
such as users’ comments and motivational factors (Shu et al., 2019b; Karami et
al., 2021) as value-adding pieces for fake news detection.

To address these challenges, we propose a domain-adaptive model called
REinforced Adaptive Learning Fake News Detection (REAL-FND), which uses
generalized and domain-independent features to distinguish between fake and
legitimate news. The proposed model is based on previous evidence that
illustrates how domain-invariant features could be used to improve the
robustness and generality of fake news detection methods. As an example of a
domain-invariant feature, it has been shown that fake news publishers use
click-bait writing styles to attract specific audiences (Zhou et al., 2020).
On the other hand, patterns extracted from the social context provide rich
information for fake news classification within a domain. For example, a
user’s comment providing evidence in refuting a piece of news is a valuable
source of auxiliary information (Shu et al., 2019a). Or, if a specific user is
a tagged fake news propagator, the related user-news interaction could be
leveraged as an additional source of information (Karami et al., 2021).

In REAL-FND, instead of applying the commonly-used method of adversarial
learning in training the cross-domain model, we transform the learned
representation from the source to the target domain by deploying a
reinforcement learning (RL) component. Other RL-based methods employ the agent
to modify the parameters of the model. However, we use the RL agent to modify
the learned representations to ensure that domain-specific features are
obscured while domain-invariant components are maintained. An RL agent
provides more flexibility over adversarial training because any classifier’s
confidence values could be directly optimized (i.e., the objective function
does not need to be differentiable).

We address the challenges in domain adaptive fake news detection algorithms by
making the following contributions:

  * •

We design a framework that encodes news content, user comments, and user-news
interactions as representation vectors and fuses these representations for
fake news detection;

  * •

We utilize a reinforcement learning setting to adjust the representations to
account for domain-independent features;

  * •

We conduct extensive experiments on real-world datasets to show the
effectiveness of the proposed model, especially when limited labeled data is
available in the target domain.

##  2\. Related Work

The proposed methodology spans the subject domains of fake news detection,
cross domain modeling, and the use of reinforcement learning in the Natural
Language Processing (NLP). Current state-of-the-art in these areas are
discussed in this section.

###  2.1. Fake News Detection

Early research in fake news detection focused on extracting features from news
content to either capture the textual information (such as lexical and
syntactic features) (Shu et al., 2017; Shu et al., 2020a) or to detect the
differences in the writing style of real and fake news such as deception (Feng
et al., 2012) and non-objectivity (Potthast et al., 2017). Qian et al. propose
to use convolutional neural network to detect fake news based on news content
only (Qian et al., 2018). Guo et al. propose to capture the sensational
emotions to create an emotion-enhanced news representation to detect fake news
(Guo et al., 2019). Recent advancements utilize deep neural networks (Karimi
and Tang, 2019) or tensor factorization (Hosseinimotlagh and Papalexakis,
2018) to model latent news representations.

While news content-based approaches result in acceptable performance,
incorporating auxiliary information improves the performance and reliability
of fake news detection models (Shu et al., 2019a). For example, Tacchini et
al. (Tacchini et al., 2017) and Guo et al. (Guo et al., 2018) aggregated
users’ social responses (topics, stances, and credibility), Ruchansky et al.
proposed a model called CSI which uses a hybrid model on news content and
users network to detect fake news (Ruchansky et al., 2017), and Shu et al.
used hierarchical attention networks to create more explainable fake news
detection models (Shu et al., 2019a). In this work, we incorporate various
auxiliary information (i.e., users’ comments and user-news interactions) along
with the news content to create a well-defined news article representation.

###  2.2. Cross Domain Modeling

Cross-domain modeling refers to a model capable of learning information from
data in the source domain and being able to transfer it to a target domain
(Ben-David et al., 2010). In general, cross-domain models are categorized into
sample-level and feature-level groups (Zhuang et al., 2015). Sample-level
domain adaptation methods focus on finding domain-independent samples by
assigning weights to these instances (Zhuang et al., 2015; Silva et al.,
2021). On the other hand, feature-level domain adaptation methods focus on
weighting or extracting domain-independent features (Kouw et al., 2016). In
addition to the aforementioned domain adaptation methods, Gong et al. combined
both sample-level and feature-level domain adaptation (Gong et al., 2020) in
BERT to create a domain-independent sentiment analysis model. Similarly, Vlad
et al. used transfer learning on an enhanced BERT architecture to detect
propaganda across domains (Vlad et al., 2019). Moreover, Zhuang et al. propose
to use auto-encoders for learning unsupervised feature representations for
domain adaptation (Zhuang et al., 2015). The goal of this model is to leverage
a small portion of the target domain data to train an auto-encoder for
learning domain-independent feature representations. In this paper, we focus
on feature-level domain adaptation by leveraging reinforcement learning to
learn a domain-independent textual representation.

###  2.3. Reinforcement Learning

In past years, reinforcement learning has shown to be useful in improving the
effectiveness of NLP models. As such, Li et al. propose a method for
generating paraphrases using inverse reinforcement learning (Li et al., 2017).
Another work by Fedus et al. uses reinforcement learning to tune parameters of
an LSTM-based generative adversarial network for text generation (Fedus et
al., 2018). A recent work by Cheng et al. proposes to use reinforcement
learning for tuning a classifier’s parameters for removing various types of
biases (Cheng et al., 2021). Inspired by these methods, we study the problem
of domain-adaptive fake news detection using reinforcement learning.

In previous research, although reinforcement learning has been used to modify
a model’s parameters, little research has been done on applying it to modify
the learned representations. In this work, we focus on designing a domain
adaptation model with an RL agent that modifies the article’s representation
based on the feedback received from both the adversary domain classifier and
the fake news detection component.

##  3\. Problem Statement

Let $\mathcal{D}_{s}=\\{$ $(\mathbf{x}_{1}^{s},y_{1}^{s})$,
$(\mathbf{x}_{2}^{s},y_{2}^{s})$, $...$, $(\mathbf{x}_{N}^{s},y_{N}^{s})\\}$
and $\mathcal{D}_{t}=\\{$ $(\mathbf{x}_{1}^{t},y_{1}^{t})$,
$(\mathbf{x}_{2}^{t},y_{2}^{t})$, $...$, $(\mathbf{x}_{M}^{t},y_{M}^{t})\\}$
denote a set of $N$ and $M$ news article with labels from source and target
domains, respectively. Each news article $\mathbf{x}_{i}$ includes a content
which is a sequence of $K$ words $\\{w_{1},w_{2},...,w_{K}\\}$, a set of
comments $\mathbf{c}_{j}\in\mathcal{C}$, and user-news interactions
$\mathbf{u}_{j}\in\mathcal{U}$. User-news interactions $\mathbf{u}_{i}$ is a
binary vector indicating users who posted, re-posted, or liked a tweet about
news $\mathbf{x}_{i}$. The goal of the reinforced domain adaptive agent is to
learn a function that converts the news representation
$\mathbf{x}_{i}\in\mathcal{D}_{t}$ from target domain to source domain. The
problem is formally defined as follows:

Figure 2. The proposed REAL-FND architecture. REAL-FND has two main
components, the representation network, and the reinforcement learning agent.
The representation network learns a representation including information about
news content, comments, and user-news interactions. The RL agent learns a
method to transfer the learned representation from source domain to the target
domain.

Definition (Domain Adaptive Fake News Detection). Given news articles from two
separate domains $\mathcal{D}_{s}$ and $\mathcal{D}_{t}$, corresponding users’
comments $\mathcal{C}_{s}$ and $\mathcal{C}_{t}$, and user-news interactions
$\mathcal{U}_{s}$ and $\mathcal{U}_{t}$ from the source $S$ and target $T$
domains, respectively, learn a domain-independent news article representation
using $\mathcal{D}_{s}$ and a small portion of $\mathcal{D}_{t}$ that can be
classified correctly by the fake news classifier $F$.

##  4\. Proposed Model

In this section, we describe our proposed model, REinforced domain Adaptation
Learning for Fake News Detection (REAL-FND). The input of this model is the
news articles from source domain $\mathcal{D}_{s}$ and a portion ($\gamma$) of
the target data set $\mathcal{D}_{t}$. As shown in Figure 2, the REAL-FND
model has two main components: (1) the news article encoder, and (2) the
reinforcement learning agent. In the following subsections, we explain these
two components in detail.

###  4.1. News Article Encoder

The problem of detecting fake news requires learning a comprehensive
representation that includes information about news content and its related
auxiliary information. In this paper, we consider news comments and user-news
interactions as auxiliary information for fake news detection. Recurrent
neural networks (RNN) such as Long Short Term Memory (LSTM) has been proven to
be effective in modeling sequential data (Shu et al., 2017). However,
Transformers, due to their attentive nature, create a better text
representation that includes vital information about the input in an efficient
manner (Vaswani et al., 2017). Thus, in this work, we use Bidirectional
Encoder Representations from Transformers (BERT) in creating the
representation vector for the news content.

BERT is a pre-trained language model that uses transformer encoders to perform
various NLP-based tasks such as text generation, sentiment analysis, and
natural language understanding. Previous research have used BERT to achieve
state-of-the-art performance on different applications of sentiment analysis
tasks (Vaswani et al., 2017; Xu et al., 2019; Li et al., 2020). Although BERT
has been pre-trained on a large corpus of textual data, it should be fine-
tuned to perform well on a specific task (Xu et al., 2019; Liu et al., 2019).
In this paper, to create a well-defined news content representation, we fine-
tune BERT using news articles $\mathbf{x}=\\{w_{1},w_{2},...,w_{K}\\}$ from
source domain dataset $\mathcal{D}_{s}$ and a portion $\gamma$ of the target
dataset $\mathcal{D}_{t}$.

In addition to the news content $\mathbf{x}_{i}$, we also consider the
article’s comments and user-news interactions. In the experiments, we show
that using auxiliary information leads to better detection performance as the
model accounts for user’s reliability and feedback on the news article.
Nonetheless, previous studies also have shown that comments on news articles
can improve fake news detection (Shu et al., 2019a) by extracting semantic
clues confirming or disapproving the authenticity of the content. Due to the
fact that not all comments are useful for fake news classification, we use
Hierarchical Attention Network (HAN) to encode the comments of a news article.
The hierarchical structure of HAN facilitates the importance of every comment,
as well as the salient word features. To create a representation for the news
article comments, we pre-train HAN by stacking it with a feed-forward neural
network classifier. After pre-training HAN, we remove the feed-forward
classifier and only use the HAN to encode the news article comments. Note that
in case a news article does not have any comments, we use vectors with zero
values for comments representation.

Moreover, in addition to the news article comments, we also consider the user-
news interactions to improve fake news detection. For a news article
$\mathbf{x}$, the user-news interactions $\mathbf{u}$ is a binary vector where
$\mathbf{u}_{i}$ indicates user $i$ has tweeted, re-tweeted, or commented on a
tweet about that news. Thus, the user-news interactions vector is a
representation of the user behaviour toward a news article. To encode this
information, we use a feed-forward neural network that takes the binary vector
of user-news interaction as input and returns a representation containing
important information about that interactions.

After constructing the representation networks (i.e., HAN, BERT, and the feed-
forward neural network) and pre-training BERT and HAN, we concatenate the
output of these three components into one vector
$E=(h^{\prime}_{comments}||h^{\prime}_{content}||h^{\prime}_{interactions})$
($||$ indicates concatenation) and pass it to another feed-forward network to
combine these information into a single vector $\mathbf{E^{\prime}}$ (Karami
et al., 2020). Once we stacked the representation network with a feed-forward
neural network classifier, we train a fake news classifier using the source
domain data $\mathcal{D}_{s}$ and a portion $\gamma$ of the target domain data
$\mathcal{D}_{t}$. After training the fake news classifier $F$, we freeze the
weights of the representation network and train a domain classifier $D$ using
the representation $\mathbf{E^{\prime}}$ of the news articles. By the end of
this process, we have created three sub-components: (1) a representation
network that encodes news content, comments, and user-news interactions, (2) a
single-domain fake news classifier, and (3) a domain classifier. In the
following section, we discuss the second component of the model (i.e., Figure
2-RL Setting) that uses reinforcement learning to create a domain adaptive
representation using $\mathbf{E^{\prime}}$.

###  4.2. Reinforced Domain Adaptation

Inspired by the success of RL (Kaelbling et al., 1996; Van Hasselt et al.,
2016; Mosallanezhad et al., 2019), we model this problem using RL to
automatically learn how to convert textual representations from both source
domain data $\mathcal{D}_{s}$ and target domain data $\mathcal{D}_{t}$.
Instead of applying a commonly-used approach of adversarial learning to train
both $F$ and $D$ classifiers, we utilized an RL-based technique. RL would
transform the representation to a new one such that it works well on the fake
news classifier $F$, but not on the $D$ classifier (i.e., the adversary). In
this approach, the agent interacts with the environment by choosing an action
$a_{t}$ in response to a given state $s_{t}$. The performed action results in
state $s_{t+1}$ with reward $r_{t+1}$. The tuple
$(s_{t},a_{t},s_{t+1},r_{t+1})$ is called an experience which will be used to
update the parameters of the RL agent.

In our RL model, an agent is trained to change the news article representation
$\mathbf{E^{\prime}}$ into a new representation that deceives the domain
classifier, but preserves the accuracy for the fake news classifier. The RL
agent learns this transition by changing the values in the input vector
$\mathbf{E^{\prime}}$ and receiving feedback from both fake news classifier
$F$ and domain classifier $D$. To create an RL setting, we define four main
parts of RL in our problem, i.e., environment, state, action, and reward.

  * •

Environment: the environment in our model includes the RL agent, pre-trained
fake news classifier $F$, and the pre-trained domain classifier $D$. In each
turn, the RL agent performs an action on the news article representation
$E^{\prime}$ (known as state $s_{t}$) by changing one of its values
(performing action $a_{t}$). The modified representation is passed through
classifiers $F$ and $D$ to get their confidence scores to calculate the reward
value. Finally, the reward value and the modified representation is passed to
the agent as reward $r_{t+1}$ and state $s_{t+1}$, respectively.

  * •

State: the state is the current news article representation
$\mathbf{E^{\prime}}$ that describes the current situation for the RL agent.

  * •

Actions: actions are defined as selecting one of the values in news article
representation $\mathbf{E^{\prime}}$ and changing it by adding or subtracting
a small value $\sigma$. The total number of actions are $|E^{\prime}|\times
2$.

  * •

Reward: since the aim is to nudge the agent into creating a domain adaptive
news article representation, the reward function looks into how much the agent
was successful in removing the domain-specific features from the input
representation. Specifically, we define the reward function at state $s_{t+1}$
according to the confidence of both the domain classifier and the fake news
classifier. Considering the news article embedding $\mathbf{E^{\prime}}_{t+1}$
with its label $i$ (being fake or real) and domain label $j$ at state
$s_{t+1}$, we formally define the reward function as follows:

(1) $r_{t+1}(s_{t+1})=\alpha Pr_{F}(l=i|\mathbf{E^{\prime}}_{t+1})-\beta
Pr_{D}(l=j|\mathbf{E^{\prime}}_{t+1})$

where $l$ indicates the label.

###  4.3. Optimization Algorithm

Given the RL setting, the aim is to learn the optimal action selection
strategy $\pi(s_{t},a_{t})$. Algorithm 1222The source code will become
publicly available upon acceptance. shows this optimization process. At each
timestep $t$, the RL agent changes one of the values in the news article
representation, $s_{t}=\mathbf{E^{\prime}}_{t}$, and gets the reward value,
$r_{t+1}$, based on the modified representation
$s_{t+1}=\mathbf{E^{\prime}}_{t+1}$ and the selected action $a_{t}$. The goal
of the agent is to maximize its reward according to Equation 1.

Algorithm 1 The Learning Process of REAL-FND

0: News article representations $\mathbf{E^{\prime}\in\mathcal{D}}$ \- fake
news classifier $F$ \- domain classifier $D$ \- parameters $\alpha,\beta$, and
$\lambda$ \- terminal time $T$.

1: Initialize state $s_{t}$ and memory $M$.

2: while training is not terminal do

3: $s_{t}\leftarrow\mathbf{E^{\prime}}$

4: for $t\in\\{0,1,...,T\\}$ do

5: Choose action $a_{t}$ according to current distribution $\pi(s_{t})$

6: Perform $a_{t}$ on $s_{t}$ and get $(s_{t+1},r_{t+1})$

7: $M\leftarrow M+(s_{t},a_{t},r_{t+1},s_{t+1})$

8: $s_{t}\leftarrow s_{t+1}$

9: for each timestep $t$, reward $r$ in $M_{t}$ do

10: $G_{t}\leftarrow\sum_{i=1}^{t}\lambda^{i}r_{i+1}$

11: end for

12: Calculate policy loss according to Equation 2

13: Update the agent’s policy according to Equation 3

14: end for

15: end while

To train the agent, we use the REINFORCE algorithm which uses policy gradient
to update the agent (Zhang et al., 2020). Considering the agent’s policy
according to parameters $\theta$ as $\pi_{\theta}(s_{t},a_{t})$, the REINFORCE
algorithm uses the following loss function to evaluate the agent:

(2) $\mathcal{L}(\theta)=\log(\pi_{\theta}(s_{t},a_{t})\cdot G_{t})$

where $G_{t}=\sum_{i=1}^{t}\lambda^{i}r_{i+1}$ is the cumulative sum of
discounted reward, and $\lambda$ indicates the discount rate. The gradient of
the loss function is used to update the agent:

(3) $\nabla\theta=\text{lr}\nabla_{\theta}\mathcal{L}(\theta)$

where $lr$ indicates the learning rate.

##  5\. Experiments

In the designed experiments, we try to understand how the differences between
domains affect the performance of fake news detection models. Moreover, we
investigate how the proposed RL-based approach will overcome performance
degradation due to the differences in news domains. Our main evaluation
questions are as follows:

  * •

Q1. How well does the current fake news detection methods perform on social
media data?

  * •

Q2. How well does the proposed model detect fake news on a target domain
$\mathcal{D}_{t}$ after training the model on a source domain
$\mathcal{D}_{s}$?

  * •

Q3. How do auxiliary information contribute to the improvement of the fake
news detection performance?

Q1 evaluates the quality of fake news detection models. We answer this
question by training and testing fake news detection models on the same domain
and comparing their performance. Q2 studies the effect of domain differences
on the performance of fake news detection models. We use a similar approach to
Q1 to answer this question by training on one domain, but testing on another.
In Q3 we perform ablation studies to analyze the impact of auxiliary
information, the reward function parameters $\alpha$ and $\beta$, and the
portion of target domain data $\gamma$ needed to achieve an acceptable
detection performance.

###  5.1. Datasets

We use the well-known fake news data repository FakeNewsNet (Shu et al.,
2020b) which contains news articles along with their auxiliary information
such as users’ metadata and news comments. These news articles have been fact-
checked with two popular fact-checking platforms - Politifact and GossipCop.
Politifact fact-checks news related to the U.S. political system, while
GossipCop fact-checks news from the entertainment industry. In addition to the
existing Politifact news articles from FakeNewsNet, we enrich the dataset by
adding $5,000$ annotated Politifact news from the dataset introduced by
Rashkin et al. (Rashkin et al., 2017). Table 1 shows the statistics of the
final dataset. The Politifact news articles from (Rashkin et al., 2017)
includes truth rating from $0$ to $5$, in which we only consider news with
$\text{label}\in\\{0,4,5\\}$.

| Politifact | GossipCop
---|---|---
# True News | 2,645 | 3,586
# Fake News | 2,770 | 2,230
# News | 5,415 | 5,819
# News with Comments | 415 | 5,819
# Users | 60,053 | 43,918
# Unique Users | 100,520
Table 1. The statistics of Politifact and GossipCop datasets. The Politifact
dataset contains news articles from both FakeNewsNet and the dataset provided
by Rashkin et al. (Rashkin et al., 2017), while GossipCop contains news
articles from FakeNewsNet only.

###  5.2. Data Pre-processing

Each news articles from the final dataset contains news content, users’
comments, and their meta-data. We pre-process the textual data (i.e., news
content and users’ comments) by removing the punctuation, mentions, and out-
of-vocabulary words. Further, as BERT has a limitation of getting $512$ words
as input, we truncate the news content and every comment to include its first
$512$ words. Finally, we utilize the users’ meta-data to create user-news
interaction matrix by tracking every user’s interactions with news articles.

###  5.3. Implementation Details

The training process has been conducted in three stages: (1) pre-training the
representation networks, (2) training the fake news and domain classifiers,
and (3) training the RL agent. In what follows, we will expand the
implementation details of each stage.

Pre-training: In this stage, we fine-tune BERT and HAN networks to generate a
reasonable text representation from news content and users’ comments,
respectively. Motivated by the low memory consumption of the distilled version
of Bert (Sanh et al., 2019), we used the base model of Distilled BERT for
creating the textual representation of the news contents. The model was fine-
tuned for $3$ iterations using a classifier on top of it. Moreover, to create
the representations related to the user’s comments, we pre-trained HAN by
stacking it with a simple fake news classifier and fine-tuning it for $5$
iterations. After fine-tuning both models, the classifier module in both
distilled Bert and HAN were removed. These pre-trained networks were placed in
the final architecture of our model.

Training classifiers $F$ and $D$: With passing the news content through the
representation network in Figure 2, we trained a fake news classifier and a
domain classifier using the news content representations. During the training,
we did not update the weights of BERT and HAN networks. In this stage, a
dropout with $p=0.2$ was used for both fake news and domain classifiers. Both
classifiers use a similar feed-forward neural network with a single hidden
layer of $256$ neurons. The networks are trained using Adam optimizer with a
learning rate of $1e-5$ and a Cross Entropy loss function:

(4)
$\mathcal{L}_{CE}=-\frac{1}{M}\sum_{i=1}^{M}(y_{i}\log(p_{i})+(1-y_{i})\log(1-p_{i}))$

Model | GossipCop $\rightarrow$ Politifact | GossipCop $\rightarrow$ GossipCop | Politifact $\rightarrow$ Politifact | Politifact $\rightarrow$ GossipCop
---|---|---|---|---
F1 | AUC | F1 | AUC | F1 | AUC | F1 | AUC
CSI |  0.532 $\pm$ 0.11 |  0.563 $\pm$ 0.14 |  0.811 $\pm$ 0.05 |  0.832 $\pm$ 0.08 |  0.756$\pm$0.02 |  0.785 $\pm$ 0.10 |  0.589$\pm$0.10 |  0.610 $\pm$ 0.13
URG |  0.448$\pm$0.11 |  0.450 $\pm$ 0.11 |  0.770$\pm$0.04 |  0.792 $\pm$ 0.10 |  0.526$\pm$0.01 |  0.741 $\pm$ 0.08 |  0.460$\pm$0.09 |  0.532 $\pm$ 0.15
DEF |  0.621$\pm$0.10 |  0.687 $\pm$ 0.11 |  0.857$\pm$0.03 |  0.893 $\pm$ 0.07 |  0.768$\pm$0.01 |  0.793 $\pm$ 0.06 |  0.423$\pm$0.06 |  0.561 $\pm$ 0.11
BBL |  0.695$\pm$0.07 |  0.715 $\pm$ 0.09 |  0.918$\pm$0.02 |  0.947 $\pm$ 0.08 |  0.824$\pm$0.02 |  0.868 $\pm$ 0.04 |  0.558$\pm$0.09 |  0.597 $\pm$ 0.05
UDA |  0.673$\pm$0.08 |  0.695 $\pm$ 0.09 |  0.901$\pm$0.02 |  0.923 $\pm$ 0.06 |  0.727$\pm$0.02 |  0.762 $\pm$ 0.02 |  0.596$\pm$0.09 |  0.602 $\pm$ 0.07
EMB |  0.704$\pm$0.07 |  0.719 $\pm$ 0.08 |  0.914$\pm$0.04 |  0.952 $\pm$ 0.05 |  0.835$\pm$0.04 |  0.852 $\pm$ 0.03 |  0.586$\pm$0.06 |  0.601 $\pm$ 0.05
SRE |  0.656$\pm$0.09 |  0.714 $\pm$ 0.11 |  0.867$\pm$0.05 |  0.931 $\pm$ 0.07 |  0.720$\pm$0.01 |  0.751 $\pm$ 0.05 |  0.429$\pm$0.08 |  0.521 $\pm$ 0.09
REAL-FND |  0.726$\pm$0.11 |  0.728 $\pm$ 0.10 |  0.905$\pm$0.10 |  0.933 $\pm$ 0.08 |  0.844$\pm$0.05 |  0.810 $\pm$ 0.08 |  0.601$\pm$0.09 |  0.612 $\pm$ 0.11
Table 2. Single-domain: Performance of REAL-FND and baselines, trained only on
one domain and tested on both Politifact and GossipCop domains (Train
$\rightarrow$ Test). The values indicate F1 and AUC measures on the test set.

(a) GossipCop as source, Politifact as target domain

(b) Politifact as source, GossipCop as target domain

Figure 3. The impact of auxiliary information and reinforcement learning for
cross-domain fake news detection. Different versions of REAL-FND are evaluated
on the target domain.

Training the RL agent: Once finished with the first two stages, the fake news
classifier and the domain classifier are placed as the reward function of the
RL setting to train the agent. We train the agent using Algorithm 1 for
$2,000$ episodes performing only $T=20$ actions. As the long term reward
function is important, we used a discount rate of $\lambda=0.99$. In updating
the agent network, we applied Adam optimizer and set the parameters with
$\alpha=\beta=0.5$, $\sigma=0.01$, and learning rate $lr=1e-4$. The RL agent
uses a feed-forward neural network with $2$ hidden layers of $512$ and $256$
neurons.

###  5.4. Baselines

For evaluating the effectiveness of REAL-FND on fake news detection task, we
compare it to the baseline models described below. For a comprehensive
comparison we consider state-of-the-art baselines that (1) only use news
content (BERT-BiLSTM and BERT-UDA), (2) use both news content and users’
comments (TCNN-URG and dEFEND), (3) consider users’ interactions (CSI), and
(4) considers propagation network (RoBERT-EMB). In addition to these
baselines, we consider two variants of REAL-FND, Simple-REAL-FND and Adv-REAL-
FND that use bi-directional GRU instead of BERT/HAN and adversarial training,
respectively. These two baselines help us to study the effectiveness of using
a complex architecture such as BERT and RL for domain adaptive fake news
detection.

  * •

TCNN-URG (URG) (Qian et al., 2018): Based on TCNN (Kim, 2014), this model uses
convolutional neural networks to capture various granularity of news content
as well as including users’ comments.

  * •

CSI (Ruchansky et al., 2017): CSI is a hybrid model that utilizes news
content, users’ comments, and the news source. The news representation is
modeled using LSTM neural network using Doc2Vec (Le and Mikolov, 2014) that
outputs an embedding for both news content and users’ comments. For fair
comparison, we disregarded the news source feature.

  * •

dEFEND (DEF) (Shu et al., 2019a): This model uses a co-attention network to
model both news content and users’ comments. dEFEND captures explainable
content information for fake news detection.

  * •

BERT-UDA (UDA) (Gong et al., 2020): This model uses feature-based and
instance-based domain adaptation on BERT to create domain-independent news
content representation.

  * •

BERT-BiLSTM (BBL) (Vlad et al., 2019): This model uses a complex neural
network model including BERT (Vaswani et al., 2017), bi-directional LSTM
layer, and a capsule layer to classify news content. This model uses transfer
learning to work across different domains.

  * •

RoBERTa-EMB (EMB) (Silva et al., 2021): This model adopt instance-based domain
adaptation. It uses RoBERTa-base (Liu et al., 2019) to create news content
representation, while uses an unsupervised neural network to create the
propagation network’s representation (Silva et al., 2020). To evaluate this
method on single-domain, we disable the domain adaptation process.

  * •

Simple-REAL-FND (SRE): To study the effect of using complex networks such as
BERT and HAN for creating the representation of the news content and users
comments, we replace BERT and HAN with a bi-directional Gated Recurrent Unit
(BiGRU) stacked with a location-based attention layer (Chung et al., 2014).
Specifically, we use two separate BiGRU neural networks to create
$h^{\prime}_{content}$ and $h^{\prime}_{comments}$. To create
$h^{\prime}_{comments}$, we concatenate all of the comments and select the
first $256$ words.

###  5.5. Experimental Results

To answer questions Q1 and Q2, we trained REAL-FND and the baselines on a
source domain $S$ and tested them on both source domain $S$ and target domain
$T$. We used $k$-fold validation and calculated the average AUC and F1 scores.
For single-domain and cross-domain, we set the number for folds as $k=9$ and
$k=10$, respectively. In single-domain training, we use the source domain data
$\mathcal{D}_{s}$, while for training REAL-FND and the baselines in a cross-
domain setting, we use the source domain data $\mathcal{D}_{s}$ combined with
portion $\gamma$ of the target domain data $\mathcal{D}_{t}$. In the
subsequent subsection it will be shown in 4(c) that performance improvements
taper off after using $30\%(\gamma=0.3)$ of the target domain data. Finally,
to answer Q3, we perform an ablation study to measure the impact of using
auxiliary information and reinforcement learning.

Fake News Detection Results (Q1). Table 2 shows a comparison of the baselines
with REAL-FND. Training is applied on a single domain dataset
$\mathcal{D}_{s}$ and tested on both source and target domains. For this
experiment, we removed the domain classifier’s feedback from the RL agent by
setting the parameter $\beta=0$. From the results we conclude that (1) all
baseline models and REAL-FND perform reliably well when both training and
testing news come from a single domain, and (2) due to the considerable
decrease in performance when testing with news from another domain, it appears
that the Politifact and GossipCop news domains have different properties.
These results imply that the evaluated models are not agnostic to domain
differences. REAL-FND performed better than baselines for the majority of
scenarios. The only case where REAL-FND is under-performed is when the model
is trained and tested on the GossipCop dataset. The better performance of
REAL-FND on the Politifact domain suggests that BBL may have overfitting
issues and REAL-FND benefits the use of auxiliary information for creating a
more general fake news classifier that performs well on both domains. It is
worth mentioning that EMB and BBL models are similar to each other in terms of
model architecture except that EMB utilizes the user’s propagation network as
well. Comparing the results between these two models also reveals that using
auxiliary information can be helpful in detecting fake news.

(a) Effect of the $\alpha$ parameter

(b) Effect of the $\beta$ parameter

(c) Effect of the target domain portion $\gamma$

Figure 4. The impact of reward function parameters and the portion of the
target domain data. Blue line shows the result of REAL-FND on GossipCop, while
the red line shows the F1 for REAL-FND trained on the Politifact domain.

Cross-Domain Results (Q2). Table 3 shows the performance of models on the
target domain. In this experiment, we used $\gamma=0.3$ of nine-folds of the
target domain dataset $\mathcal{D}_{t}$ in addition to all the source domain
dataset $\mathcal{D}_{s}$. Performance measures are calculated based on the
tenth-fold of the target domain dataset. The results indicate that REAL-FND
detects fake news in the target domain most efficiently, in comparison to the
baselines. For example, compared with the best baselines, both F1 and AUC
scores have improved. This indicates that most baselines suffer from
overfitting on the source domain. Moreover, the results show that using a
small portion of the target domain dataset can lead to a large increase in the
cross-domain classification, indicating that the RL agent learns more domain-
independent features by the feedback received from the domain classifier. It
is also notable that Simple-REAL-FND (SRE) perform surprisingly well despite
using weaker network architecture than the most baselines.

Impact of RL and Auxiliary Information (Q3). To show the relative impact using
reinforcement learning and auxiliary information, we created the following
variants of REAL-FND:

  * -

REAL-FND $\backslash$A: To study the effects of using auxiliary information in
fake news detection, we created a variant of REAL-FND which does not use
users’ comments and user-news interactions. Thus, this model only uses BERT to
create news article representations.

Model | GossipCop $\rightarrow$ Politifact | Politifact $\rightarrow$ GossipCop
---|---|---
AUC | F1 | AUC | F1
CSI |  0.581 $\pm$ 0.03 |  0.547 $\pm$ 0.02 |  0.612 $\pm$ 0.03 |  0.598 $\pm$ 0.02
URG |  0.503 $\pm$ 0.04 |  0.486 $\pm$ 0.02 |  0.545 $\pm$ 0.04 |  0.471 $\pm$ 0.02
DEF |  0.712 $\pm$ 0.02 |  0.634 $\pm$ 0.01 |  0.583 $\pm$ 0.02 |  0.583 $\pm$ 0.01
BBL |  0.748 $\pm$ 0.01 |  0.711 $\pm$ 0.01 |  0.634 $\pm$ 0.01 |  0.634 $\pm$ 0.01
UDA |  0.812 $\pm$ 0.02 |  0.778 $\pm$ 0.01 |  0.702 $\pm$ 0.01 |  0.702 $\pm$ 0.01
EMB |  0.870 $\pm$ 0.01 |  0.876 $\pm$ 0.02 |  0.846 $\pm$ 0.03 |  0.795 $\pm$ 0.03
SRE |  0.885 $\pm$ 0.03 |  0.881 $\pm$ 0.04 |  0.838 $\pm$ 0.03 |  0.791 $\pm$ 0.05
REAL-FND |  0.901 $\pm$ 0.02 |  0.892 $\pm$ 0.03 |  0.862 $\pm$ 0.05 |  0.815 $\pm$ 0.04
Table 3.  Cross-domain: Cross-domain fake news detection results on the target
domain. source $\rightarrow$ target indicates the models have been trained on
the source domain and tested on the target domain. In this case, we use the
source domain data in addition to a small portion of the target domain data.

  * -

Adv-REAL-FND (ARE): To study the effect of the RL agent in domain adaptation,
we use adversarial training to create a domain adaptive news article
representation. In this model, we use the representation network, fake news
classifier $F$, ARE domain classifier $D$ in an adversarial setting using the
following loss function to train a fake news classifier and representation
network that is capable of creating domain independent features.

(5) $\min_{F}\max_{D}\mathcal{L}_{CE}(F)-\mathcal{L}_{CE}(D),$

where $L$ is the binary cross entropy loss similar to Equation 4. In this
model we do not need to pre-train the fake news classifier $F$ and the domain
classifier $D$. Instead, we train the representation network, fake news
classifier $F$, and domain classifier $D$ as a whole using Equation 5.

  * -

Adv-REAL-FND $\backslash$A: In this model, we removed both RL component and
the auxiliary information.

According to Figure 3, it is notable that removing the users’ comments, and
the user-news interactions severely impacts the performance of cross-domain
fake news detection. Although using adversarial training for cross-domain fake
news detection performs well, reinforcement learning allows us to adopt the
news article representation using any pre-trained domain and fake news
classifier in the reward function without considering its differentiability.

###  5.6. Parameter Analysis

Figure 4 shows the results of varying parameters in the reward function (i.e.,
Equation 1) and the portion of target domain data. We perform sensitivity
analysis on the reward function parameters $\alpha,\beta$ by fine-tuning the
parameters across the range $[0.0,1.0]$. We also examine the effect of the
$\gamma$ parameter by changing it across the range $[0.0,0.7]$. We evaluate
the effect of varying these parameters by evaluating the F1 score of cross-
domain fake news detection.

4(a) shows the F1 score for different values of $\alpha$ when $\beta=0.5$. The
$\alpha$ parameter controls the amount of the contribution of the fake news
classifier in the reward function. This figure indicates that the performance
does not increase when $\alpha\geq 0.6$ and even lead to a decrease in the
performance if we use a higher value than $\beta$.

4(b) illustrates the results of varying $\beta$ when $\alpha=0.5$. By using
$\beta=0.0$, REAL-FND does not consider the feedback of the domain classifier,
thus, the performance on the target domain will be dismal. By increasing
$\beta$, RL penalizes the agent more when the agent cannot decrease the
confidence of the domain classifier. Similar to the $\alpha$ parameter, the
performance does not increase by a large margin by using $\beta\geq 0.6$.

To analyze the impact of $\gamma$, we use various portions of the target
domain dataset to train REAL-FND. Intuitively, using all the data from target
domain will improve the performance; however, in real-world scenarios, we may
not have access to a comprehensive dataset from the target domain and using
such dataset can lead to an expensive computation time. In real-world
scenarios, domain-specific data is usually sparse and training on $100\%$ of
the target domain’s data is impractical and unrealistic. Based on our
analysis, using $30\%$ of target domain dataset seems to be sufficient for
training a reliable domain adaptive fake news detection model.

##  6\. Conclusion and Future Work

Collecting and integrating news articles from different domains and providing
human annotations by fact-checking the contents for the purpose of aggregating
a dataset are resource-intensive activities that hinder the effective training
of automated fake news detection models. Although many deep learning models
have been proposed for fake news detection and some have exhibited good
results on the domain they were trained on, we show in this work that they
have limited effectiveness in other domains. To overcome these challenges, we
propose the REinforced domain Adaptation Learning for Fake News Detection
(REAL-FND) task which could effectively classify fake news on two separate
domains using only a small portion of the target domain data. Further, REAL-
FND also leverages auxiliary information to enhance fake news detection
performance. Experiments on real-world datasets show that in comparison to the
current SOTA, REAL-FND adopts better to a new domain by using auxiliary
information and reinforcement learning and achieves high performance in a
single-domain setting.

Enhancements on REAL-FND could focus on cross-domain fake news detection under
limited supervision to address the effects of limited or imprecise data
annotations by applying weak supervision learning in a domain adaptation
setting. Additionally, REAL-FND could benefit from methods that automatically
defend against adversarial attacks. For example, malicious comments that
promote a fake news article could degrade the performance of REAL-FND because
of REAL-FND’s contingency on users’ comments as auxiliary information.
Finally, we also propose addressing the inconsistencies in multi-modal
information due to the breadth of the types and sources of information that
are required for a cross-domain fake news classifier such as REAL-FND.

##  7\. Acknowledgements

This material is based upon work supported by ONR (N00014-21-1-4002), and the
U.S. Department of Homeland Security under Grant Award Number
17STQAC00001-05-00333Disclaimer: “The views and conclusions contained in this
document are those of the authors and should not be interpreted as necessarily
representing the official policies, either expressed or implied, of the U.S.
Department of Homeland Security.” . Kai Shu is supported by the NSF award
#2109316.

## References

  * (1)
  * Ben-David et al. (2010) Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010.  A theory of learning from different domains.  _Machine learning_ 79, 1-2 (2010), 151–175.
  * by Statista Research Department and 7 (2021) Published by Statista Research Department and Sep 7. 2021.  Daily social media usage worldwide.  https://www.statista.com/statistics/433871/daily-social-media-usage-worldwide/
  * Cheng et al. (2021) Lu Cheng, Ahmadreza Mosallanezhad, Yasin Silva, Deborah Hall, and Huan Liu. 2021.  Mitigating bias in session-based cyberbullying detection: A non-compromising approach. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 2158–2168.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Coleman (2020) Alistair Coleman. 2020\.  ’hundreds dead’ because of covid-19 misinformation.  https://www.bbc.com/news/world-53755067
  * Fedus et al. (2018) William Fedus, Ian Goodfellow, and Andrew M Dai. 2018\.  MaskGAN: better text generation via filling in the_.  _arXiv preprint arXiv:1801.07736_ (2018).
  * Feng et al. (2012) Song Feng, Ritwik Banerjee, and Yejin Choi. 2012\.  Syntactic stylometry for deception detection. In _ACL_.
  * Gong et al. (2020) Chenggong Gong, Jianfei Yu, and Rui Xia. 2020.  Unified Feature and Instance Based Domain Adaptation for End-to-End Aspect-based Sentiment Analysis. In _EMNLP_.
  * Guo et al. (2019) Chuan Guo, Juan Cao, Xueyao Zhang, Kai Shu, and Huan Liu. 2019.  DEAN: Learning Dual Emotion for Fake News Detection on Social Media.  _arXiv preprint arXiv:1903.01728_ (2019).
  * Guo et al. (2018) Han Guo, Juan Cao, Yazi Zhang, Junbo Guo, and Jintao Li. 2018.  Rumor detection with hierarchical social attention network. In _CIKM_.
  * Hosseinimotlagh and Papalexakis (2018) Seyedmehdi Hosseinimotlagh and Evangelos E Papalexakis. 2018.  Unsupervised content-based identification of fake news articles with tensor decomposition ensembles. In _Proceedings of the Workshop on Misinformation and Misbehavior Mining on the Web (MIS2)_.
  * Islam et al. (2020) Md Saiful Islam, Tonmoy Sarkar, Sazzad Hossain Khan, Abu-Hena Mostofa Kamal, SM Murshid Hasan, Alamgir Kabir, Dalia Yeasmin, Mohammad Ariful Islam, Kamal Ibne Amin Chowdhury, Kazi Selim Anwar, et al. 2020\.  COVID-19–related infodemic and its impact on public health: A global social media analysis.  _The American journal of tropical medicine and hygiene_ 103, 4 (2020), 1621\.
  * Kaelbling et al. (1996) Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996\.  Reinforcement learning: A survey.  _Journal of artificial intelligence research_ 4 (1996), 237–285.
  * Karami et al. (2020) Mansooreh Karami, Ahmadreza Mosallanezhad, Michelle V Mancenido, and Huan Liu. 2020\.  ” Let’s Eat Grandma”: When Punctuation Matters in Sentence Representation for Sentiment Analysis.  _arXiv preprint arXiv:2101.03029_ (2020).
  * Karami et al. (2021) Mansooreh Karami, Tahora H Nazer, and Huan Liu. 2021.  Profiling Fake News Spreaders on Social Media through Psychological and Motivational Factors. In _Proceedings of the 32nd ACM Conference on Hypertext and Social Media_. 225–230.
  * Karimi and Tang (2019) Hamid Karimi and Jiliang Tang. 2019.  Learning hierarchical discourse-level structure for fake news detection.  _arXiv preprint arXiv:1903.07389_ (2019).
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification.  _arXiv preprint arXiv:1408.5882_ (2014).
  * Kouw et al. (2016) Wouter M Kouw, Laurens JP Van Der Maaten, Jesse H Krijthe, and Marco Loog. 2016.  Feature-level domain adaptation.  _The Journal of Machine Learning Research_ (2016).
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _ICML_.
  * Li et al. (2020) Xinlong Li, Xingyu Fu, Guangluan Xu, Yang Yang, Jiuniu Wang, Li Jin, Qing Liu, and Tianyuan Xiang. 2020\.  Enhancing BERT representation with context-aware embedding for aspect-based sentiment analysis.  _IEEE Access_ 8 (2020), 46868–46876.
  * Li et al. (2017) Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2017\.  Paraphrase generation with deep reinforcement learning.  _arXiv preprint arXiv:1711.00279_ (2017).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Mitchell and Jurkowitz (2020) Amy Mitchell and Mark Jurkowitz. 2020.  Americans Who Mainly Get Their News on Social Media Are Less Engaged, Less Knowledgeable.  https://www.journalism.org/2020/07/30/americans-who-mainly-get-their-news-on-social-media-are-less-engaged-less-knowledgeable/
  * Mosallanezhad et al. (2019) Ahmadreza Mosallanezhad, Ghazaleh Beigi, and Huan Liu. 2019\.  Deep reinforcement learning-based text anonymization against private-attribute inference. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 2360–2369.
  * Pérez-Rosas et al. (2018) Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018.  Automatic Detection of Fake News. In _Proceedings of the 27th International Conference on Computational Linguistics_. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 3391–3401.  https://aclanthology.org/C18-1287
  * Potthast et al. (2017) Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2017\.  A stylometric inquiry into hyperpartisan and fake news.  _arXiv preprint arXiv:1702.05638_ (2017).
  * Qian et al. (2018) Feng Qian, Chengyue Gong, Karishma Sharma, and Yan Liu. 2018\.  Neural User Response Generator: Fake News Detection with Collective User Intelligence.. In _IJCAI_.
  * Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017.  Truth of varying shades: Analyzing language in fake news and political fact-checking. In _Proceedings of the 2017 conference on empirical methods in natural language processing_. 2931–2937.
  * Ruchansky et al. (2017) Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017.  Csi: A hybrid deep model for fake news detection. In _CIKM_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Schuster et al. (2020) Tal Schuster, Roei Schuster, Darsh J. Shah, and Regina Barzilay. 2020.  The Limitations of Stylometry for Detecting Machine-Generated Fake News.  _Computational Linguistics_ 46, 2 (06 2020), 499–510.  https://doi.org/10.1162/coli_a_00380 arXiv:https://direct.mit.edu/coli/article-pdf/46/2/499/1847559/coli_a_00380.pdf
  * Shabani and Sokhn (2018) Shaban Shabani and Maria Sokhn. 2018.  Hybrid machine-crowd approach for fake news detection. In _2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC)_. IEEE, 299–306.
  * Shu et al. (2020a) Kai Shu, Amrita Bhattacharjee, Faisal Alatawi, Tahora H Nazer, Kaize Ding, Mansooreh Karami, and Huan Liu. 2020a.  Combating disinformation in a social media age.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ 10, 6 (2020), e1385.
  * Shu et al. (2019a) Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019a.  dEFEND: Explainable Fake News Detection. In _KDD_.
  * Shu et al. (2020b) Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020b.  Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media.  _Big data_ 8, 3 (2020), 171–188.
  * Shu et al. (2017) Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017.  Fake news detection on social media: A data mining perspective.  _ACM SIGKDD Explorations Newsletter_ 19, 1 (2017), 22–36.
  * Shu et al. (2019b) Kai Shu, Suhang Wang, and Huan Liu. 2019b.  Beyond news contents: The role of social context for fake news detection. In _WSDM_. ACM.
  * Silva et al. (2020) Amila Silva, Yi Han, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2020.  Embedding Partial Propagation Network for Fake News Early Detection.. In _CIKM (Workshops)_.
  * Silva et al. (2021) Amila Silva, Ling Luo, Shanika Karunasekera, and Christopher Leckie. 2021.  Embracing Domain Differences in Fake News: Cross-domain Fake News Detection using Multi-modal Data.  _arXiv preprint arXiv:2102.06314_ (2021).
  * Tacchini et al. (2017) Eugenio Tacchini, Gabriele Ballarin, Marco L Della Vedova, Stefano Moret, and Luca de Alfaro. 2017\.  Some like it hoax: Automated fake news detection in social networks.  _arXiv preprint arXiv:1704.07506_ (2017).
  * Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver. 2016\.  Deep reinforcement learning with double q-learning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NeurIPS_.
  * Vlad et al. (2019) George-Alexandru Vlad, Mircea-Adrian Tanase, Cristian Onose, and Dumitru-Clementin Cercel. 2019\.  Sentence-Level Propaganda Detection in News Articles with Transfer Learning and BERT-BiLSTM-Capsule Model. In _Proceedings of the Second Workshop on NLP for Internet Freedom_. 148–154.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019\.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  _arXiv preprint arXiv:1904.02232_ (2019).
  * Zhang et al. (2020) Junzi Zhang, Jongho Kim, Brendan O’Donoghue, and Stephen Boyd. 2020.  Sample efficient reinforcement learning with REINFORCE.  _arXiv preprint arXiv:2010.11364_ (2020).
  * Zhou et al. (2020) Xinyi Zhou, Atishay Jain, Vir V Phoha, and Reza Zafarani. 2020\.  Fake news early detection: A theory-driven model.  _Digital Threats: Research and Practice_ 1, 2 (2020), 1–25.
  * Zhuang et al. (2015) Fuzhen Zhuang, Xiaohu Cheng, Ping Luo, Sinno Jialin Pan, and Qing He. 2015.  Supervised representation learning: Transfer learning with deep autoencoders. In _AAAI_.