#  $Tweets2Stance$: Users stance detection exploiting Zero-Shot Learning
Algorithms on Tweets

Margherita Gambini<EMAIL_ADDRESS>Tiziano Fagni<EMAIL_ADDRESS>Caterina Senette<EMAIL_ADDRESS>Maurizio Tesconi<EMAIL_ADDRESS>Institute of Informatics and Telematics (IIT) - CNR, Via Giuseppe Moruzzi, 1
56124 Pisa – Italy  Engineering School - University of Pisa, Largo Lucio
Lazzarino, 2, 56122 Pisa - Italy

###### Abstract

In the last years there has been a growing attention towards predicting the
political orientation of active social media users, being this of great help
to study political forecasts, opinion dynamics modeling and users
polarization. Existing approaches, mainly targeting Twitter users, rely on
content-based analysis or are based on a mixture of content, network and
communication analysis. The recent research perspective exploits the fact that
a user’s political affinity mainly depends on his/her positions on major
political and social issues, thus shifting the focus on detecting the stance
of users through user-generated content shared on social networks. The work
herein described focuses on a completely unsupervised stance detection
framework that predicts the user’s stance (along five levels of agreement)
about specific social-political statements by exploiting content-based
analysis of its Twitter timeline. The ground-truth user’s stance may come from
Voting Advice Applications (VAAs), online tools that help citizens to identify
their political leanings by comparing their political preferences with party
political stances. Starting from the knowledge of the agreement level of six
parties on 20 different statements (VAA’s statements), the objective of the
study is to predict the stance of a Party $p$ in regard to each statement $s$
exploiting what the Twitter Party account wrote on Twitter. To this end we
propose $Tweets2Stance$ ($T2S$), a novel and totally unsupervised stance
detector framework which relies on the zero-shot learning technique to quickly
and accurately operate on non-labeled data. Interestingly, T2S can be applied
to any social media user for any context of interest, not limited to the
political one. Results obtained from multiple experiments (analysis,
measurement, and evaluation) show that, although the general maximum F1 value
is $0.4$, $T2S$ can correctly predict the stance with a general minimum MAE of
$1.13$, which is a great achievement considering the task complexity.

###### keywords:

stance-detection , social media , zero-shot learning , language models

††journal: Journal of Information Sciences

##  1 Introduction

During the last years, there has been a growing attention towards social media
as for what is explicitly shared among users (content, thoughts, and
behavior), as well as about what is hidden and latent. Among this latent
information, the user’s stance, i.e. the expression of a user’s point of view
and perception toward a given statement [1], is particularly interesting; in
fact, stance detection on social media is an emerging opinion mining paradigm
that well applies in different social and political contexts, and for which
many researchers are working to propose solutions ranging from natural
language processing, web science, and social computing [2, 3, 4, 5, 6, 7, 8,
9].

Many works [2] dealt with the stance detection at the user-level; however, to
the best of our knowledge, a completely unsupervised technique exploiting
user’s textual content only has never been explored. Hence, the work herein
described focuses on the design and implementation of a stance detection
framework named $Tweets2Stance$ ($T2S$), able to detect the stance of a
Twitter account using its timeline. The idea for this framework stems out from
observing how Voting Advice Applications (VAAs) work. Voting Advice
Applications, originally developed in the 1980s as paper-and-pencil civic
education questionnaires [10], are online tools that aid citizens, mainly
before elections, to identify their political leaning by comparing their
policy preferences with the political stances of parties or candidates running
for office. VAAs are widespread in many countries and have a crucial role in
online election campaigns worldwide. Basically, the user marks its positions
on a range of policy statements. The application compares the individual’s
answers to the positions of each Party or candidate and generates a rank-
ordered list or a graph indicating which Party or candidate is located closest
to the user’s policy preferences. One of the crucial elements of the VAAs is
the questionnaire: the selection of the questions, their balance among the
political poles, and their phrasing have an impact both on the way in which
users respond, as well as on the overall users‘ engagement on the poll itself.
For these reasons, the VAA’s issued statements should cover the spectrum of
the most important topics of an election campaign and adequately show crucial
differences among all the competitors on political scenario for which the VAA
is designed[11]. This careful definition of the questionnaire, i.e. taking
into high consideration the main topics under discussion at a certain time,
suggested us the possibility of using the official position of Italian parties
about specific political statements (during a certain political election
period) as the ground-truth to determine that stance from the timeline of the
Twitter Party accounts in a completely unsupervised way 111the Italian
Parties’ official positions about 20 political statements were kindly provided
by the Observatory on Political Parties and Representation [12] based on the
VAA NavigatoreElettorale for the European Elections 2019 [13]; notice that
only tweets written during the pre-election period are considered.

##### Objectives

Starting from the knowledge of the agreement level of six parties on 20
different statements (VAA’s statements), the objective of the study is to
predict the stance of a Party $p$ in regard to each statement $s$ exploiting
what the Twitter Party account wrote on Twitter. According to a recent
literature review [2], our work can be classified as a stance detection task
at the user level, since it relies on the textual content of Twitter posts
(timelines of parties’ Twitter accounts) with a further specialization as
target-specific, the most common form of stance-detection on social media;
here, the target identifies a topic (whose stance is predicted), but
differently from previous works in the literature our classification model is
built for different topics. Focusing on content-based approaches, some
limitations that emerge in existing solutions are: (i) intrinsic difficulty of
processing natural language; (ii) need for large corpora of manually-annotated
tweets and language-specific resources; (iii) supervised approaches, which
lack generalizability and suffer from the limited availability of
comprehensive and reliable ground-truth datasets [14].

To overcome these drawbacks, we focus on the newest pre-trained language
models based on the Transformer architecture [15] which currently allow to
achieve the best effectiveness on practically all important supervised
learning NLU tasks [16]. Specifically, we rely on an unsupervised technique
called zero-shot learning (ZSL) [17] to exploit pre-trained models as generic
classifiers to infer the stance of a party towards a given topic of interest
(see subsection 5.1 for more details). Moreover, differently from state-of-
the-art methods, our goal is to come up with a fine-grained stance-detector
solution working along five classes that could be generalized to various
spheres, not just the political one.

##### Contribution

The main contributions of the work can be summarized as follows:

  * •

To the best of our knowledge, our $Tweets2Stance$ ($T2S$) framework represents
an innovative method to stance-detection task; it uses a bottom-up approach
based on stance-detection centered on textual content posted on the Party’s
Twitter accounts and a set of VAA statements on which we know the official
position taken by the Party.

  * •

As detailed in the Related Work section 2, our contribution is innovative
since we offer a relatively simple and totally unsupervised context-based
method where the contents are not individual texts but the users’ timelines.
Our framework can be applied to the timelines of other social media as well.

  * •

The proposed $T2S$ technique is organized as a generic framework which can be
easily customized to be adapted to specific social media user requirements. In
this work, the tested social media users are six Italian parties, since we had
their official stance about 20 political statements [12, 13].

  * •

The described approach applies well to the political scenario, since it is the
first logical step of an inference pipeline useful to predict the users’
political orientation; nevertheless, this technique can also be generalized to
other contexts, such as extremism/radicalization to infer whether a user has
become radicalized on certain themes (immigration, vaccines, etc.).

  * •

The proposed approach is easily adaptable to different languages, even in the
absence of pretrained ZSL models specific to the language of interest. In this
study we focus on the Italian language and compare various ZSL models to deal
with it.

##### Road-map

The remainder of the paper is organized as follows: Section 2 discusses the
related work, while Section 3 clearly defines the prediction task on which our
framework will work. Section 4 describes the preliminaries of the study
detailing the political scenario under analysis and the data collection and
cleaning procedures. Then, Section 5 details our framework for the prediction
task. Afterwards, Sections 6 and 7 describe the experimental setup and the
results achieved. Finally, Section 8 ends the paper by drawing conclusions and
future works.

##  2 Related Work

Stance-detection is an emerging opinion mining paradigm that well applies in
several social and political scenarios. The state of the art resumed in a
highly valuable recent survey [2] highlights the importance of categorization,
since stance-detection can be classified according to the target (single,
multi-related, or claim-based) or to the task type (detection or prediction).
Moreover, in order to better narrow the related work, we need to distinguish
between works at the user level and works at the statement level. At the
statement level [18, 19], whose objective is to predict the stance described
in a piece of text, previous research works are mainly based on Natural
Language Processing (NLP) methods and classification tasks with three classes
(support/against/none). Instead, at the user level, the objective is to
predict the stance of a user toward a given topic and generally, prediction
solutions incorporate different users’ attributes along with the text of their
posts. The work of Aldayel and Magdy [20] examined how various users’ online
features such as their posts, the network they interact with or follow, the
websites they visit, and the content they like can reveal their stance towards
different topics. In this case, rather than focusing on the detection
accuracy, the authors’ main focus was to understand how stance could be
revealed testing multiple sets of features on a stance benchmark dataset of
different topics. Lynn et al. [21] investigated the predictive power of user-
level features alone compared to document-level features for document-level
tasks (it means predicting the stance of a tweet without the tweet). To this
end, they present some NLP tasks that might be better primarily cast as user-
level, supporting the importance on integrating user features into predictive
systems.

###  2.1 Stance detection target-specific

According to the mentioned literature Review [2], our work could be classified
as a stance detection task at the user level with a further specialization as
target-specific, the most common form of stance-detection on social media.
Here the target identifies a topic (whose stance is predicted) and generally a
separated classification model is built and trained for each topic. Approaches
described in literature [22, 3, 4] are based on the text incorporated in a
post together with multiples post/users attributes; in those cases, task
classification mainly used two classes, support and against. Some examples are
detailed in the following. In the work of Gottipati et al. [22], the authors
proposed a collaborative filtering approach to solve the data sparsity problem
of users stances on ideological topics, and apply clustering method to group
the users with the same party. The experiments showed that using ideological
stances with Probabilistic Matrix Factorization (PMF) technique achieved a
high accuracy of 88.9% at 22.9% data sparsity rate and 80.5% at 70% data
sparsity rate on users’ party prediction task. The proposed approach was
applied to bipolar stance-detection (two positions per target). In the work of
Dias and Becker [3], a weakly supervised solution for detecting stance in
tweets was described; it used n-grams representing opinion targets, common
terms denoting stance (e.g. hashtags) and sentiment strategies to build a
large corpus representing the input of a supervised prediction algorithm. In
the work of Igarashi et al. [4], the authors compared feature-based and Neural
Network based approaches on the supervised stance classification task for
tweets. They achieved the best results with Convolutional Neural Network (CNN)
based approach in the cross validation on the training data, and the worst
result on the test data due to the CNN overfitting; in contrast, the feature-
based approach was more robust, leveraging the external knowledge. Other
target-specific strategies in literature were conducted at the statement
level, such as those proposed by Augenstein et al. [5],Mohammad et al. [6],
Sardar Hamidian et al. [7], Yingjie Li et al. [23], and Umme Aymun Siddiqua
[24]. The strategy presented in [5] was to use conditional LSTM encoding (Long
short-terms memory networks), which built a target-dependent representation of
the tweet and it outperformed the technique of encoding the tweet and the
target independently. Performance was further improved by augmenting
conditional model with bidirectional encoding. The approach was conducted at
the statement level through unsupervised methods, and classification was made
along three positions (favour, against, neither). In [6] the authors reported
results obtained involving different teams in a stance-detection shared task.
The task was to infer a three-level stance of tweets using automatic natural
language systems: in favor of the given target, against the given target, or
neither inference is likely. To this end, they subdivided the task into a
supervised (task A) and an unsupervised (task B) sub-task, receiving 19 and 9
submissions from teams respectively. The highest classification F-score
obtained was 67.82 for Task A and 56.28 for Task B. In the work of Sardar
Hamidian et al. [7], the authors performed a rumor stance classification
experimenting different levels of pre-processing through supervised
classificators; they exploited a new set of meta linguistic and pragmatic
features, and performed the experiments with and without pre-processing on the
textual content. Their approach did not seem to benefit from pre-processing.
In the work of Yingjie Li et al. [25], the authors designed a framework which
incorporates a target-specific attention mechanism and sentiment
classification, outperforming the state of the art of deep learning methods.
The goal of Siddiqua et al.’s work [24] was to determine the stance of a tweet
according to three positions. To this end, a particular neural network model
was employed: it adopted the strengths of two LSTM variants coupled with an
attention mechanism that amplified the contribution of important elements in
the final representation. As a result, the method went over the current state-
of-the-art deep learning based methods both for single and multi-target
benchmark datasets.

As mentioned above, target-specific approaches could consider single or
multiple targets. Usually, the concept of multi-target classification has been
used to analyse the relation between two political candidates by using domain
knowledge about these targets to improve the classification performance. In
that case, the same model can be applied to different targets on the
hypothesis that the same piece of text that contains the stance in favour to a
target, it also implicitly contains the stance against the other [26, 8, 27].
Obviously, those approaches need multi-target stance-detection datasets. In
the political context examined in our study, each statement is representative
of a specific target, thus we must face a multi-target classification task in
the broadest sense, that is not limited to a target and its opposite.
Previously cited methods well apply on one target at a time, since they
operate on datasets containing texts that have already been classified as
relevant to the topic (whose stance is predicted); consequently, these
strategies need pre-filtered data containing additional elements such as
particular hashtags or keywords. On the contrary, our method does not need to
work with selected texts, because it includes a data filtering step which
guarantees in input only relevant data. Moreover, our approach works well in
the multi-target scenario without differentiating the classification model for
each target thanks to the advantage of operating with a generic zero-shot
learning classifier.

###  2.2 Machine Learning approaches

Considering features to model stance-detection tasks and machine learning (ML)
algorithms applied, we focus on linguistic cues, as they are the element we
are interested in. Here, the literature reports a further subdivision on (i)
linguistic features, i.e. the most used approach that reveals the stance based
on text-linguistic features [28, 6], and (ii) users’ vocabulary (stance-
detection is based on the user’s choice of vocabulary) [9, 16]. Since textual
cues could refer both to textual features, sentiment, and semantics, we limit
our attention to textual features. Given the above, and focusing on machine
learning approaches used to model and train classifiers employed on all the so
far mentioned works, the most are based on supervised ML techniques [22, 4,
27, 19, 29]. Some works attempted to enrich datasets entities applying
unconstrained supervised methods such as transfer learning, weak-supervision,
and distant supervision methods for stance detection [5, 3]. Others innovative
approaches are those that propose unsupervised learning strategies [9, 30, 31]
exploiting clustering techniques and embeddings representations of users’
tweet[32].

The major limitations that emerge from all the aforementioned studies are
represented by:

  * •

For the approaches that exploit textual and non-textual information in stance-
detection, data collection and analysis are highly time-consuming, especially
in the case of network-based solutions;

  * •

The required data are often not available or very difficult to retrieve,
mainly due to increasing data protection policies adopted by Social Media over
the years;

  * •

The majority of the aforementioned detection models have only two (maximum
three) stance classes;

  * •

Almost all the mentioned techniques require supervised or semi-supervised
prediction models whose limits, well known in literature [14], include, among
others, the need for large amounts of data and the poor generalizability,
being strongly dependent on the dataset used as training-set. Among
unsupervised attempts it is worth mentioning the work of Darwish et al. [9]
close related to our study, that used dimensional reduction to project users
onto a low-dimensional space, followed by a clustering step, which allowed to
find core users that are representative of the different stances. Their
proposed solution relied on unsupervised methods not requiring prior labeling
of users, neither a domain- or topic-level knowledge, but needing a manually
labeled cluster (low effort). Comparing it with our work, the main emerging
differences are: (i) Darwish’s method only analyses users who produce the most
of the posts on a given topic, instead ours approach considers any type of
user, even those who have very few (1 or 2) significant tweets (non-vocal
users); (ii) The Darwish method works on 2 classes, our method detects five
classes; (iii) The Darwish method provides a top-down solution to stance-
detection starting from a group analysis that includes the users who discuss
the topic the most and trying to separate them into two classes (for,
against). The Darwish method is not suitable for making individual analysis if
there is no context information deriving from other users, i.e. it cannot
analyse a single user and understand if he is in favour or against a certain
topic. Our method, on the other hand, is a bottom-up approach that analyses
the information of the target user without the need to have access to network
or context information from other users (e.g. retweets); (iv) Moreover, in the
Darwish method the stance detection must be done on pre-filtered data relevant
to the topic of interest. Our method, having set the sentence and the
associated topic, can work on any type of content, appropriately self-
filtering relevant data for the topic.

For all these reasons, the recent challenge to user-level and target-specific
stance detection is to move towards unsupervised systems exploiting textual
content only. To this aim, a ZSL technique exploiting advanced pre-trained
Natural Language Inference (NLI) models [28, 25] can be a viable solution (see
Section 5.1 for further details), as our stance-detection framework proved.
All things considered, our work may open the way to a novel approach to
stance-detection both at the user and target-specific level (as detailed in
the subsequent sections); to the best of our knowledge, this particular
stance-detection task is not yet described in the closely related literature,
hence representing the first of further contributions from/to the community of
researchers working on this topic.

##  3 Task Definition

The task (Fig. 1) is to predict the stance $A^{u}_{s}$ of a Social Media User
$u$ with respect to a social-political statement (or sentence) $s$ making use
of the User’s textual content timeline on the considered social media (e.g.,
the Twitter timeline).

Figure 1: The task is predicting the agreement level $A^{u}_{s}$ of Twitter
User $u$ in regard to sentence $s$ exploiting the User’s textual content
timeline $TL_{u}$.

The stance $A^{u}_{s}$ is a five-valued categorical label, as shown in Table
1.

Table 1: The agreement/disagreement levels Categorical | Integer
---|---
Mapping
completely agree | 5
agree | 4
neither disagree, nor agree | 3
disagree | 2
completely disagree | 1

The desired ground-truth is the label $G^{u}_{s}$, that is the known
agreement/ disagreement level of User $u$ in regard to sentence $s$. Remind
that the ground-truth is only used to evaluate our proposed $Tweets2Stance$
framework and find its optimal parameters; no training step ever occurs. In
this work, we assume that users are the Twitter accounts of six Italian
Parties, as the following section will detail.

##  4 Data collection and Pre-processing

The political scenario under analysis refers to the European and Municipal
elections in Italy on 26th May 2019, when Italian citizen were called for the
election of the Italian representatives to the European Parliament. The number
of Members of the European Parliament (751 deputies in total) for each country
is approximately proportional to the population. In 2019, Italy had to elect
76 deputies. Contextually, Italian voters had also to participate to municipal
election of mayors, municipal and district councillors (in about 3800 Italian
municipalities), with planned run-off on 9th June 2019. In that context, we
focused our attention on the six major parties in Italy: three center-right
parties including Forza Italia (FI), Fratelli d’Italia (FDI), and Lega, two
left-wing parties including Partito Democratico (PD) and +Europa
(+Eu)222+Europa was recently born in 2018 and it is characterized for a pro-
European and liberal orientation., and the Movimento 5 Stelle (M5S)
representing a sort of third pole at that time. The Italian parliament
included other minor parties, especially on the left- wing, representing less
than 5% of the Italian population each. We did not consider these parties in
the current study. As previously said, we started from the assumption that
knowing the parties’ answers on the VAA’s statements, it is possible to
predict the stance of a Party $p$ in regard to each statement $s$ exploiting
what the Party wrote on Twitter. The definition of the 20 statements (Table
LABEL:tab:topic_definition_eng and Table LABEL:tab:topic_definition_ita) that
express the political positions of the six referenced parties towards selected
themes under discussion in Italy and in Europe in 2019, was entrusted to a
group of political experts [12, 13] who provided us with the ground-truth
$G^{p}_{s}$ for each Party $p$ and statement $s$ on which the current work is
based.

At first, we collected timelines of the official Twitter account of each party
using the official Twitter API333https://developer.twitter.com/en/docs.
Considering the speed with which the political discussion nowadays takes place
especially on social media, the observation period was adequately chosen in
order to maximize the number of tweets avoiding noise and off-topics content.
Furthermore, to intercept any valuable information or discussion trends
overtime we have extended the analysis considering four temporal ranges and
built the associated datasets444 The four raw datasets can be found at
https://github.com/marghe943/Tweets2Stance_dataset as described in Table 2.

Table 2: The four studied datasets with the total number of tweets before the pre-processing step. $D_{j}$ contains $j$ months of tweets. |  $D_{3}$ |  $D_{4}$ |  $D_{5}$ |  $D_{7}$
---|---|---|---|---
Period |  [2019-03-01, 2019-05-25] |  [2019-02-01, 2019-05-25] |  [2019-01-01, 2019-05-25] |  [2018-11-01, 2019-05-25]
#tweets |  20’266 |  25’979 |  34’736 |  44’370

As a preliminary step, since the text collected from tweets contains a lot of
noise and irrelevant information, we pre-processed the tweets in order to
remove anything which don’t have predicting significance, that is:

  * •

URLs (uniform resource locator), $++$, $\&gt$, $\&lt$, \$n$ were replaced with
an empty string,

  * •

we removed the ”$RT@user:$” prefix of retweets,

  * •

we removed the mentions at the beginning of a reply tweet,

  * •

the hashtags were replaced with empty string, being difficult to analyse by
the Zero-Shot Classifier component (see Section 5.1),

  * •

the emojis were replaced with an empty string, being difficult to analyse by
the Zero-Shot Classifier component (see Section 5.1),

  * •

heading whitespaces were replaced with an empty string,

  * •

more than two subsequent whitespaces were replaced with a single whitespace,

  * •

finally, we removed tweets with $\\{1,2,3\\}$ words and empty tweets.

Lastly, since we wanted to test our prediction approach on English tweets as
well (as explained in Section 5.1), we further translated the Italian tweets
using the google_trans_new555https://pypi.org/project/google-trans-new/ Python
package.

##  5 Methodology

This section presents the proposed $Tweets2Stance$ (T2S) framework (Fig. 2) to
predict the stance $A^{u}_{s}$ of a Twitter User $u$ in regard to a sentence
$s$, exploiting its Twitter timeline $TL_{u}=[tw_{1},...,tw_{n}]$. A User
might either not talk about a specific political argument (here expressed with
sentence $s$), or debate on an issue not risen by our pre-defined set of
statements. For these reasons, our framework executes a preliminary $Topic$
$Filtering$ step, exploiting a zero-shot classifier $C$ to get only those
tweets talking about the topic $tp$ of the sentence $s$. Finally, the
$Agreement$ $Predictor$ uses the in-topic tweets $I^{u}_{tp_{s}}$ and the same
zero-shot classifier $C$ to predict the agreement/disagreement level
$A^{u}_{s}$.

Figure 2: Our Tweets2Stance framework to compute the agreement level
$A^{u}_{s}$ of User $u$ in regard to sentence $s$. The inputs are the Twitter
timeline $TL_{u}$ extracted from a certain dataset $D_{i}$, the sentence $s$,
the topic $tp$ associated to $s$, a language model $LM$, a threshold $th$ and
an algorithm $Alg$. The highlighted components are the parameters that we’ll
vary during our experiments, as explained in Section 6.

As Fig. 2 illustrates, the parameters of the $T2S$ framework to tune are four:

  1. 1.

the language model $LM$ to use for the zero-shot classification in both the
$Topic$ $Filtering$ and the $Agreement$ $Predictor$ modules,

  2. 2.

the dataset $D_{i}$ from which extracting the timeline $TL_{u}$,

  3. 3.

the algorithm $Alg$ to use in the $Agreement$ $Predictor$ module,

  4. 4.

the threshold $th$ to get the in-topic tweets $I^{u}_{tp_{s}}$ in the $Topic$
$Filtering$ module.

Through the experiments described in Section 6, we performed a grid search
over the four parameters to get the optimal setup.

The following subsections will discuss each of the three main modules of our
framework: the zero-shot classifier $C$ to comprehend what a tweet is about,
and the $Topic$ $Filtering$ and $Agreement$ $Predictor$ modules. Hereinafter,
we’ll detail the proposed framework into a specific political scenario where
the Twitter accounts of interest are those of the six Italian parties reported
in Section 4, hence the User $u$ becomes the Party $p$.

###  5.1 Zero-shot Classification

To comprehend the textual semantic of a tweet, we relied on the advanced
Natural Language Processing (NLP) methods based on recent and powerful neural
language models [33], since they have shown extremely competitive performance
on many popular Natural Language Understanding (NLU) tasks [34, 35, 36].
Particularly, we exploited the zero-shot learning technique first proposed by
[17] to use language models pre-trained on the Natural Language Inference
(NLI) task [23] as generic classifiers able to quickly and accurately operate
on non-labelled data. This specific classification method is called zero-shot-
classification (Fig. 3).

Figure 3: A zero-shot classifier receives in input a tweet (premise) and one
or more sentence/topic (hypothesis). The output is the classification score
for each couple (tweet, sentence or topic).

The HuggingFace Transformer library [37] provides ready-to-use state-of-the-
art neural language models (the ones based on the Transformer architecture
[15]) along with an easy framework to work on most of the NLU tasks. For this
reason, the HuggingFace library was chosen to perform zero-shot
classification. Many neural language models were pre-trained solely on
English-language texts (e.g., GPT2 [34] and BERT [35]). Unfortunately, our
Twitter datasets contain tweets written in Italian. There are currently three
ways of addressing this English-centricity:

  1. 1.

translating the documents in English and using the English models, or

  2. 2.

relying on models pre-trained on a single non-English Language (e.g., the
Italian AlBERTo [38], the Portuguese’s PTT5 [39], the Dutch BERTje [40]), or

  3. 3.

using multi-lingual models that have been pre-trained on a mixture of many
languages (XLM [41], XLM-Roberta [42], mBART [43], mT5 [44]).

Since the zero-shot classification stems from the NLI task, only those
language models pre-trained or already fine-tuned on one or more NLI tasks can
be used, which is not the case for the Italian Transformer-based language
models (AlBERto [38], BERTino [45]). Therefore, we chose to follow the 1) and
3) solutions, in particular adopting these 3 configurations:

BART

The BART-large model [46] fine-tuned on the MultiNLI dataset [47]
($BART\\_Large_{MNLI}$, [48]) was used with tweets translated in English. We
exploited the library google_trans_new666https://pypi.org/project/google-
trans-new/, a wrapper for Google Translate service, to perform automatic
translation of tweets from Italian to English language.

XRoberta_1

The XLM-Roberta-Large model [42] fine-tuned on the XNLI dataset [49] was
employed as multi-lingual model over Italian tweets ($XLM\\_Roberta$
$\\_Large_{XNLI}$, [50]). Notice that the XNLI dataset doesn’t include
Italian, but the authors assure that since the base model (XLM-Roberta-Large)
was pre-trained on 100 different languages (Italian included), the fine-tuned
model has shown good performances in NLI task even with other languages
available only in base model.

XRoberta_2

We evaluated the $XLM\\_Roberta\allowbreak\\_Large_{IT\\_MNLI}$ [51] on
Italian tweets: this model takes XLM-Roberta-Large and fine-tunes it on an
Italian translation of a subset of the MNLI dataset (85% of the train set).

###  5.2 Topic Filtering

The $Topic$ $Filtering$ module outputs the in-topic tweets $I^{p}_{tp_{s}}$ to
be fed to the $Agreement$ $Predictor$ module. To this aim, we had to define a
topic for each of the considered statements: Table
LABEL:tab:topic_definition_eng and Table LABEL:tab:topic_definition_ita show
the English and Italian topic definitions for the $20$ statements that were
used for all experiments; as Section 6 will describe, we experimented both
with Italian tweets and the same ones translated in English by varying the
language model $LM$ used for C.

Table 3: Defined topic for each of the $20$ sentence (English version). nr. |  Sentence |  Topic
---|---|---
1 |  overall, being EU members is a disadvantage |  European Union disadvantages
2 |  Italy should exit the euro |  exit the euro
3 |  a common European army should exist |  common European army
4 |  multinational corporations like Google and YouTube should pay copyright and taxes according to the rules of each European country |  taxes for multinationals compliant with rules of each European country
5 |  european economic integration has driven too far: member states should regain greater autonomy |  members economic autonomy in the European Union
6 |  the European Union should reform its immigration policy: Italy should receive more support from other Member States |  immigration handling in EU
7 |  Italy should strengthen economic cooperation with China |  economic cooperation between China and Italy
8 |  the recreational use of cannabis should be legal |  recreational use of cannabis
9 |  Islam is a threat to Italy’s values |  Islam threatens Italian values
10 |  women must be guaranteed autonomy of choice over abortion |  autonomy of choice over abortion
11 |  any form of self-defense within the private property should be legitimate |  self-defense in your own home
12 |  the activities of the judiciary must be independent of political pressures |  independence of the judiciary from politics
13 |  children who were born in Italy to foreign parents should receive Italian citizenship automatically |  Italian citizenship for children born in Italy to foreign parents
14 |  wealth should be redistributed from wealthier citizens to poorer citizens |  redistribution of wealth to the poorest
15 |  companies should be able to dismiss employees more easily |  ease of dismissal of employees
16 |  healthcare should be more open to private operators |  healthcare open to private operators
17 |  protecting the environment is more important than the economic growth |  environmental protection versus economic growth
18 |  cutting public spending is a good way to solve the economic crisis |  public spending cuts as a solution for the economic crisis
19 |  income support for the poorest sections of the population is positive for the Italian economy |  improve the economy by helping low-income segments of population
20 |  the introduction of a single tax rate on income (”Flat Tax”) would be beneficial for the Italian economy |  taxes with a single rate without progressive taxation
Table 4: Defined topic for each of the $20$ sentence (Italian version). nr. |  Sentence |  Topic
---|---|---
1 |  nel complesso, essere membri dell’UE è uno svantaggio |  svantaggi dell’Unione Europea
2 |  l’Italia dovrebbe uscire dall’Euro |  uscire dall’euro
3 |  dovrebbe esistere un esercito comune europeo |  esercito europeo comune
4 |  le multinazionali come Google e Youtube dovrebbero pagare i diritti d’autore e le tasse secondo le regole di ciascun paese europeo |  tasse per le multinazionali in relazione alle regole di ciascun Paese Europeo
5 |  l’integrazione economica europea si è spinta troppo oltre: gli Stati membri dovrebbero riguadagnare maggiore autonomia |  autonomia economica dei membri dell’Unione Europea
6 |  l’Unione Europea dovrebbe riformare la propria politica dell’immigrazione: l’Italia dovrebbe ricevere più supporto dagli altri Stati membri |  gestione dell’immigrazione nell’Unione Europea
7 |  l’Italia dovrebbe intensificare le sue relazioni economiche con la Cina |  relazioni economiche dell’Italia con la Cina
8 |  l’uso ricreativo della cannabis dovrebbe essere legale |  uso ricreativo della cannabis
9 |  l’Islam è una minaccia per i valori dell’Italia |  minaccia dell’Islam nei confronti dei valori italiani
10 |  alle donne deve essere garantita autonomia di scelta sull’aborto |  autonomia di scelta sull’aborto
11 |  ogni forma di auto-difesa all’interno della proprietà privata dovrebbe essere legittima |  legittima difesa nella propria abitazione con armi
12 |  le attività della magistratura devono essere indipendenti dalle pressioni della politica |  indipendenza della magistratura dalla politica
13 |  i bambini, nati in Italia da cittadini stranieri, dovrebbero ricevere la cittadinanza italiana automaticamente |  cittadinanza italiana per bambini nati in Italia da famiglie straniere
14 |  la ricchezza dovrebbe essere redistribuita dai cittadini più abbienti ai cittadini più poveri |  redistribuzione della ricchezza verso i piu poveri
15 |  le imprese dovrebbe poter licenziare i dipendenti più facilmente |  possibilita delle imprese di licenziare facilmente i propri dipendenti
16 |  la Sanità dovrebbe essere più aperta agli operatori privati |  apertura della Sanità ad operatori privati
17 |  proteggere l’ambiente è più importante della crescita economica |  importanza della protezione dell’ambiente
18 |  tagliare la spesa pubblica è un buon modo per risolvere la crisi economica |  tagli alla spesa pubblica come soluzione per la crisi economica
19 |  il sostegno al reddito alle fasce più povere della popolazione è positivo per l’economia italiana |  migliorare l’economia aiutando le fasce a basso reddito
20 |  l’introduzione di una aliquota unica sui redditi (”flat tax”) sarebbe di beneficio all’economia italiana |  conseguenze della flat tax per l’economia italiana

In detail, the $Topic$ $Filtering$ module (Fig. 2) receives in input the
Twitter Timeline $TL_{p}$ of Party $p$ and the topic $tp_{s}$ related to
sentence $s$; by exploiting the zero-shot classifier $C$, the in-topic tweets
$I^{p}_{tp_{s}}$ and the relative topic scores $T^{p}_{tp_{s}}$ are retrieved

$I^{p}_{tp_{s}}=\\{tw_{1},...,tw_{m}|C(tw_{i},tp_{s})>=th\\}$ (1)
$T^{p}_{tp_{s}}=\\{C(tw_{i},tp_{s})|tw_{i}\in I^{p}_{tp_{s}}\\}$ (2)

The C’s default hypothesis reformulation ”This text is about $<tp_{s}>$.” was
used. $C(tw_{i},tp_{s})\in[0,1]$ indicates how much tweet $tw_{i}$ refers to
topic $tp_{s}$. $th$ is the filtering threshold value that we varied during
the experiments defined in Section 6.2.

###  5.3 Agreement Predictor

Afterwards, the $Agreement$ $Predictor$ module (Fig. 2) computes the final
five-valued label $A^{p}_{s}$ through an algorithm
$Alg(T^{p}_{tp_{s}},S^{p}_{s})$, defining

$S^{p}_{s}=\\{C(tw_{i},s)|tw_{i}\in I^{p}_{tp_{s}}\\}$ (3)

as the C scores of tweets $I^{p}_{tp_{s}}$ with respect to sentence $s$; each
score suggests how much tweet $tw_{i}$ refers to and agrees with the sentence
$s$. In this case, the C hypothesis (the sentence $s$) was not reformulated,
but used as it is.

Each employed algorithm $Alg$ exploits one of the following mapping functions:

$M1(s)=\begin{cases}1&\text{if $s\in[0,0.2)$}\\\ 2&\text{if
$s\in[0.2,0.4)$}\\\ 3&\text{if $s\in[0.4,0.6)$}\\\ 4&\text{if
$s\in[0.6,0.8)$}\\\ 5&\text{if $s\in[0.8,1]$}\\\ \end{cases}$ (4)

or

$M2(s)=\begin{cases}1&\text{if $s\in[0,0.25)$}\\\ 2&\text{if
$s\in[0.25,0.5)$}\\\ 3&\text{if $s\in[0.5,0.75)$}\\\ 4&\text{if
$s\in[0.75,1]$}\\\ \end{cases}$ (5)

where the $M1(s)$’s integer values from $1$ to $5$ are related to the five
agreement/disagreement labels defined in Table 1, while $M2(s)$’s integer
values from $1$ to $4$ are related to an intermediate agreement/disagreement
Likert scale: $\\{1,2\\}$ have got the same meaning as in Table 1, then $3$
indicates $agree$ and $4$ stands for $completely$ $agree$; the reason behind
this intermediate mapping will be explained in $Alg4$ (subsection Algorithm
4). $s\in[0,1]$ refers to a score value both for $M1$ and $M2$.

The proposed algorithms ordered by complexity are the followings:

Algorithm 1

[Alg1] The label $A^{p}_{s}$ is computed as

$A^{p}_{s}=\begin{cases}M1(\frac{\sum_{i=1}^{|I^{p}_{tp_{s}}|}s_{i}\cdot
t_{i}}{\sum_{i=1}^{|I^{p}_{tp_{s}}|}s_{i}})&\text{if $\mid
I^{p}_{tp_{s}}\mid\neq 0$}\\\ 3&\text{otherwise}\end{cases}$ (6)

where $s_{i}\in S^{p}_{tp_{s}}$ and $t_{i}\in T^{p}_{tp_{s}}$.

Algorithm 2

[Alg2] First, it maps each tweet $tw_{i}\in I^{p}_{tp_{s}}$ into the label
$l_{i}\in\\{1,2,3,4,5\\}$ using its sentence score $s_{i}\in S^{p}_{s}$

$l_{i}=M1(s_{i})$ (7)

then, $A^{p}_{s}$ is

$A^{p}_{s}=\begin{cases}\Bigl{\lfloor}\frac{\sum^{|I^{p}_{tp_{s}}|}_{i=1}l_{i}}{|I^{p}_{tp_{s}}|}\Bigr{\rceil}&\text{if
$\mid I^{p}_{tp_{s}}\mid\neq 0$}\\\ 3&\text{otherwise}\end{cases}$ (8)

The step of assigning $l_{i}$ to each tweet $tw_{i}\in I^{p}_{tp_{s}}$ (Eq.
7), hopefully returns a more fair $A^{p}_{s}$. In fact, the tweet
normalization may help in aggregating the contribution of each tweet ($l_{i}$)
using the standard mean, which means applying the macro aggregation. In a
multi-class classification setup, macro-metric aggregation is preferable if it
is suspected that there may be class imbalance; in fact, the values $l_{i}$
are not balanced with respect to the current sentence $s$: likely, if a Party
$p$ agrees with a sentence, there will be lot of tweets in agreement with it
(many $l_{i}=4$ or $l_{i}=5$) and a few (errors) or no tweets in disagreement
(few labels $l_{i}=1$, or $l_{i}=2$, or $l_{i}=3$), and vice-versa.

Algorithm 3

[Alg3] Like $Alg2$, but slightly modifying how $A^{p}_{s}$ is computed (Eq.
8). Let’s further define $V_{l}$ as the number of voters for the integer label
$l\in\\{1,2,3,4,5\\}$

$V_{l}=|\\{l_{i}:l_{i}=l\\}^{|I^{p}_{tp_{s}}|}_{i=1}|$ (9)

where $l_{i}$ are the labels computed from Eq. 7. Let’s define $v=max(V_{l})$,
then

$\displaystyle A^{p}_{s}=$ $\displaystyle l$ if $|\\{l:V_{l}=v\\}|=1$ (10a)
$\displaystyle A^{p}_{s}=$
$\displaystyle\Bigl{\lfloor}\frac{\sum^{|I^{p}_{tp_{s}}|}_{i=1}l_{i}}{|I^{p}_{tp_{s}}|}\Bigr{\rceil}$
if $|\\{l:V_{l}=v\\}|>1$ (10b) $\displaystyle A^{p}_{s}=$ $\displaystyle 3$
otherwise (10c)

where $\Bigl{\lfloor}...\Bigr{\rceil}$ is the round function. The majority
voting (case 10a) may have a bigger contribution in assigning correct labels
than plain standard mean (case 10b taken from Eq. 8 of $Alg2$), since it
better accounts for class imbalance.

Algorithm 4

[Alg4] The previous algorithms take into consideration the neutral label
$nl=3$ (neither disagree, nor agree) also when $\mid I^{p}_{tp_{s}}\mid\neq
0$. However, we wondered how the results would change if $nl$ was only
considered when $\mid I^{p}_{tp_{s}}\mid=0$. The neutral label may also be
assigned in presence of a low number of in-topic $I^{p}_{tp_{s}}$: in this
particular situation, the user may have not taken a position about the current
sentence $s$ yet; also, choosing $A^{p}_{s}$ looking at just one tweet may not
be significant. Therefore, $Alg4$ stems from $Alg3$ having

$l_{i}=M2(s_{i})$ (11)

where $l_{i}\in\\{1,2,3,4\\}$; we define

$a^{p}_{s}=\begin{cases}3&\text{if $\mid I^{p}_{tp_{s}}\mid<m$}\\\
\text{majority voting (case \ref{eq:sub_case_majority_voting})}\\\
\text{rounded standard mean (case \ref{eq:sub_case_mean})}\\\ \end{cases}$
(12)

where $m$ is the minimum number of tweets for which the majority voting
algorithm or the standard mean are executed. Since the $\\{3,4\\}$ labels in
output from $M2(s)$ represent the $agree$ and $completely$ $agree$ final
labels, they must be mapped again to the real final integer labels $4$ and $5$
respectively (as coded in Table 1)

$A^{p}_{s}=\begin{cases}a^{p}_{s}&\text{if $a^{p}_{s}=1\lor a^{p}_{s}=2$}\\\
a^{p}_{s}+1&\text{if $a^{p}_{s}=3\lor a^{p}_{s}=4$}\\\ \end{cases}$ (13)

##  6 Experimental Setup

This section first defines the baseline methods to which compare our $T2S$
method. Then, the experiments are described adhering to a set of research
questions. Finally, the chosen evaluation metrics are explained.

###  6.1 Baselines

It is a good practice to compare the proposed methods with a bunch of
baselines. To the best of our knowledge, no baseline method has been devised
for the typology of our stance detection task yet: unlike our approach, the
state-of-the-art unsupervised user-stance detection method proposed by Darwish
et al. [9] cannot operate without context information from other users and it
is not suitable for a multi-class ordinal classification like our case.
Therefore, the following baselines to compute $A^{p}_{s}$ for Party $p$ and
sentence $s$ were used:

Random

$A^{p}_{s}$ is set to a random integer picked from a discrete uniform
distribution of $int\in[1,5]$. The numpy random
method777https://numpy.org/doc/stable/reference/random/generated/numpy.random.randint.html
was used with random seed set to 42.

Predict 3

$A^{p}_{s}$ is set to 3 (neither disagree, nor agree).

Sentence Bert

The newest Transformer-based language models like BERT can be used as feature
extractors [52], providing contextual word and sentence embeddings. The
Sentence-Bert architecture of the Sentence Transformers Python
library888https://www.sbert.net/ was used with the English all-mpnet-base-v2
model on translated tweets, and with the multi-lingual model distiluse-base-
multilingual-cased-v1 on the Italian tweets.

Let’s define $e_{i}$ as the embeddings of tweet $t_{i}\in T_{p}$ and $e_{s}$
as the embeddings of sentence $s$. $S$ is the set of cosine similarities
between each $e_{i}$ and $e_{s}$

$S=\\{s_{i}|s_{i}=cos\\_sim(e_{i},e_{s})\\}^{|T_{p}|}_{i=1}$ (14)

Since $s_{i}\in[-1,1]$, each $s_{i}$ is $MinMaxScaled$999https://scikit-
learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html to
$ns_{i}\in[0,1]$. Finally, $A^{p}_{s}$ is defined as the mapping $M1$ over the
average of the $top_{K}$ scores $ns_{i}$

$A^{p}_{s}=M1(\frac{\sum^{K}_{i=1}ns_{i}}{K})$ (15)

Heuristically, $K$ was set to 10.

###  6.2 Experiments in detail

As already explained in section 5, our $T2S$ method has got four parameters to
tune: the language model $LM$ to be used for zero-shot classification, the
dataset $Di$ from which extract the Twitter timeline $TL_{p}$, the algorithm
$Alg$ for the $Agreement$ step, and the threshold value $th$ for the $Topic$
$Filtering$ step. Considering the values of those parameters in Fig. 2, we
carried out each experiment having in mind the four research questions
summarized in Table 5 and ordered by specificity.

Table 5: Description of all carried out experiments Experiment Name |  Research Question
---|---
Best language model $LM$ |  Which is the best language model $LM$ for zero-shot classification? Which is the best model to deal with Italian tweets? All in all, is an English model better?
Best dataset $D$ |  Fixed the language model $LM$, which is the best dataset to work on, considering all proposed algorithms? Hence, which is the best time period to listen to before a Political Election?
Best algorithm $Alg$ |  Fixed the language model $LM$ and dataset $Di$, which is the best algorithm to work on, considering all evaluated thresholds $th$? Are all our proposed algorithms better than the baselines (subsection 6.1)? Are the more complex algorithms better or not?
Best threshold $th$ |  Fixed the language model $LM$, the dataset $D-Nmonths$ and the algorithm $Alg$, which is the best filtering threshold $th$, hence the optimal set-up?
Party and Sentence Analysis |  Fixed the optimal setup for our framework, which are the Parties on which $T2S$ behaves well or poorly? What about the sentences?

###  6.3 Evaluation

Generally, the performance of a prediction model is evaluated through several
error metrics such as the Mean Squared Error (MSE), the R2 Score, Residual
Plots and/or the Mean Absolute Error (MAE); moreover, [53] suggests to use the
Macro Averaged MAE for Ordinal Regression tasks with imbalanced classes.

Our task of predicting the agreement/disagreement level is close to Ordinal
Regression because there’s an inherent ordering between classes ({completely
disagree, disagree, neither disagree nor agree, agree, completely agree});
however, it differs from an ordinal regression because there are meaningful
differences between classes: for example, predicting disagree instead of
completely disagree is a more acceptable error than predicting neither
disagree nor agree or agree/completely agree. Besides, the class imbalance is
not an issue for our task: it doesn’t matter if more disagree or agree labels
(classes) $G^{p}_{s}$ must be predicted, the point is measuring the error
between the predicted ($A^{p}_{s}$) and true ($G^{p}_{s}$) values as fair as
possible. For these reasons, the Macro Averaged MAE was not used. Also, the
other error metrics do not fit well our task; however, since an appropriate
error metric has not been devised yet, the MAE metric was employed. The MSE
was used at first too, but it was just a replica of the MAE result, so it was
discarded.

Lastly, since the predicted value is an integer among $\\{1,2,3,4,5\\}$, a
classification evaluation metric was considered as well: the weighted F1 score
was picked, since it summarizes both Precision and Recall [54].

The sklearn.metrics Python package was used to compute both
MAE101010https://scikit-
learn.org/stable/modules/generated/sklearn.metrics.mean_absolute_error.html
and F1_weighted111111https://scikit-
learn.org/stable/modules/generated/sklearn.metrics.f1_score.html.

##  7 Results and Discussion

The results are discussed adhering to the experiments summarized in Table 5.
We also carried out the same experiments considering different tweet
typologies (e.g., without retweets, which represented a large percentage of
the total tweets), but we obtained worse results than using all of them
(original tweets, retweets, replies, quotes).

###  7.1 Best Language Model LM

First, we asked ourselves which is the best language model to be used for
zero-shot classification: among cross-lingual language models employed on
Italian tweets, is it better a model pre-trained on a mixture of languages
including Italian, or a model further fine-tuned on an Italian corpus?
Furthermore, would the results benefit from using an English language model on
translated tweets instead? We answered these questions by looking at Fig. 4:
each cell $(LM_{i},D_{j})$ indicates the minimum MAE (maximum F1) obtained
with our $T2S$ method for a certain language model $LM_{i}$ and dataset
$D_{j}$ by varying the algorithm $Alg$ and the threshold $th$ according to
Fig. 2.

Figure 4: Best MAE and F1 values of our $T2S$ method for each couple
$(LM_{i},D_{j})$ of language models and datasets. Darker colors indicate
optimal values for both metrics.

Among the cross-lingual models $XRoberta_{1}$ and $XRoberta_{2}$, the best one
seemed to be $XRoberta_{1}$: it had an overall better MAE, while F1 results
were close to $XRoberta_{2}$’s; we considered MAE as the first metric to judge
the performances, since it tells how much we are close to the correct answer.
Apparently, fine-tuning on an Italian translation of a subset of the MNLI
dataset ($XRoberta_{2}$) doesn’t contribute a lot to text classification in
our $T2S$ framework.

All in all, the best choice is translating the pre-processed tweets in English
and using an English model like $BART$: it reached significantly higher values
on both MAE and F1. Supposedly, using a model pre-trained and fine-tuned on a
single language gives better results for our prediction task: learning on a
single language allows to focus on more details, features of the language.

###  7.2 Best Dataset D

Fixed the language model $LM=BART$, the dataset $D_{4}$ was immediately
detected as the best one, since it had the best MAE and F1 (Fig. 4).
Presumably, the Twitter political discussion since four months before the
Italian elections were enough to grasp the Parties’ stances.

We evaluated the mean MAE and mean F1 for each cell $(LM_{i},D_{j})$ of Fig. 4
as well, but the results confirmed $BART$ and $D_{4}$ as the best language
model and dataset.

###  7.3 Best Algorithm Alg

Once the language model and dataset were chosen, we wondered whether our
algorithms $Alg$s are better than the three baselines $random$, $predict\\_3$
and $sentence\\_bert$, and which is the best $Alg$ considering all thresholds
$th$. To this aim, we fixed the language model $LM=BART$ and dataset $D_{4}$,
and we varied the algorithms $Alg$ and thresholds $th$s; Fig. 5 describes how
much each algorithm performed across different thresholds. These results
include the performances of the three baselines as well.

Figure 5: MAE and F1 of our four proposed algorithms $Alg$s and the three
baselines by varying the threshold $th$. It is shown $Alg4$ with $m=3$.

As far as it concerns the MAE, all proposed $Alg$ performed much better than
the three baselines. However, only $Alg3$ and $Alg4$ surpassed the baselines
on F1, indicating that aggregating the contribution of each tweet ($l_{i}$)
provides a better prediction precision than directly weight-averaging on the
sentence scores $s_{i}$ of each tweet. Altogether, the optimal algorithm can
be identified in $Alg3$: F1 seemed to contradict it and bend over $Alg4$
instead, but the gain over the prediction error is far more important. This
result suggests that assigning the neutral label (neither disagree, nor agree)
only when there’s a minimum number of tweets $m$ does not boost the
performance of our $T2S$ method. Also, we executed $Alg4$ with $m=\\{2,3\\}$,
finding out that the results didn’t vary a lot from each other; therefore, we
showed $Alg4_{m=3}$ in Fig. 5.

###  7.4 Best Threshold th

Fixed the language model $LM=BART$, the dataset $D_{4}$ and the algorithm
$Alg3$, threshold $th=0.6$ was immediately detected as the optimal one, since
it had the best MAE and a good F1 (Fig. 5).

Therefore, the best setup $su_{opt}$ of our $T2S$ framework was $(LM,$
$D_{j},$ $Alg,$ $th)=(BART,$ $D_{4},$ $Alg3,$ $0.6)$

###  7.5 Party and Sentence Analysis

####  7.5.1 Party

To explore the specific performance of our $T2S$ method over the Parties, we
used the optimal setup $su_{opt}$ but by varying the threshold $th$. Fig. 6
shows the results. Each point indicates the MAE (F1) on the 20 sentences’
agreement level $A^{p}_{s}$ for a certain Party $p$.

Figure 6: MAE and F1 computed for each Party over the stance predictions of
the $20$ VAA statements. The optimal $su_{opt}$ is used, but the threshold
$th$ varies.

Each Party behaves differently, thus it is likely that $T2S$ highly depends on
the Party’s timeline in terms of how much it generally writes, how much it
writes in-topic, how much it writes using figures of speech or hashtags and
emojis (which we removed). Looking at both the MAE and F1, we observed a
regular trend for thresholds $th=\\{0.8,0.9\\}$ for five parties out of six:
the outlier Party $Mov5Stelle$ was more predictable for those thresholds. That
may happen because the user’s timeline deal with a certain statement in a
clearer way; for example, looking at $Mov5Stelle$ and $forza\\_italia$’s
tweets filtered for the sentence $S19$ and $th=0.9$, we saw that $Mov5Stelle$
wrote clearer and explicit tweets supporting the argument (it completely
agrees), while from $forza\\_italia$’s timeline it’s not immediately clear
that it disagrees; $forza\\_italia$ tweeted about tax reduction, fewer fees on
families, and job creation, but it didn’t explicitly say that it disagrees
with income support for the poorest sections of the population being a
positive trait for the Italian economy, thus our $T2S$ framework assigned it
’completely agrees’.

####  7.5.2 Sentence

To explore the specific performance of our $T2S$ method over the sentences and
Parties, we used the optimal setup $su_{opt}$: Fig. 7 presents the absolute
error for each (Party, sentence), while Fig. 8 shows the error bars for the
Parties on which $T2S$ performed better ($Mov5Stelle$) and worse
($FratellidItalia$), and the Party having the highest number of absolute
errors equal to ’1’ ($LegaSalvini$); Fig. 10 displays the number of in-topic
tweets in output from the $Topic$ $Filtering$ step.

Figure 7: Absolute Error for each (party, sentence) couple using the optimal
setup $su_{opt}$.

The acceptable error value is ’1’, since on a five-valued Likert scale (Table
1) being wrong by one level does not have a great impact on the agree/disagree
levels, unless the correct label is the intermediate (neutral) one. To
understand whether the ’1’ errors are acceptable or not, an error measure
taking into account that predicting disagree instead of completely disagree is
a more acceptable error than predicting neither disagree nor agree or
agree/completely agree could have provided more insightful hints; however,
such an error measure has not been devised yet in literature. Therefore, such
insights can be analyzed by looking at error bars like the ones in Fig. 8:
again, the goodness of the ’1’ absolute errors depended on the Party’s
timeline, with $Mov5Stelle$ having $70\%$ of good ’1’, while $LegaSalvini$
only the $20\%$.

Figure 8: Error bars for $Mov5Stelle$, $FratellidItalia$ and $LegaSalvini$
using $su_{opt}$: $\blacklozenge$ indicates the ground-of-truth for a
sentence, while $|$ the predicted label.

Looking at both Fig. 7 and 10, it seems that there’s no correlation between
the error magnitude and the number of tweets. $S1$ clearly shows this,
therefore we investigated how $T2S$ with $su_{opt}$ took the decision about
each Party (Fig. 9): $pdnetwork$’s prediction ’completely disagree’ is
correct, indeed it wrote lot of pro Europe tweets; $forza\\_italia$ tweeted
about the need to change Europe, hence the ’disagree’ label correctly
assigned; $piu\\_europa$ completely disagrees with $S1$, but it was predicted
as ’neither disagree, nor agree’: out of 18 tweets, five talked about the
European countries not having neither common foreign policy nor common defence
and were labelled as ’neither disagree, nor agree’, eight were pro Europe
(five labelled as ’disagree’, three labelled as ’completely disagree’), while
the remaining ones talked about Europe in a negative way and hence labelled
either as ’agree’ or ’completely agree’. In this case, the standard mean was
applied (see Eq. 10b of $Alg3$) and rounded to ’3’ (’neither disagree, nor
agree’); this case suggests that more tweets supporting the correct stance are
needed to correctly predict the $A^{p}_{s}$ label. The majority of
$Mov5Stelle$’s tweets were pro Europe, hence our $T2S$ predicted ’completely
disagree’ instead of ’disagree’. Instead, $LegaSalvini$ is neutral about $S1$
but it was predicted as in disagreement: the majority of its tweets talked
about the need to change Europe; we believe that it’s difficult to predict
neutrality. Setting the neutral label only when there are no in-topic tweets
surely helps, and $Alg4$ should have brought advantages in this regard;
nonetheless, $Alg4$’s $m$ param is not easy to set, since it may depend on how
a Twitter user expresses itself. Last but not least, $FratellidItalia$ tweeted
a lot about the need to change Europe, hence it was predicted as in
disagreement with $S1$ (as correctly done for $forza\\_italia$); however, its
real stance is ’agree’. $FratellidItalia$’s tweets classified as in agreement
with $S1$ have got words like ’failure’, disappointment’, ’save from’,
’penalized’, ’disappear’ together with ’Europe’, ’EU’ or ’European’; again, we
found another proof regarding the dependency of our $T2S$ framework with how a
Twitter user communicates its stance about something.

Figure 9: Error bars for $S1$ ”overall, being EU members is a disadvantage”
(topic: ”European Union disadvantages”). $su_{opt}$ has been used.
$\blacklozenge$ indicates the ground-of-truth for a sentence, while $|$ the
predicted label. Figure 10: Number of tweets over which each $A^{p}_{s}$ is
computed. The optimal setup $su_{opt}$ is used.

###  7.6 Possible limitations of our framework

The usage of the Tweets2Stance framework has to be seen in light of some
limitations that can be addressed in several ways. First and foremost, the
availability of the official position of a social media user $u$ (here an
Italian Party) about a social-political statement $s$ is crucial to measure
our framework’s performance; if we wanted to generalize T2S to social media
users different from political parties and dealing with other typologies of
statements (e.g., about extremist ideologies), two steps could be followed: a
domain expert would be needed to accurately design the statements, and users
should voluntarily express their opinions through a VAA-like questionnaire.
Second, it is highly probable that using a more accurate translation model
than the plain Google Translator would increase the performances. Finally, a
ZSC may not be pre-trained on current hot-topics and new coined words, hence
it performs poorly on those arguments; we encountered this issue when looking
at results for sentence $s13$ (Table LABEL:tab:topic_definition_eng) about the
automatic citizenship of children who were born in Italy to foreign parents
(related to the Italian decree-law called ”DDL Zan”): the three tested ZSC
models didn’t recognize tweets containing ”DDL” and ”Zan” words as associated
to sentence $s13$. This limitation can be addressed in the future by using the
most sophisticated and updated ZSC models, which hopefully will deal with
current hot-topics.

##  8 Conclusions and Future Works

In this work we introduced Tweets2Stance, a novel unsupervised stance-
detection framework which uses the timeline of a Social Media user to infer
its stance towards a reference political-social statement on a 5-value Likert
scale. In particular, we dealt with the stance of 20 political statements for
the six major parties in Italy. Results showed that, although the general
maximum F1 value was $0.4$, $T2S$ could correctly predict the stance with a
general minimum MAE of $1.13$, which is a great achievement considering that
MAE tells how much we are close to the correct answer, and that we worked with
a final five-valued label. Also, as we hypothesized, the $T2S$’s performance
highly depends on how the Twitter account of the Party (hence the social media
user) writes, e.g. the employed figures of speech, the words used, and so on.
Notice that we removed both hashtags and emojis because the ZSC component had
issues in processing them, since it was not pre-trained on social media texts;
however, since they may contain useful information to the ZSC component, it
would be worth trying to include and normalize both of them by separating each
hashtag’s words and removing the $\\#$ symbol, and by substituting each emoji
with words. As mentioned when introducing the work, our major goal herein is
to design and evaluate, through multiple experiments, a framework working as a
stance detector potentially generalizable to several topics. If applied to
political discourse, it could cover the first step of a pipeline whose output
is the user’s political leaning. In the near future, we will investigate how
T2S’s agreement levels output can be used to that aim. Besides, we hope to
apply it to detect extremists accounts on social media; however, a domain-
expert may be needed to define precise social statements to use. At last, our
$T2S$ coped-with Italian tweets by translating them to English, hence it can
be applied to English tweets as well, and hopefully reaching better
performances.

## Acknowledgments

We are immensely grateful to the Observatory on Political Parties and
Representation [12] for providing us the 2019 official position of the six
major Italian parties about 20 political statements [13].

This work was supported by the SoBigData++: European Integrated Infrastructure
for Social Mining and Big Data Analytics project [871042].

## References

  * [1] D. Biber, E. Finegan, Adverbial stance types in english, Discourse processes 11 (1) (1988) 1–34.
  * [2] A. ALDayel, W. Magdy, Stance detection on social media: State of the art and trends, Information Processing & Management 58 (4) (2021) 102597.
  * [3] M. Dias, K. Becker, Inf-ufrgs-opinion-mining at semeval-2016 task 6: Automatic generation of a training corpus for unsupervised identification of stance in tweets, in: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 378–383.
  * [4] Y. Igarashi, H. Komatsu, S. Kobayashi, N. Okazaki, K. Inui, Tohoku at semeval-2016 task 6: Feature-based model versus convolutional neural network for stance detection, in: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 2016, pp. 401–407.
  * [5] I. Augenstein, T. Rocktäschel, A. Vlachos, K. Bontcheva, Stance detection with bidirectional conditional encoding, arXiv preprint arXiv:1606.05464.
  * [6] S. Mohammad, S. Kiritchenko, P. Sobhani, X. Zhu, C. Cherry, Semeval-2016 task 6: Detecting stance in tweets, in: Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), 2016, pp. 31–41.
  * [7] S. Hamidian, M. T. Diab, Rumor detection and classification for twitter data, arXiv preprint arXiv:1912.08926.
  * [8] K. Darwish, W. Magdy, T. Zanouda, Improved stance prediction in a user similarity feature space, in: Proceedings of the 2017 IEEE/ACM international conference on advances in social networks analysis and mining 2017, 2017, pp. 145–148.
  * [9] K. Darwish, P. Stefanov, M. Aupetit, P. Nakov, Unsupervised user stance detection on twitter, in: Proceedings of the International AAAI Conference on Web and Social Media, Vol. 14, 2020, pp. 141–152.
  * [10] L. Cedroni, Voting Advice Applications in Europe: The state of the art, Scriptaweb, 2010.
  * [11] T. Louwerse, M. Rosema, The design effects of voting advice applications: Comparing methods of calculating matches, Acta politica 49 (3) (2014) 286–312.
  * [12] OPPR, Opi - observatory on political parties and representation [cited 2010-04-13].
URL http://opi.sp.unipi.it/opi-political-parties/

  * [13] O. on Political Parties, R. (OPPR), Navigatoreelettorale europee 2019 (2019) [cited 2010-04-13].
URL http://opi.sp.unipi.it/opi-political-parties/oppr-projects/

  * [14] R. Cohen, D. Ruths, Classifying political orientation on twitter: It’s not easy!, in: Proceedings of the International AAAI Conference on Web and Social Media, Vol. 7, 2013.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, Curran Associates Inc., 2017, p. 6000–6010.
  * [16] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, H.-W. Hon, Unified language model pre-training for natural language understanding and generation, Advances in Neural Information Processing Systems 32.
  * [17] M.-W. Chang, L. Ratinov, D. Roth, V. Srikumar, Importance of semantic representation: Dataless classification, in: Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, AAAI Press, 2008, p. 830–835.
  * [18] A. Murakami, R. Raymond, Support or oppose? classifying positions in online debates from reply activities and opinion expressions, in: Coling 2010: Posters, 2010, pp. 869–875.
  * [19] M. A. Walker, P. Anand, R. Abbott, J. E. F. Tree, C. Martell, J. King, That is your evidence?: Classifying stance in online political debate, Decision Support Systems 53 (4) (2012) 719–729.
  * [20] A. Aldayel, W. Magdy, Your stance is exposed! analysing possible factors for stance detection on social media, Proceedings of the ACM on Human-Computer Interaction 3 (CSCW) (2019) 1–20.
  * [21] V. Lynn, S. Giorgi, N. Balasubramanian, H. A. Schwartz, Tweet classification without the tweet: An empirical examination of user versus document attributes, in: Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, 2019, pp. 18–28.
  * [22] S. Gottipati, M. Qiu, L. Yang, F. Zhu, J. Jiang, Predicting user’s political party using ideological stances, in: International Conference on Social Informatics, Springer, 2013, pp. 177–191.
  * [23] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3914–3923.  doi:10.18653/v1/D19-1404.
URL https://aclanthology.org/D19-1404

  * [24] U. A. Siddiqua, A. N. Chy, M. Aono, Tweet stance detection using multi-kernel convolution and attentive lstm variants, IEICE TRANSACTIONS on Information and Systems 102 (12) (2019) 2493–2503.
  * [25] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach, arXiv preprint arXiv:1909.00161.
  * [26] P. Sobhani, D. Inkpen, X. Zhu, A dataset for multi-target stance detection, in: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 2017, pp. 551–557.
  * [27] M. Lai, V. Patti, G. Ruffo, P. Rosso, Stance evolution and twitter interactions in an italian political debate, in: International Conference on Applications of Natural Language to Information Systems, Springer, 2018, pp. 15–27.
  * [28] S. Ghosh, P. Singhania, S. Singh, K. Rudra, S. Ghosh, Stance detection in web and social media: a comparative study, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2019, pp. 75–87.
  * [29] B. Zhang, M. Yang, X. Li, Y. Ye, X. Xu, K. Dai, Enhancing cross-target stance detection with transferable semantic-emotion knowledge, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 3188–3197.
  * [30] A. Joshi, P. Bhattacharyya, M. Carman, Political issue extraction model: A novel hierarchical topic model that uses tweets by political and non-political authors, in: Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 2016, pp. 82–90.
  * [31] T. Fagni, S. Cresci, Fine-Grained Prediction of Political Leaning on Social Media with Unsupervised Deep Learning, Journal of Artificial Intelligence Research 73 (2022) 633–672.
  * [32] A. Rashed, M. Kutlu, K. Darwish, T. Elsayed, C. Bayrak, Embeddings-based clustering for target specific stances: The case of a polarized turkey, arXiv preprint arXiv:2005.09649.
  * [33] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, X. Huang, Pre-trained models for natural language processing: A survey, Science China Technological Sciences 63 (10) (2020) 1872–1897.
  * [34] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, In: OpenAI Blog [Internet] (2019) [cited 2021 March 09].
URL https://openai.com/blog/better-language-models/

  * [35] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186.  doi:10.18653/v1/N19-1423.
URL https://aclanthology.org/N19-1423

  * [36] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, in: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 1877–1901.
URL
https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

  * [37] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45.  doi:10.18653/v1/2020.emnlp-demos.6.
URL https://aclanthology.org/2020.emnlp-demos.6

  * [38] M. Polignano, P. Basile, M. de Gemmis, G. Semeraro, V. Basile, Alberto: Italian BERT language understanding model for NLP challenging tasks based on tweets, in: R. Bernardi, R. Navigli, G. Semeraro (Eds.), Proceedings of the Sixth Italian Conference on Computational Linguistics, Bari, Italy, November 13-15, 2019, Vol. 2481 of CEUR Workshop Proceedings, CEUR-WS.org, 2019.
URL http://ceur-ws.org/Vol-2481/paper57.pdf

  * [39] D. Carmo, M. Piau, I. Campiotti, R. Nogueira, R. de Alencar Lotufo, PTT5: pretraining and validating the T5 model on brazilian portuguese data, CoRR abs/2008.09144.  arXiv:2008.09144.
URL https://arxiv.org/abs/2008.09144

  * [40] W. de Vries, A. van Cranenburgh, A. Bisazza, T. Caselli, G. v. Noord, M. Nissim, BERTje: A Dutch BERT Model, arXiv:1912.09582 (Dec. 2019).
URL http://arxiv.org/abs/1912.09582

  * [41] A. CONNEAU, G. Lample, Cross-lingual language model pretraining, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems, Vol. 32, Curran Associates, Inc., 2019\.
URL
https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf

  * [42] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, V. Stoyanov, Unsupervised cross-lingual representation learning at scale, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 8440–8451.  doi:10.18653/v1/2020.acl-main.747.
URL https://aclanthology.org/2020.acl-main.747

  * [43] Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, L. Zettlemoyer, Multilingual Denoising Pre-training for Neural Machine Translation, Transactions of the Association for Computational Linguistics 8 (2020) 726–742.  arXiv:https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl\\_a\\_00343/1923401/tacl\\_a\\_00343.pdf, doi:10.1162/tacl_a_00343.
URL https://doi.org/10.1162/tacl_a_00343

  * [44] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, C. Raffel, mT5: A massively multilingual pre-trained text-to-text transformer, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 483–498.  doi:10.18653/v1/2021.naacl-main.41.
URL https://aclanthology.org/2021.naacl-main.41

  * [45] M. Muffo, E. Bertino, Bertino: An italian distilbert model, in: CLiC-it, 2020.
  * [46] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 7871–7880.  doi:10.18653/v1/2020.acl-main.703.
URL https://aclanthology.org/2020.acl-main.703

  * [47] A. Williams, N. Nangia, S. Bowman, A broad-coverage challenge corpus for sentence understanding through inference, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Association for Computational Linguistics, 2018, pp. 1112–1122.
URL http://aclweb.org/anthology/N18-1101

  * [48] The checkpoint for bart-large after being trained on the multinli (mnli) dataset - hugginface, https://huggingface.co/facebook/bart-large-mnli, (Accessed on 01/18/2022) (08 2021).
  * [49] A. Conneau, R. Rinott, G. Lample, A. Williams, S. Bowman, H. Schwenk, V. Stoyanov, XNLI: Evaluating cross-lingual sentence representations, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Brussels, Belgium, 2018, pp. 2475–2485.  doi:10.18653/v1/D18-1269.
URL https://aclanthology.org/D18-1269

  * [50] Xlm-roberta-large fine-tuned on a combination of nli data in 15 languages, https://huggingface.co/joeddav/xlm-roberta-large-xnli, (Accessed on 01/18/2022) (03 2021).
  * [51] This model takes xlm-roberta-large and fine-tunes it on a subset of nli data taken from a automatically translated version of the mnli corpus, https://huggingface.co/Jiva/xlm-roberta-large-it-mnli, (Accessed on 01/18/2022) (10 2021).
  * [52] N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using Siamese BERT-networks, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019, pp. 3982–3992.  doi:10.18653/v1/D19-1410.
URL https://aclanthology.org/D19-1410

  * [53] S. Baccianella, A. Esuli, F. Sebastiani, Evaluation measures for ordinal regression, in: 2009 Ninth International Conference on Intelligent Systems Design and Applications, 2009, pp. 283–287.  doi:10.1109/ISDA.2009.230.
  * [54] F. Sebastiani, Machine learning in automated text categorization, ACM computing surveys (CSUR) 34 (1) (2002) 1–47.


& \phi_{\tilde{E}}(\effname)[V^r/x][\lambda o. \hndl {E^r(o)} {x} {E[\gamma_2(x)]} {\phi_E} /k]
) \\
&\quad \quad \in \simierel {\sigma} {\omega} {\simivrel {B} {}}.
\end{align*}

By transitivity (Lemma <ref>), it will suffice to show

\begin{align*}
( &E[E^l[\raiseOpwithM{\effname}{V^l}]], \\
& \hndl {E^l(\raiseOpwithM{\effname}{V^l})} {x} {E[x]} {\phi_E}
) \\
&\quad \quad \in \simierel {\sigma} {k} {\simivrel {B} {}}.
\end{align*}

Suppose first that $E$ does not catch $\effname$.
Observe that neither term above steps, so we will show they are related in
$\simirrel {\sigma} {k} {\simivrel {B} {}}$.

To this end, we need to show that $E[E^l]$ and $\hndl {E^l} {x} {E[x]} {\phi_E}$
are related later in the K-relation. In other words, let $(V_1, V_2) \in \simivrel {A} {k'}$.
We need to show that

\begin{align*}
( &E[E^l[\raiseOpwithM{\effname}{V^l}]], \\
& \hndl {E^l(\raiseOpwithM{\effname}{V^l})} {x} {E[x]} {\phi_E}
) \\
&\quad \quad \in (\later \simierel {\sigma})_{k'} (\simivrel {B} {}).
\end{align*}

But this follows from the Löb induction hypothesis (with $M = E^l[\raiseOpwithM{\effname}{V^l}])$.

Suppose $M \stepsin {i_1} {M'}$ and $\dncast {B} {A} {N}\stepsin {i_2} {N'}$, and
suppose $(M', N') \in \simierel {d_\sigma} {j} {\simivrel A {}}$.

Then $(M, N) \in \simierel {d_\sigma} {j} {\simivrel A {}}$.