# Wearing Masks Implies Refuting Trump?: Towards Target-specific User Stance
Prediction across Events in COVID-19 and US Election 2020

Hong Zhang<EMAIL_ADDRESS>0000-0003-1193-0337 Singapore Management
UniversitySingapore178903 ,  Haewoon Kwak<EMAIL_ADDRESS>0000-0003-1418-0834
Indiana University BloomingtonBloomington, INUSA47408 ,  Wei Gao
<EMAIL_ADDRESS>Singapore Management UniversitySingapore178903 and  Jisun An
<EMAIL_ADDRESS>0000-0002-4353-8009 Indiana University
BloomingtonBloomington, INUSA47408

(2023)

###### Abstract.

People who share similar opinions towards controversial topics could form an
echo chamber and may share similar political views toward other topics as
well. The existence of such connections, which we call _connected behavior_ ,
gives researchers a unique opportunity to predict how one would behave for a
future event given their past behaviors. In this work, we propose a framework
to conduct connected behavior analysis. Neural stance detection models are
trained on Twitter data collected on three seemingly independent topics, i.e.,
wearing a mask, racial equality, and Trump, to detect people’s stance, which
we consider as their online behavior in each topic-related event. Our results
reveal a strong connection between the stances toward the three topical events
and demonstrate the power of past behaviors in predicting one’s future
behavior.

stance detection, natural language processing, COVID-19, distant supervision

††journalyear: 2023††copyright: acmlicensed††conference: 15th ACM Web Science
Conference 2023; April 30-May 1, 2023; Evanston, TX, USA††booktitle: 15th ACM
Web Science Conference 2023 (WebSci ’23), April 30-May 1, 2023, Evanston, TX,
USA††price: 15.00††doi: 10.1145/3578503.3583606††isbn:
979-8-4007-0089-7/23/04††ccs: Applied computing Sociology††ccs: Computing
methodologies Natural language processing

##  1\. Introduction

COVID-19 unfolded in late 2019 and rapidly spread to the whole world. In the
following two years, the virus caused more than 860,000 deaths in the US and
infected more than 69 million people. It became one of the most severe
economic, political, and social catastrophes that impacted countless
businesses and lives. As the number of cases surged in the US since its first
occurrence, it quickly became a politicized health crisis (Young et al.,
2022).

According to a survey by Pew Research (Van Green and Tyson, 2020), 78% of
Democrats believed that COVID-19 was a major threat to the health of the US
population, whereas only 52% of Republicans shared the same view. Donald
Trump, the 45th president of the United States from 2017 to 2021, made public
announcements on multiple occasions stating that he would not wear a mask
despite the US Centers for Disease Control and Prevention (CDC) advising doing
so. Trump became an icon for anti-mask, and his action spurred the same
sentiment toward wearing masks for many Republican supporters. Partisanship
was one of the most important factors for mask adoption; the willingness to
wear a mask is significantly lower in counties where Trump found strong
support during the 2016 presidential election (Kahane, 2021). Many studies
found political influence on mask adoption as a response to COVID-19 pandemic
(Young et al., 2022; Shin et al., 2022; Adolph et al., 2022).

On May 25, 2020, Minneapolis police officer arrested George Floyd, a 46-year-
old black man, following a 911 call reporting Mr. Floyd bought cigarettes from
a convenience store with a counterfeit bill. Unfortunately, a series of
violent and fatal actions by officers left Mr. Floyd unable to breathe,
leading to his death. The tragic death of George Floyd caused social unrest
across the United States, and tens of thousands of people swarmed the streets
to express their outrage toward racial inequality (Taylor, 2021). A study had
shown that Trump was unconcerned about racial inequality; he actually appeared
to set out to exacerbate and inflame the issues in the United States (McClain,
2021). A poll in 2020 also found that two-thirds of Americans think Trump has
increased racial tensions following the death of George Floyd (MONTANARO,
2020). Mask adoption and racial inequalities are not the only politicized
social issues. Recent studies found that opinions about gun control (Mejova et
al., 2022) and hate towards Asians during COVID-19 (An et al., 2021) are also
shown to be politicized.

We increasingly observe that some opinions and behaviors are linked to each
other. This could be due to the existence of “echo chambers” (Cinelli et al.,
2021; Del Vicario et al., 2016; Colleoni et al., 2014) on social media. Social
media users with a certain view on a topic tend to post and read content from
people having a similar view (An et al., 2014). This creates echo chambers
where users favoring content adhering to their beliefs form online communities
with a shared narrative. People within an echo chamber expose themselves to
information reinforcing their own opinions, which could facilitate social
extremism (Lee et al., 2022a) and political polarization (Barberá et al.,
2015). As such, people who share similar opinions towards controversial topics
could form an echo chamber and share similar political views toward other
topics as well.

The existence of such connections, which we call _connected behavior_ , gives
researchers a unique opportunity to predict how one would behave for a future
event given their past behaviors. Such ability to explain and predict future
behavior could bring immense value to both the scientific community and the
business world. Despite the unprecedented body of work done on understanding
online opinion and behavior based on social media data (Mahmud et al., 2016;
Bing et al., 2014; Kallus, 2014; O’Connor et al., 2010), little was done on
the ability to predict one’s behavior given their behaviors in related events.

Our goal is to investigate how one’s behavior in a target event is associated
with his behaviors in other related events (i.e., _connected behavior_) and,
ultimately, predict one’s future behavior given his previous behaviors. As a
first step to studying connected behavior, this study considers opinion
expression as online behavior. In particular, we examine individuals’ stances
on three seemingly independent but related controversial topics during
COVID-19: wearing masks, racial equality, and Donald Trump. These topics
correspond to one’s behavior in three events in 2020: mask adoption, racial
unrest, and US Election. We adopt the stance detection technique to
effectively and accurately identify stances towards selected topics in the
study population. Stance detection has been a popular task studied by many NLP
researchers in recent years; it detects a user’s opinion toward a specific
target. Targets range from abstract ideas to concrete entities and events
(Ranade et al., 2013).

In this study, we collect and study tweets on the three selected controversial
topics and classify users’ stances towards each topic using NLP techniques and
neural network models. We then analyze users’ stances on three topics to
uncover any underlying relationship and validate the theory of connected
behavior. Our contributions are summarized as follows:

  * •

We empirically validate connected behavior on three selected topics, which can
be helpful for researchers in the field of political and social science. We
believe this is the first-of-its-kind study that examines the existence of
connected behavior in a data-driven way, going beyond self-reported surveys
and small-scale experiments.

  * •

We propose a novel two-step stance detection framework that allows for a fast
and effective way of analyzing connected behavior on multiple topics without
having costly labeled data.

  * •

We build a model that predicts user stance on an event based on his stances on
the related events and compares it with the models based on other content-
based features. We demonstrate the predictive power of stance as compared to
other features.

  * •

A large dataset is collected for three controversial topics to study social
polarization and NLP-related tasks. The data is available at
https://github.com/tommyzhanghong/connected_behavior.

##  2\. Background

The United States presidential election 2020 was an unprecedented event. The
incumbent president, Republican Donald Trump, was defeated by the former 47th
vice president of the United States, Democrat Joe Biden. The 2020 election had
the highest number of voter turnout in the entire history of the United
States, casting nearly 158.4 million ballots (Desilver, 2021). Even after the
election results were announced, Trump acknowledged Biden had won but refused
to concede the election. Trump claimed the election was rigged and launched a
slew of lawsuits in several states that he had lost, but was unsuccessful
(bbc, 2020). Widespread acceptance of Trump’s prolonged baseless insistence
about the stolen election ultimately led to the storming of the US Capitol by
Trump supporters on January 6, 2021 (Lee et al., 2022b). The intense
competition between Trump and Biden started long before the election when
parties had run presidential and congressional campaigns that cost almost $14
billion—more than double the amount for the 2016 election (Horton et al.,
2020).

The advent of technology has reshaped the landscape of election campaigns;
politicians across the world actively use social media to communicate with a
general audience (Hong, 2013). Trump’s Twitter account, the @realdonaldtrump
handle, had 88.7 million followers at the time it was suspended in January
2021. It was one of the major channels Trump used to communicate with the
public during his campaign. Communications generated on social media about US
Election 2020 and its related topics had provided us with a large volume of
data for this study.

##  3\. Related Work

###  3.1. Opinion Mining on Social Media

Twitter is one of the most popular social media, which began as an SMS text-
based service. The micro-blog platform originally has a length limit of 140
characters for each tweet, which was driven by the 160-character length of
SMS. Over time, Twitter evolved to allow up to 280 characters for each tweet.
As opposed to traditional media, Twitter has provided a platform for users to
produce media content and voice their opinion publicly. This spawned the
creation of a large public collection of opinions for many events happening
around the globe (Kwak et al., 2010; Wang et al., 2012; Conover et al., 2011).

Many fields have enjoyed rapid growth since the creation of social media, as
it is the first time in human history that researchers get access to a huge
volume of digital data expressing individual opinions. Opinion mining, the
field of studying and analyzing people’s opinions and sentiments towards an
entity, has certainly been one of them (Chen and Zimbra, 2010; Fang and Zhan,
2015; Kharde et al., 2016). Chen and Zimbra (2010) had applied opinion mining
on online forum messages to understand business constituents’ opinions towards
a company. Fang and Zhan (2015) and Kharde et al. (2016) leveraged the huge
volume of data on Twitter to conduct studies on sentimental analysis.

###  3.2. Stance Detection

Stance detection is the task of automatically determining from the text
whether the author of the text is in favor of, against, or neutral towards a
proposition or target (Mohammad et al., 2016). Stance is usually expressed in
the form of categorical values, such as “Support”, “Neutral” and “Against”.
Stance detection can be applied in a variety of downstream tasks, for example,
to classify the stance of posts on online debate forum (Walker et al., 2012)
and to help detect fake news and false rumours (Ma et al., 2018).

Stance detection can be divided into two categories (Ghosh et al., 2019).
Target-specific stance detection considers each individual target separately,
whereas multiple-target stance detection jointly detects stance towards
multiple related targets (Sobhani et al., 2017). In this study, a target-
specific stance detection model is built for each of the three topics to
classify _stance of individual users_.

There are two ways to approach a stance detection task. Unsupervised learning
involves capturing user clusters and labeling the stance for each cluster
(Darwish et al., 2020). While it is effective at identifying clusters of vocal
users with similar stances, it fails on less vocal users who only post very
few tweets (Samih and Darwish, 2020). On the other hand, supervised learning
leverages labeled training data to perform classification. Many supervised
classification methods are developed for this purpose (Igarashi et al., 2016;
Lai et al., 2020). However, it requires obtaining a large amount of training
data, which can be expensive.

###  3.3. Language Models

Since the development of Recurrent Neural Networks (RNN) and Long Short-Term
Memory (LSTM), they have been used widely in the area of language modeling and
machine translation. However, RNN and LSTM are unable to enjoy the full
benefit of the huge computation power we have now with GPU and parallelization
due to their sequential computation design. The attention mechanism provides
an effective way for sequence modeling by leveraging parallelization, allowing
the model to capture dependencies regardless of their distance in the
sequence. In 2017, Google proposed the Transformer model (Vaswani et al.,
2017) for sequence transduction tasks. BERT (Bidirectional Encoder
Representations from Transformer) based on Transformers revolutionized the NLP
world by producing state-of-the-art performance on numerous tasks (Zhang et
al., 2020; Wang et al., 2019). It is pre-trained on a large corpus of
unlabeled text and can be fine-tuned with just one additional output layer to
perform a wide range of tasks.

###  3.4. Distant Supervision

As deep learning requires a large amount of labeled training data, manual
labeling becomes more and more costly and non-scalable. On social media,
distant supervision using hashtags has been widely used to obtain weak labels
for political affiliation (Maulana and Situngkir, 2020), the emotion of tweets
(Hasan et al., 2014), or stance toward constitutional referendum (Di Giovanni
and Brambilla, 2021) due to its cost-effectiveness. We propose a more robust
hashtag-based labeling method for a stance detection task on a huge corpus of
English tweets. Instead of discarding tweets containing hashtags of opposing
stances, we design a three-way classifier with a non-opinionated class. It
classifies tweets based on a scoring approach. A neural stance detection model
based on BERT is subsequently built in the next step to achieve even higher
performance on a small sample of ground truth.

##  4\. Our Dataset

Since the event, US Election 2020, is closely related to mask adoption and
racial unrest events, we chose Trump as the target topic of the connected
behavior study and selected wearing masks and racial equality as related
topics to study their impact on people’s behavior during the US Election 2020.

The selection criteria for wearing masks and racial equality were based on
four main factors. Firstly, both of them are highly controversial events which
had spurred huge debates on social media and created a large volume of content
containing individual opinions to support our study. Secondly, COVID-19
response measures and racial equality were among the issues covered in the
presidential debate (Kaplan and Michael, 2020). The clear ideological division
between Trump and Biden could induce political partisanship in people’s
behaviors. Next, topics should follow a chronological order for us to model
causal effect, i.e., the events of mask adoption and racial unrest should take
place before the US Election to affect people’s decisions in the election,
hence validating the ability to predict human behavior. Lastly, both topics
were selected for taking place in the same year as US Election 2020 to
minimize opinion drift over time.

###  4.1. Topic-related Twitter Data

For this study, we collected a total of 106,580,239 tweets on three selected
events from Twitter using its official Academic API (twi, 2022). For each
topic, we collected tweets using topic-related keywords in the API query. We
carefully selected those keywords via a systematic approach. We first queried
100 tweets for each topic using root keywords describing the topic, such as
‘election’ for the topic Trump, ‘COVID’ and ‘mask’ for the wearing masks
topic, and ‘George Floyd’ and ‘#blacklivesmatter’ for racial equality topic
(Yang, 2016). Then, we extracted a list of topic-related keywords by
inspecting those tweets. The keywords selected were mostly generic words,
phrases, or hashtags related to each topic, to avoid introducing stance bias
in the data collection process. Full lists of the keywords used for data
collection are listed in Appendix A.

Specifically, we collected 81,028,363 Trump-related tweets for a period of one
month between November 3 to December 2, 2020. Trump-related Twitter activity
for this period was extremely high as people were very expressive about their
political inclination. We collected 14,834,164 wearing masks tweets from April
3 to November 2, 2020. CDC made a statement on April 3, 2020, recommending
people to wear face masks following a record high daily death due to COVID-19.
It has spurred a debate about mask adoption, and many people voiced their
opinion on Twitter. For racial equality, a few controversial events happened
in 2020, including the tragic incident of George Floyd, which sparked a huge
debate on social media platforms (Barkan, 2020; Weine et al., 2020). We
gathered 10,717,712 racial equality related tweets from May 25, 2020, the date
George Floyd’s incident took place, to November 2, 2020. Data collection time
range for both wearing masks and racial equality topics were selected to
benefit from the explosion of Twitter activity due to major events and to
ensure tweets collected were close to the US Election 2020 to minimize opinion
drift.

###  4.2. Data Cleaning

To further clean our dataset, we removed retweets, bot-likely accounts, and
less active accounts to retain users of interest.

Retweets. For this study, we only look at original tweets to detect users’
stance towards target topics as retweets or quoted tweets may not always mean
a user endorsing the stance of the original tweets (Opgenhaffen and
Scheerlinck, 2014), which could add noise if included. Thus, we excluded all
retweets and quoted tweets from our dataset.

Bots and Spammer. We further removed bot-likely accounts to better capture
individual opinions of real human users as they tend to be automated to post
information (Ferrara et al., 2016). One of the strong signatures of these bot
accounts is that they leave a much larger amount of tweets than genuine users
typically do. We calculated the number of posts per user and removed the top 5
percentile by user post count as spammer accounts. We manually examined those
highly active users with different thresholds for bots removal and observed a
significantly higher percentage of tweets having no opinion at the top 5
percentile of users.

Users who rarely post. To more reliably detect one’s stance towards a topic,
we excluded those users who have only posted one tweet for the given topic in
the defined time frame.

The above three cleaning steps left us with 29 million tweets across the three
topics.

###  4.3. User-level Twitter Data

We further collected users’ profile information (i.e., location, description,
protected account, verified account, # of followers, # of friends, # of
tweets, and # of lists that include the user) and their historical tweets
using Twitter API as baseline information to validate the additional
prediction power that stance can provide. For each user, we collected the most
recent 100 tweets between November 3, 2019 to November 2, 2020 to engineer
individual-level features.

###  4.4. User Social Network

In order to build a social network graph to validate the quality of stances
obtained from the proposed framework, we collected retweets from the most
active 10,000 users ranked by the number of posts for a period of 1 year prior
to the US Election 2020.

##  5\. Distantly Supervised Stance Detection

Supervised learning requires labeled training data. However, manually labeling
a large amount of data is an expensive and time-consuming task (Chen et al.,
2016; Magdy et al., 2015). Many previous works on NLP classification tasks
obtained manually annotated data via crowdsourcing (Becker et al., 2011), but
it was costly and sometimes the results were unreliable if the workers did not
have sufficient knowledge of the topic of tweets they were assigned to
annotate. To overcome these challenges, we developed a hashtag-based stance
labeling method to obtain weak labels for millions of tweets with little human
effort. Then, we use distant supervision techniques to build a BERT-based
stance detector.

Table 1. Result of hashtag-based stance labeling. | Against | Non-opinionated | Support
---|---|---|---
Trump | 209,662 | 1,741,650 | 250,625
Wearing Masks | 18,677 | 486,706 | 192,357
Racial Equality | 46,763 | 1,648,350 | 253,579
Table 2. Stance detection results. Performance is measured using macro F1
score rounded off to two decimal places. The best performing model for each
topic with hashtag masked is shown in bold.

Topic | Sampling Method | # of Samples | Baseline | BERT
---|---|---|---|---
Random | NB | SVM | Train w/ hashtag | Train w/ hashtag masked
Test w/ hashtag | Test w/o hashtag | Test w/ hashtag | Test w/o hashtag
Trump | Random | 100,000 | 0.27 | 0.33 | 0.38 | 0.99 | 0.30 | 0.42 | 0.47
Stratified | 54,000 | 0.33 | 0.55 | 0.52 | 0.99 | 0.19 | 0.57 | 0.58
90,000 | 0.33 | 0.56 | 0.53 | 1.00 | 0.18 | 0.60 | 0.60
Mask | Random | 100,000 | 0.28 | 0.34 | 0.54 | 1.00 | 0.35 | 0.34 | 0.58
Stratified | 54,000 | 0.33 | 0.59 | 0.61 | 0.99 | 0.35 | 0.49 | 0.69
Racial | Random | 100,000 | 0.24 | 0.34 | 0.45 | 1.00 | 0.31 | 0.43 | 0.54
Stratified | 54,000 | 0.33 | 0.54 | 0.59 | 1.00 | 0.17 | 0.66 | 0.67
90,000 | 0.33 | 0.63 | 0.60 | 1.00 | 0.17 | 0.70 | 0.68
135,000 | 0.33 | 0.63 | 0.62 | 1.00 | 0.17 | 0.70 | 0.69

Table 3. Model performance on the 100 tweets randomly sampled and manually
labeled. Macro F1 score for best performing model are shown in bold.

| Baseline | Hashtag-based stance labeling | BERT-based stance detector
---|---|---|---
| Random | NB | SVM | Against | Non-opinionated | Support | Against | Non-opinionated | Support
Gold Labels | Against | - | - | - | 12 | 21 | 0 | 22 | 9 | 2
Non-opinionated | - | - | - | 2 | 51 | 2 | 8 | 45 | 2
Support | - | - | - | 0 | 9 | 3 | 1 | 3 | 8
F1 Score | 0.29 | 0.56 | 0.51 | 0.54 | 0.72

###  5.1. Hashtag-Based Stance Labeling

A hashtag is a word or phrase without space preceded by a hash symbol (#),
which is used as a keyword to indicate the content of a tweet or the topic it
is related to (Small, 2011). On Twitter, users primarily use hashtags to
convey sentiments and opinions (Han and Baldwin, 2011). Many studies have
shown the effectiveness of hashtags to be used in tweets analytic tasks (She
and Chen, 2014; An and Weber, 2016).

To label the stance of tweets using hashtags, a set of stance-expressing
hashtags is required for both the “Support” and “Against” classes. To achieve
this, we examined the top 50 most frequently used hashtags for each topic, and
extracted a set of stance-expressing hashtags representing Support or Against
for every topic. Some hashtags were used to reference a topic or attract
attention, such as #election2020. These hashtags do not show any stance and
were hence omitted from the study. Hashtags which were obvious in their
stance, such as #trumpisaloser, were added to their respective class of
stance-expressing hashtags. For hashtags whose stance is not obvious to tell,
we randomly sampled 20 tweets containing each of these hashtags to manually
annotate them, before allocating them to their respective groups.

It has been observed that users with a certain stance toward a topic tend to
use similar hashtags. To improve the discriminating power of the hashtag-based
stance labeling, we expanded the list of hashtags for the “Support” and
“Against” classes by adding a strong stance-expressing hashtags discovered
during the manual labeling process.

Another challenge with using hashtags to determine the stance of tweets is
that a tweet can contain more than one hashtag, and tweets can even contain
hashtags expressing opposing stances. For example, #maga is a hashtag used
mainly by Trump supporters to support Trump’s election campaign. However, in
the tweet ”Yet again, #MAGA #Republicans @GOP refuse to hold #TraitorTrump
accountable for removal of @EPA protections. @realDonaldTrump Pile of poo
#TrumpIsALaughingStock #TrumpCrimeFamilyForPrison #TrumpIsACompleteFailure
#TrumpIsALoser #ClimateEmergency #ToxicTrump”, the author used #maga as a
subject to be criticised. The tweet contains hashtags having both supporting
(#MAGA) and against (#TraitorTrump, #TrumpIsALaughingStock,
#TrumpCrimeFamilyForPrison, #TrumpIsACompleteFailure, #TrumpIsALoser,
#ToxicTrump) stance towards Trump. To address this issue, we assumed that,
when a tweet contains more “Support” hashtags than “Against” hashtags, it is
more likely to be a tweet with a “Support” stance and vice versa. With this,
we created a hashtag-based scoring algorithm relying on the ratio between
“Support” and “Against” hashtags for stance labeling. The detailed algorithm
can be found in Appendix B.

By hashtag-based stance labeling, we produced stance labels for approximately
5 million tweets. Table 1 presents the number of tweets for each stance.
However, only about 15% of all tweets contain hashtags. To further classify
all 29 million tweets, we build a stance detection model, which is described
in the next section.

###  5.2. BERT-based Stance Detector

In a study that compares the performance of numerous language models for the
stance detection task on two separate datasets, Ghosh et al. (2019) observed
that BERT outperforms other widely used models, such as convolution neural
network and SVM. Similar to (Ghosh et al., 2019), we used the pre-trained
BERT-Base Uncased model with 12 layers, 768 hidden size, 12 self-attention
heads, and 110M parameters in this study. The output of the first head of the
final layer is fed to a feed-forward layer along with softmax, and the model
is fine-tuned to perform a target-specific stance detection task. We split the
data in 80%:20% ratio for training and testing, and report macro F1 score in
Table 2.

The BERT stance detection model was fine-tuned only on tweets labeled by the
hashtag-based labeling method described in $\S$5.1. Since the majority of
tweets did not contain a hashtag, we needed to ensure that the model tuned on
tweets with hashtags can be generalized to tweets without hashtags. As such,
we conducted experiments with various models trained on data with different
sampling methods, training data sizes, and hashtag mask settings.

We allowed hashtags in both training and test data to be masked to examine the
generalizability of the model. With randomly sampled 100,000 tweets, the model
achieved excellent performance with F1 score of 0.99. However, this high
performance was expected as stances were labeled based on hashtags. The
model’s performance is only meaningful when hashtags in test data were masked.
With hashtags masked in test data, the model’s F1 score dropped significantly.
It dropped to 0.3, 0.35, and 0.31 for Trump, wearing masks, and racial
equality topics, respectively. One way to improve the generalizability of the
model is to train it with masked hashtags so that the model learns semantic
and linguistic features of input text rather than hashtags. After masking
hashtags in training data, the performance increased to 0.47, 0.58, and 0.54
for the three topics, respectively.

It is observed that the data is highly imbalanced. In Table 1, almost 80% of
tweets were labeled as “Non-opinionated.” Furthermore, only about 2% of tweets
expressed “Against” stance in wearing masks and racial equality data. This
resulted in most stances being likely to be classified as the majority class.
Thus, we used stratified sampling to address this issue. The stratified
sampling method has effectively improved the model performance, achieving F1
score of 0.6 for Trump and about 0.68 for wearing masks and racial equality
topics.

From the results of Trump and racial equality topics, we found that increasing
the size of training data does not yield a significant improvement in model
performance. As a result, we chose 54,000 tweets with masked hashtags by
stratified sampling as our training data.

We created three baseline models as references. We present their performance
in Table 2. Both multinomial Naive Bayes and SVM classifiers were implemented
using scikit-learn library (Pedregosa et al., 2011). The SVM classifier was
applied with a linear kernel and without class weight. CountVectorizer from
scikit-learn library was used to remove English stop words and generate
unigram features. As expected, BERT achieved better performance than baseline
models in all sampling methods and sample size configurations.

Figure 1. Retweet Networks of the three topics. A node represents a user, and
an edge represents a retweet between two users. Nodes are colored by the
user’s stance towards each topic. Red denotes Support and blue denotes
Against. All three networks show that the two clusters are well separated.

###  5.3. Model Performance

The BERT-based stance detection model was trained using labeled data from the
hashtag-based stance labeling method. It is important to analyze the
performance of both models to understand the quality of data we have for
studying connected behavior. To achieve this, we randomly sampled 100 tweets
from Trump data and manually annotated them. These manually annotated stances
are treated as “gold labels” to be compared with results from baseline models
and two stance detectors proposed. Table 3 summarized the result.

Although the F1 score for hashtag-based stance labeling is only 0.54, this is
largely driven by a low recall. Many “Support” and “Against” tweets were
classified as “Non-opinionated” because limited hashtags were chosen for
hashtag-based labeling to avoid false positives. Instead, the accuracy is high
as shown in Table 3. Having high accuracy is crucial as it allows us to build
a good BERT-based stance classifier based on quality data. The BERT-based
stance detection was able to achieve a significantly better performance of
0.78, overcoming the limitation of having insufficient stance expressing
hashtags. For example, the hashtag #Trump is frequently used by both Trump
supporters and haters, a tweet with #Trump is labeled as Non-opinionated by
the hashtag-based stance labeling method rather than either “Support” or
“Against”. In the tweet ” _Time to get serious people…we need to get #Trump
out of there…hes broken so many laws…_ ”, the BERT-based model successfully
captured the meaning of the entire sentence and classified it as “Against”
Trump.

###  5.4. Validation Using A Retweet Network

Retweet has been shown to be an effective network feature to reveal the
stances of users (Darwish et al., 2020). To validate the quality of stances
obtained using our proposed distantly supervised stance detection framework,
we performed a network analysis on the most 10,000 active users ranked by the
number of posts. The retweet network is created with nodes representing users
and edges representing retweets between users. Nodes without any edges were
dropped, resulting in a network with 3,104 nodes and 9,546 edges. In Figure 1,
users formed two clusters with dense connections within the cluster and sparse
connections between them. The ratio of intra-stance edges to inter-stance
edges is 4.26, 1.14, and 1.79 for topics of Trump, wearing masks, and racial
equality, respectively. Coloring each node by stance obtained via the proposed
framework, we observed that the majority of users within the same cluster have
the same stance.

##  6\. Connected Behavior Analysis

Connected behavior analysis is the study of the relationship between people’s
behaviors in related events. To examine connected behavior among the three
topics, users who have shown a stance on all three topics are required. This
reduced the size of the dataset from the result of the BERT-based stance
classifier. A total of 183,848 Twitter users satisfied this condition.
Moreover, we found that the high proportion of Non-opinionated tweets
overshadowed the number of tweets containing a stance. We thus removed all
Non-opinionated tweets and only consider tweets with “Support” or “Against”
stances. In total, we have 99,342 users who have expressed “Support” or
“Against” towards all three topics.

Figure 2. Correlation between stance of each topic pair

Left and middle chart show dense clusters at top left and bottom right
quadrant, and right chart shows dense clusters at bottom left and top right
quadrant. Coupled with regression line, it demonstrated strong correlation
between stance one each pair of topics.

Next, we aggregated stances at the user level by computing the percentage of
tweets with a Support stance out of all topic-related tweets posted by a user.
The percentage of Support shows the inclination of a user towards supporting
that topic. Figure 2 illustrated a correlation of the percentages of Support
between each pair of topics. Black dashed lines divide each subplot into 4
quadrants. Most people who support wearing masks were against Trump, most
people who support racial equality were also against Trump, and most people
who were against racial equality were against wearing masks. Hypothesis tests
were performed to verify the significance of Spearman’s rank correlation
coefficient between each pair of topics. As shown in Table 4, Spearman’s rank
correlations between all topics were significant at $p<0.05$ significance
level.

Table 4. Spearman’s rank correlation coefficients and 2-tailed p-values Topics | Correlation | p-value
---|---|---
Trump | Wearing Masks | -0.34 | 0.0
Trump | Racial Equality | -0.44 | 0.0
Wearing Masks | Racial Equality | 0.32 | 0.0

If users have similar percentages of Support and Against tweets for a single
topic, their stances can be considered weak. We omit them for further
analysis. By a manual inspection, we empirically set the percentage threshold
as 55%; when users have a percentage of Support greater than or equal to 55%,
they are likely to support the topic. However, when users have support
percentage smaller or equal to 45%, they are likely to be against the topic.
Accordingly, we categorized every user as either “Support” or “Against”
towards each of the three topics. The result has demonstrated the presence of
strongly connected behavior between topics of wearing masks, racial equality,
and Trump. As illustrated in Table 5, if a person supports wearing masks and
racial equality, there is only a probability of 6% that they would vote for
Trump. By contrast, if a person is against wearing masks and racial equality,
there is a very high probability of 71% that he would vote for Trump.

Table 5. Probability of a person voting for trump given his stance on wearing masks and racial equality. | Wearing Masks
---|---
| Support | Against
Racial Equality | Support | 0.06 | 0.17
Against | 0.19 | 0.71

##  7\. Connected Behavior Prediction

After establishing a strong correlation between people’s stance on wearing
masks, racial equality, and Trump, we performed an in-depth study to validate
the capability of predicting a person’s stance on Trump given his stances on
wearing masks and racial equality. This analysis outlines the value of
predicting people’s behavior in a target event given their behaviors in
related events and paves the way for further exploration.

Table 6. Ablation study of proposed features for predicting stance on Trump.
Dim. indicates the dimension of features for each model.

Model | # | Features | Dim. | F1 | Accuracy
---|---|---|---|---|---
Baseline | 1 | Random | - | 0.47 | 0.50
2 | Majority Class | - | 0.42 | 0.73
Content-agnostic features | 3 | Profile | 1,536 | 0.62 | 0.76
4 | Statistics | 7 | 0.48 | 0.73
5 | Prof. + Stats. | 1,543 | 0.64 | 0.77
Content Features | 6 | Historical | 768 | 0.81 | 0.86
7 | Mask | 768 | 0.76 | 0.82
8 | Racial | 768 | 0.82 | 0.86
9 | Mask + Racial | 1,536 | 0.84 | 0.88
10 | Mask + Racial + Historical | 2,304 | 0.84 | 0.88
Stance features | 11 | All | 2 | 0.80 | 0.84
Combination | 12 | Row 5 + Mask Stance | 1,544 | 0.71 | 0.79
13 | Row 5 + Racial Stance | 1,544 | 0.76 | 0.82
14 | Row 5 + All Stance | 1,545 | 0.79 | 0.84
15 | Row 5 + Historical Cont. | 2311 | 0.81 | 0.86
16 | Row 15 + All Stance | 2,313 | 0.83 | 0.88
17 | Row 15 + Mask Cont. + Racial Cont. | 3,847 | 0.85 | 0.89
18 | Row 15 + Mask Cont. + Racial Cont. + All Stance | 3,849 | 0.85 | 0.89

###  7.1. Features

We use users’ stances on wearing masks and racial equality as features to
build a prediction model for their stance on Trump. In addition, we extract 1)
content features from their historical tweets and 2) content-agnostic features
from their profile and Twitter-related statistics to perform an ablation
study.

####  7.1.1. Content Features

We used three content features for Trump stance prediction. We extracted mask
and racial equality-related content features using their respective topic-
related tweets. We also extracted historical content features using each
user’s most recent 100 tweets, excluding all topic-related tweets. For each
content feature, we encoded tweets with Sentence-BERT (SBERT) (Reimers and
Gurevych, 2019) and compute their average embeddings as a user-level
representation.

####  7.1.2. Content Agnostic Features

Twitter profile features were extracted from the profile description and URL
listed in each user’s profile. They were both encoded using SBERT and averaged
to obtain user-level content-agnostic representation. Twitter statistics
features include: 1) number of followers, 2) number of followings, 3) number
of tweets posted by the user, 4) number of public lists that the user is a
member of, 5) number of days since the creation of account, 6) whether the
account protects its tweets, and 7) whether the account is verified.

###  7.2. Experiment Setup

With 69,308 users, excluding deleted users whose profile or content could not
be collected, we trained an XGBoost classifier to predict a user’s stance on
Trump. Stance features, content features, and content-agnostic features were
evaluated separately and in combination to compare the model’s performance.
Numerical content-agnostic features were standardized, and categorical
content-agnostic features and stance features were encoded as one-hot vectors
in the pre-possessing stage. We split the data in an 80%:20% ratio for
training and testing. Macro F1 score and accuracy were presented for each
model.

###  7.3. Impact of Stance

Table 6 shows the result of the connected behavior prediction model. Models
using Twitter profile features, content features, and stance features
exhibited substantially better performance than baseline models. Models with
racial equality-related content features or stance consistently achieved
better performance than models using those of wearing masks. It implies the
racial equality topic is more closely related to Trump than wearing masks,
making the stance on racial equality a better predictor for the stance on
Trump. The model with only stance features on wearing masks and racial
equality achieves an F1 score of 0.8, which highlights the importance of
stance features and the potential of connected behavior prediction. Although
the improvement is marginal, it is worth noting that content features of
wearing masks and racial equality give a better performance than stance
features. It is reasonable because those content features somehow encode
stances regarding those topics as well, but they require a much larger model
with remarkably higher dimensions. If we extend our model to cover more topics
and issues, predictions based on stances only will have a huge advantage in
terms of implementations over those based on all the content features.

##  8\. Conclusion and Limitations

In this study, we have collected a large volume of Twitter data on three
social issues and proposed a hashtag-based mechanism to annotate training data
with three-way stance labels. We also built a BERT-based distantly supervised
model extending stance detection to all tweets collected regardless of hashtag
inclusion. Connected behavior analysis has shown strong correlations between
people’s stances on wearing masks, racial equality, and Trump. Finally, as the
first step towards connected behavior research, we studied the capability of
predicting people’s behavior in a target event (e.g., not voting for Trump)
based on their behaviors in related events (e.g., wearing masks or support for
racial equality). The result is promising. Stances toward related events are
good predictors of the stance toward a target event.

There are some limitations in this work. Firstly, the hashtag-based stance
labeling method relies on a list of carefully curated hashtags, and its
performance depends on the quality and quantity of stance-expressing hashtags
picked. Due to various constraints, the 50 most frequently used hashtags for
each topic were manually inspected for hashtag-based stance labeling. However,
hashtags can exist in any words combinations, and 50 might be insufficient to
cover the vast range of hashtags used by Twitter users. Hence, tweets
containing stance-expressing hashtags that were not studied would have been
classified as Non-opinionated using the hashtag-based stance labeling method,
resulting in a low recall. The overall F1 score for the hashtag-based stance
detector from the manual evaluation is 0.54, similar to that of machine
learning baseline models. Having too few stance-expressing hashtags has led to
imbalanced training data, and the quality of hashtags picked has a direct
impact on labeling accuracy. If we examine more hashtags, we might expect an
improvement in model performance. Alternatively, obtaining fully labeled data
by humans would be ideal for training stance detection models, although being
very costly. Secondly, bot removal was performed by evaluating users who have
posted a significantly large amount of tweets within our dataset. A more
effective method could be to use the BotOrNot service proposed by Davis et al.
(2016). The BotOrNot service uses more than 1,000 features, including network,
user, temporal, content, and sentimental features, to classify an account as
either a bot or human. The BotOrNot API has a rate limit of 180 requests per
15 minutes. Given enough time, the BotOrNot service could be applied to
identify and remove bot accounts more accurately and allow us to focus on
human users for connected behavior study.

## Ethics Statement

This work aims to predict human behaviors in target events given his behaviors
in related events, a question that can be answered by analyzing individual-
level Twitter data. For individual Twitter accounts, we take several steps to
limit the potential negative impacts of our work. We only collect publicly
available tweets to detect the stance of users toward controversial events and
report aggregate information. Considering the sensitive level of topics
discussed, we refrain from singling out individuals. Lastly, for sharing our
tweet data, we will publish only a list of tweet IDs, without any text or
account information, according to Twitter’s guidelines.

###### Acknowledgements.

This research is supported by the Singapore Ministry of Education (MOE)
Academic Research Fund (AcRF) Tier 1 grant and the SMU-A*STAR Joint Lab in
Social and Human-Centered Computing (Grant No. SAJL-2022-CSS003).

## References

  * [1]
  * bbc [2020] 2020\.  US election: Trump says Biden won but again refuses to concede.  _BBC_ (Nov 2020).  https://www.bbc.com/news/election-us-2020-54952098
  * twi [2022] 2022\.  Twitter API for academic research — products — twitter developer platform.  https://developer.twitter.com/en/products/twitter-api/academic-research
  * Adolph et al. [2022] Christopher Adolph, Kenya Amano, Bree Bang-Jensen, Nancy Fullman, Beatrice Magistro, Grace Reinke, and John Wilkerson. 2022.  Governor partisanship explains the adoption of statewide mask mandates in response to COVID-19.  _State Politics & Policy Quarterly_ 22, 1 (2022), 24–49.
  * An et al. [2021] Jisun An, Haewoon Kwak, Claire Seungeun Lee, Bogang Jun, and Yong-Yeol Ahn. 2021.  Predicting Anti-Asian Hateful Users on Twitter during COVID-19. In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * An et al. [2014] Jisun An, Daniele Quercia, and Jon Crowcroft. 2014\.  Partisan sharing: Facebook evidence and societal consequences. In _Proceedings of the second ACM conference on Online social networks_. 13–24.
  * An and Weber [2016] Jisun An and Ingmar Weber. 2016.  # greysanatomy vs.# yankees: Demographics and Hashtag Use on Twitter. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 10. 523–526.
  * Barberá et al. [2015] Pablo Barberá, John T Jost, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau. 2015.  Tweeting from left to right: Is online political communication more than an echo chamber?  _Psychological science_ 26, 10 (2015), 1531–1542.
  * Barkan [2020] Ross Barkan. 2020\.  High-profile Killings of Unarmed Black People Spark Calls for Reform.  _ABAJ_ 106 (2020), 14–16.
  * Becker et al. [2011] Hila Becker, Mor Naaman, and Luis Gravano. 2011.  Beyond trending topics: Real-world event identification on twitter. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 5. 438–441.
  * Bing et al. [2014] Li Bing, Keith CC Chan, and Carol Ou. 2014.  Public sentiment analysis in Twitter data for prediction of a company’s stock price movements. In _2014 IEEE 11th International Conference on e-Business Engineering_. IEEE, 232–239.
  * Chen et al. [2016] Chao Chen, Yu Wang, Jun Zhang, Yang Xiang, Wanlei Zhou, and Geyong Min. 2016\.  Statistical features-based real-time detection of drifted Twitter spam.  _IEEE Transactions on Information Forensics and Security_ 12, 4 (2016), 914–925.
  * Chen and Zimbra [2010] Hsinchun Chen and David Zimbra. 2010.  AI and opinion mining.  _IEEE Intelligent Systems_ 25, 3 (2010), 74–80.
  * Cinelli et al. [2021] Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. 2021.  The echo chamber effect on social media.  _Proceedings of the National Academy of Sciences_ 118, 9 (2021).
  * Colleoni et al. [2014] Elanor Colleoni, Alessandro Rozza, and Adam Arvidsson. 2014\.  Echo chamber or public sphere? Predicting political orientation and measuring political homophily in Twitter using big data.  _Journal of communication_ 64, 2 (2014), 317–332.
  * Conover et al. [2011] Michael D Conover, Bruno Gonçalves, Jacob Ratkiewicz, Alessandro Flammini, and Filippo Menczer. 2011.  Predicting the political alignment of twitter users. In _2011 IEEE third international conference on privacy, security, risk and trust and 2011 IEEE third international conference on social computing_. IEEE, 192–199.
  * Darwish et al. [2020] Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020.  Unsupervised user stance detection on twitter. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 14. 141–152.
  * Davis et al. [2016] Clayton Allen Davis, Onur Varol, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2016\.  Botornot: A system to evaluate social bots. In _Proceedings of the 25th international conference companion on world wide web_. 273–274.  https://doi.org/10.1145/2872518.2889302
  * Del Vicario et al. [2016] Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H Eugene Stanley, and Walter Quattrociocchi. 2016.  The spreading of misinformation online.  _Proceedings of the National Academy of Sciences_ 113, 3 (2016), 554–559.
  * Desilver [2021] Drew Desilver. 2021\.  Turnout soared in 2020 as nearly two-thirds of eligible U.S. voters cast ballots for president.  _Pew Research Center_ (Jan 2021).  https://www.pewresearch.org/fact-tank/2021/01/28/turnout-soared-in-2020-as-nearly-two-thirds-of-eligible-u-s-voters-cast-ballots-for-president/
  * Di Giovanni and Brambilla [2021] Marco Di Giovanni and Marco Brambilla. 2021.  Content-based Stance Classification of Tweets about the 2020 Italian Constitutional Referendum. In _Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media_. 14–23.
  * Fang and Zhan [2015] Xing Fang and Justin Zhan. 2015.  Sentiment analysis using product review data.  _Journal of Big Data_ 2, 1 (2015), 1–14.
  * Ferrara et al. [2016] Emilio Ferrara, Onur Varol, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2016\.  The rise of social bots.  _Commun. ACM_ 59, 7 (2016), 96–104.
  * Ghosh et al. [2019] Shalmoli Ghosh, Prajwal Singhania, Siddharth Singh, Koustav Rudra, and Saptarshi Ghosh. 2019\.  Stance detection in web and social media: a comparative study. In _International Conference of the Cross-Language Evaluation Forum for European Languages_. Springer, 75–87.
  * Han and Baldwin [2011] Bo Han and Timothy Baldwin. 2011.  Lexical normalisation of short text messages: Makn sens a# twitter. In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_. 368–378.
  * Hasan et al. [2014] Maryam Hasan, Emmanuel Agu, and Elke Rundensteiner. 2014\.  Using hashtags as labels for supervised learning of emotions in twitter messages. In _Acm sigkdd workshop on health informatics, new york, usa_.
  * Hong [2013] Sounman Hong. 2013\.  Who benefits from Twitter? Social media and political competition in the US House of Representatives.  _Government Information Quarterly_ 30, 4 (2013), 464–472.
  * Horton et al. [2020] Jake Horton, Sarah Glatte, and Soraya Auer. 2020\.  US election 2020: How much did it cost and who paid for it?  _BBC_ (Nov 2020).  https://www.bbc.com/news/av/election-us-2020-54696386
  * Igarashi et al. [2016] Yuki Igarashi, Hiroya Komatsu, Sosuke Kobayashi, Naoaki Okazaki, and Kentaro Inui. 2016\.  Tohoku at SemEval-2016 task 6: Feature-based model versus convolutional neural network for stance detection. In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_. 401–407.
  * Kahane [2021] Leo H Kahane. 2021\.  Politicizing the mask: Political, economic and demographic factors affecting mask wearing behavior in the USA.  _Eastern economic journal_ 47, 2 (2021), 163–183.
  * Kallus [2014] Nathan Kallus. 2014\.  Predicting crowd behavior with big public data. In _Proceedings of the 23rd International Conference on World Wide Web_. 625–630.
  * Kaplan and Michael [2020] Thomas Kaplan and D. Shear Michael. 2020.  Biden and Trump will debate six main issues. here’s where they stand.  https://www.nytimes.com/2020/09/29/us/politics/biden-on-the-issues-trump-on-the-issues-debate.html
  * Kharde et al. [2016] Vishal Kharde, Prof Sonawane, et al. 2016\.  Sentiment analysis of twitter data: a survey of techniques.  _arXiv preprint arXiv:1601.06971_ (2016).
  * Kwak et al. [2010] Haewoon Kwak, Changhyun Lee, Hosung Park, and Sue Moon. 2010\.  What is Twitter, a social network or a news media?. In _Proceedings of the 19th international conference on World wide web_. 591–600.
  * Lai et al. [2020] Mirko Lai, Alessandra Teresa Cignarella, Delia Irazú Hernández Farías, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020\.  Multilingual stance detection in social media political debates.  _Computer Speech & Language_ 63 (2020), 101075.
  * Lee et al. [2022a] Claire Seungeun Lee, Juan Merizalde, John D Colautti, Jisun An, and Haewoon Kwak. 2022a.  Storm the Capitol: Linking Offline Political Speech and Online Twitter Extra-Representational Participation on QAnon and the January 6 Insurrection.  _Frontiers in Sociology_ 7 (2022).
  * Lee et al. [2022b] Claire Seungeun Lee, Juan Merizalde, John D. Colautti, Jisun An, and Haewoon Kwak. 2022b.  Storm the Capitol: Linking Offline Political Speech and Online Twitter Extra-Representational Participation on QAnon and the January 6 Insurrection.  _Frontiers in Sociology_ 7 (2022).  https://doi.org/10.3389/fsoc.2022.876070
  * Ma et al. [2018] Jing Ma, Wei Gao, and Kam-Fai Wong. 2018.  Detect rumor and stance jointly by neural multi-task learning. In _Companion proceedings of the the web conference 2018_. 585–593.
  * Magdy et al. [2015] Walid Magdy, Hassan Sajjad, Tarek El-Ganainy, and Fabrizio Sebastiani. 2015.  Distant supervision for tweet classification using youtube labels. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 9. 638–641.
  * Mahmud et al. [2016] Jalal Mahmud, Geli Fei, Anbang Xu, Aditya Pal, and Michelle Zhou. 2016.  Predicting attitude and actions of twitter users. In _Proceedings of the 21st International Conference on Intelligent User Interfaces_. 2–6.
  * Maulana and Situngkir [2020] Ardian Maulana and Hokky Situngkir. 2020.  Measuring Media Partisanship during Election: The Case of 2019 Indonesia Election.  (2020).
  * McClain [2021] Paula D. McClain. 2021\.  Trump and racial equality in America? No pretense at all!  _Policy Studies_ 42 (2021), 491–508.
  * Mejova et al. [2022] Yelena Mejova, Jisun An, Gianmarco De Francisci Morales, and Haewoon Kwak. 2022.  Modeling political activism around gun debate via social media.  _Transactions on Social Computing_ 5, 1-4 (2022), 1–28.
  * Mohammad et al. [2016] Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. 2016.  Semeval-2016 task 6: Detecting stance in tweets. In _Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016)_. 31–41.
  * MONTANARO [2020] DOMENICO MONTANARO. 2020\.  Majority Of Americans Say Trump Increased Racial Tensions After George Floyd’s Death, Poll Finds.  _The New York Times Company_ (Jun 2020).  https://www.npr.org/2020/06/06/871404645/majority-of-americans-say-trump-increased-racial-tensions-after-george-floyds-de
  * O’Connor et al. [2010] Brendan O’Connor, Ramnath Balasubramanyan, Bryan R Routledge, and Noah A Smith. 2010.  From tweets to polls: Linking text sentiment to public opinion time series. In _Fourth international AAAI conference on weblogs and social media_.
  * Opgenhaffen and Scheerlinck [2014] Michaël Opgenhaffen and Harald Scheerlinck. 2014.  Social media guidelines for journalists: An investigation into the sense and nonsense among Flemish journalists.  _Journalism Practice_ 8, 6 (2014), 726–741.
  * Pedregosa et al. [2011] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011\.  Scikit-learn: Machine learning in Python.  _the Journal of machine Learning research_ 12 (2011), 2825–2830.
  * Ranade et al. [2013] Sarvesh Ranade, Rajeev Sangal, and Radhika Mamidi. 2013\.  Stance classification in online debates by recognizing users’ intentions. In _Proceedings of the SIGDIAL 2013 Conference_. 61–69.
  * Reimers and Gurevych [2019] Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Samih and Darwish [2020] Younes Samih and Kareem Darwish. 2020.  A few topical tweets are enough for effective user-level stance detection.  _arXiv preprint arXiv:2004.03485_ (2020).
  * She and Chen [2014] Jieying She and Lei Chen. 2014.  Tomoha: Topic model-based hashtag recommendation on twitter. In _Proceedings of the 23rd international conference on World Wide Web_. 371–372.
  * Shin et al. [2022] Jieun Shin, Aimei Yang, Wenlin Liu, Hye Min Kim, Alvin Zhou, and Jingyi Sun. 2022\.  Mask-wearing as a partisan issue: Social identity and communication of party norms on social media among political elites.  _Social Media+ Society_ 8, 1 (2022), 20563051221086233\.
  * Small [2011] Tamara A Small. 2011\.  What the hashtag? A content analysis of Canadian politics on Twitter.  _Information, communication & society_ 14, 6 (2011), 872–895.
  * Sobhani et al. [2017] Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017\.  A dataset for multi-target stance detection. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. 551–557.
  * Taylor [2021] Derrick Bryson Taylor. 2021\.  George Floyd Protests: A Timeline.  _The New York Times Company_ (Nov 2021).  https://www.nytimes.com/article/george-floyd-protests-timeline.html
  * Van Green and Tyson [2020] Ted Van Green and Alec Tyson. 2020.  facts about partisan reactions to COVID-19 in the US.  _Pew Research Center_ 2 (2020).
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Walker et al. [2012] Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012\.  Stance classification using dialogic properties of persuasion. In _Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies_. 592–596.
  * Wang et al. [2012] Hao Wang, Doğan Can, Abe Kazemzadeh, François Bar, and Shrikanth Narayanan. 2012\.  A system for real-time twitter sentiment analysis of 2012 us presidential election cycle. In _Proceedings of the ACL 2012 system demonstrations_. 115–120.
  * Wang et al. [2019] Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019.  Multi-passage bert: A globally normalized bert model for open-domain question answering.  _arXiv preprint arXiv:1908.08167_ (2019).
  * Weine et al. [2020] Stevan Weine, Brandon A Kohrt, Pamela Y Collins, Janice Cooper, Roberto Lewis-Fernandez, Samuel Okpaku, and Milton L Wainberg. 2020.  Justice for George Floyd and a reckoning for global mental health.  _Global Mental Health_ 7 (2020).
  * Yang [2016] Guobin Yang. 2016\.  Narrative agency in hashtag activism: The case of# BlackLivesMatter.  _Media and communication_ 4, 4 (2016), 13\.
  * Young et al. [2022] Dannagal G Young, Huma Rasheed, Amy Bleakley, and Jessica B Langbaum. 2022.  The politics of mask-wearing: Political preferences, reactance, and conflict aversion during COVID.  _Social Science & Medicine_ (2022), 114836.
  * Zhang et al. [2020] Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020.  Semantics-aware BERT for language understanding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 9628–9635.

##  Appendix A Data Collection Keywords

  * •

Election Data Keywords: election, presidential election, election candidate,
#2020election, #vote2020, #election2020, #elections2020, #debates, #debate,
#debatenight, presidentialdebate, joe, biden, joebiden, joe biden, bernie,
sanders, berniesanders, bernie sanders, warren, elizabeth, elizabethwarren,
#votebluenomatterwho, #votebluetosaveamerica, #votethemallout2020,
#votebluenomatterwho2020, #nevertrump, #democrats, donald, trump, donaldtrump,
donald trump, #trump2020, #maga, #tcot, #p2, #republicanparty, #gop, kamala,
harris, kamalaharris, kamala harris

  * •

COVID-19 Data keywords: coronavirus, covid, chinese virus, wuhan, ncov, sars-
cov-2, koronavirus, corona, cdc, N95, kungflu, epidemic, outbreak, sinophobia,
china, pandemic, covid

  * •

Racial Data Equality keywords: george floyd, race, #blacklivesmatter, racial,
racist, racism, nigga, nigger, black america, white american, black people,
white people, black lives matter, black man, white man, black men, white men,
black woman, white woman, black women, white women, #georgefloyd, #blm,
blackman, #whiteman, #blackmen, #whitemen, #blackwoman, #whitewoman,
#blackwomen, #whitewomen, #blackamerican, #whiteamerican, #blackpeople,
#whitepeople

##  Appendix B Hashtag Based Stance Detector Algorithm

The algorithm 1 describes running hashtag-based stance detector on tweet $t$
with Support hashtags $S$ and Against hashtags $A$.

Input: $t,S,A$

Output: $stance$

Function  _detectStance(_t, S, A_)_:

$score\longleftarrow 0$;

foreach  _hashtag in S_ do

if  _hashtag exist in t_ then

$score\longleftarrow score+1$

end if

end foreach

foreach  _hashtag in A_ do

if  _hashtag exist in t_ then

$score\longleftarrow score-1$

end if

if  _score ¿ 0_ then

return Support ;

else if  _score = 0_ then

return Non-Opinionated ;

else

return Against ;

end if

end foreach

End Function

Algorithm 1 Hashtag Based Stance Detector