# Into the crossfire: evaluating the use of a language model to crowdsource
gun violence reports

Adriano Belisario
Oxford Internet Institute
University of Oxford
Oxford, UK
&Scott Hale∗
Oxford Internet Institute
University of Oxford
Oxford, UK &Luc Rocher∗
Oxford Internet Institute
University of Oxford
Oxford, UK

###### Abstract

Gun violence is a pressing and growing human rights issue that affects nearly
every dimension of the social fabric, from healthcare and education to
psychology and the economy. Reliable data on firearm events is paramount to
developing more effective public policy and emergency responses. However, the
lack of comprehensive databases and the risks of in-person surveys prevent
human rights organizations from collecting needed data in most countries.
Here, we partner with a Brazilian human rights organization to conduct a
systematic evaluation of language models to assist with monitoring real-world
firearm events from social media data. We propose a fine-tuned BERT-based
model trained on Twitter (now X) texts to distinguish gun violence reports
from ordinary Portuguese texts. Our model achieves a high AUC score of 0.97.
We then incorporate our model into a web application and test it in a live
intervention. We study and interview Brazilian analysts who continuously fact-
check social media texts to identify new gun violence events. Qualitative
assessments show that our solution helped all analysts use their time more
efficiently and expanded their search capacities. Quantitative assessments
show that the use of our model was associated with more analysts’ interactions
with online users reporting gun violence. Taken together, our findings suggest
that modern Natural Language Processing techniques can help support the work
of human rights organizations.

**footnotetext: These authors contributed equally to this work and share
senior authorship.

##  1 Introduction

Numbers can hardly convey the tragic losses caused by gun violence. Yet they
provide valuable insights to understand and address pressing violations of
human rights. According to a comprehensive international study from 2022 [25],
cases of physical violence by firearm have increased over the past three
decades. These cases are heterogeneously distributed across countries, with
nations such as Brazil and the USA accounting for a considerable share of the
global burden of firearm violence [24, 25]. Beyond losses of life, this burden
impacts nearly every dimension of the social fabric, such as healthcare,
education, psychology, and economy [25].

Reliable data on firearm events is paramount to developing more effective
public policy and emergency responses, but documenting these cases is
challenging. As comprehensive databases of human rights violations are rare
[28], organizations tackle this issue through substantial manual work. To
supersede risky and costly physical studies counting firearm events,
humanitarian organizations are trialling the use of social media data to
better discover and document gun violence events. They typically use keyword-
based search to crowdsource online evidence, which results in very large
datasets with a high proportion of unrelated texts. Reviewing and filtering
these datasets to find citizen reports of firearm violence can be akin to
searching for a needle in a haystack.

To handle this problem, Natural Language Processing (NLP) methods have emerged
as a promising solution to automate text classification, and researchers
developed models to automate tasks stemming from social media data in
quantitative human rights studies. For instance, classification models have
been successfully developed to assist human rights investigations based on
social media data in English [27], Spanish [5], in the Arab world using
Twitter data [2], and on the Russian-Ukraine war using Telegram messages [23].
These studies suggest that NLP methods can achieve promising results for
quantitative human rights classification tasks when evaluated on benchmark
datasets. However, little is known about the effectiveness of NLP methods on
live and dynamic social media data in real-world monitoring applications.

Here, we publish the first systematic real-world evaluation of a language
model to assist human rights analysts in crowdsourcing gun violence reports
from social media. We partnered with Fogo Cruzado Institute111Instituto Fogo
Cruzado stands for “Crossfire Institute”: https://fogocruzado.org.br, a human
rights organization that monitors gun violence in Brazil, and produces real-
time alerts for local citizens. Its team of analysts continuously monitor
online data, including Twitter posts, to identify gun violence events in four
Brazilian regions. Analysts’ work on Twitter has two stages: first, they
monitor certain keywords and profiles. If a potential gun violence report is
found, they interact publicly with Twitter users to collect further
information and fact-check the event. We aim to automate and augment the first
stage with NLP methods.

Our research sought to answer the following research questions:

  * •

RQ1 - Can Transformer-based language models accurately identify gun violence
reports in Brazilian Portuguese social media texts?

  * •

RQ2 - What are the advantages and challenges of adopting a language model for
real-time monitoring compared to manually reviewing social media texts?

We first develop a Bidirectional Encoder Representations from Transformers
(BERT) [9] model to classify whether social media texts contain gun violence
reports. We build upon BERTimbau [33], an open-source BERT model pre-trained
on Brazilian Portuguese text. To classify gun violence events, we fine-tune
this model using Twitter texts with semi-supervised learning techniques. This
leads to a model that can accurately classify whether a text message contains
a gun violence report (positive cases) or not (negative) with 93% accuracy on
our human-reviewed holdout evaluation set (with a random baseline at 51%) and
an AUC score of 0.97.

We then develop a web application that uses our BERT model to help human
rights analysts identify gun violence reports from Twitter data in real-time.
Using an intervention design, we introduced Fogo Cruzado’s analysts in Rio de
Janeiro to our web application, and they used it between May 28th and July 2nd
2023. Qualitative assessments show that our solution helped all analysts use
their time more efficiently and expanded their search capacities. Quantitative
assessments show that the use of our model was associated with more analysts’
interactions with online users reporting gun violence. Taken together, our
findings suggest that language models can significantly augment the ability of
human rights analysts to monitor reports from social media.

##  2 Related work

To apply NLP methods to quantitative human rights studies on gun violence,
researchers have developed tailored benchmark datasets and methods. Examples
of datasets include the Gun Violence Database in the USA context [26] and the
Arabic Violence in Twitter dataset [3]. Common methods include named entity
recognition [20, 6] and classification [10, 2, 23]. Archives have long been
used as data sources in quantitative human rights studies, and recently,
online social media has gained popularity. Archive documents include survivor
testimonies [20, 10] or newspaper articles [26, 6, 30] and report past events.
Social media has emerged as a “live source" that can help researchers
understand ongoing conflicts [17], but obtaining and processing large volumes
of online data is challenging.

Modern methods such as Transformer models are increasingly being proposed to
classify text from social media data in quantitative human rights studies.
Transformer-based models [34] have proven particularly effective in processing
and analyzing text-based data, notably for text classification and entity
recognition. Researchers developed BERT-based models to classify English texts
about political violence and conflict [16], as well as human rights violations
[27]. In Spanish, the detection of violent incidents from social media was
proposed as a challenge at a recent NLP conference; all participants used
variations of pre-trained Transformer models, including a wide variety of BERT
implementations [5]. In Russian, multiple BERT-based models have been used to
classify Telegram messages and detect human rights violations during the
Russian-Ukraine war [23]. To the best of our knowledge, there has been no
previous work on human rights abuse detection using social media texts in
Portuguese.

Few studies have evaluated the adoption of NLP models by human rights
organizations. Alhelbawy et al. built a corpus of tweets reporting violent
acts in Arabic [3] and compared different NLP architectures, including
baseline models (Naive Bayes and Support Vector Machine) and two types of long
short-term memory (LSTM) neural networks. The LSTM architecture achieves the
highest performance, while the authors note that Transformer-based
architectures could offer higher scores. The authors concisely comment on the
impact of its real-world implementation, stating that “collected reports
contributed to a number of publications by human rights organizations".

##  3 Background and context

Rio de Janeiro is systematically affected by firearm violence [32, 19] but
lacks granular data about these conflicts. Recent estimates calculate that
approximately 20% of Rio’s metropolitan region is dominated by armed groups
[13]. As these groups compete for territorial control, firearm events and
armed clashes with the police intensify [7]. Nevertheless, few official
sources have information about firearm events in Rio de Janeiro. The Mortality
Information System222https://opendatasus.saude.gov.br/dataset/sim offers
national microdata about deaths, but not all gun violence implies deaths.
Records from police stations, which could provide detailed descriptions of
events of armed violence, are not publicly available. The local government
offers only aggregated public security metrics per month and administrative
units. This lack of microdata about gun violence events is critical and
hinders academic research and society’s capacity to produce evidence that
informs public policy.

Fogo Cruzado Institute is a non-profit organization founded in 2016 out of the
need for detailed data sources about shootings in Rio de Janeiro. The
initiative was initially incubated by Amnesty International Brazil. It now has
nearly twenty professionals distributed across four metropolitan areas. Fogo
Cruzado plays a significant role in advocacy, academic research and public
awareness about gun violence. Internationally, it is an important data source
for ACLED, the Armed Conflict Location and Event Data Project [29], which
gathers disaggregated data on political violence and protest events worldwide
and has been used by academic research. In Brazil, Fogo Cruzado co-authored
several academic reports on Rio de Janeiro’s public security, including an
analysis of police raids with high rates of lethality [15], robberies [14] and
an extensive mapping the territorial control of armed groups in Rio de Janeiro
[13].

We conducted five video interviews with Fogo Cruzado’s staff between May and
July 2023; this section builds upon the information gathered from these
meetings. Fogo Cruzado employs a dedicated team of analysts to monitor events
in each of the four regions monitored. The core team for monitoring gun
violence events in Rio de Janeiro has four analysts, a team leader, and an
additional position that rotates among other Brazilian states as needed. They
work in shifts to record information related to gun violence events, such as
the number of dead or injured civilians and police forces and whether there
was an ongoing police operation. The team monitors new cases seven days a
week, except in the period between midnight and 6 am. The analyst working
early in the morning is in charge of retrospectively catching up and recording
the cases reported during this night period. They receive reports submitted by
citizens using Fogo Cruzado’s mobile application and actively search for
reports on social media platforms. Analysts follow local WhatsApp groups,
Facebook pages and groups, Twitter posts, and local press websites. After
identifying a report, the team checks the event using cross-verification with
other online and on-the-ground sources.

Fogo Cruzado has systematically used Twitter to track and interact with gun
violence reports since 2018. This social network is an “essential platform for
research", according to Fogo Cruzado’s Systematisation Protocol333Fogo
Cruzado’s internal document kindly shared for this research.. Fogo Cruzado
uses a different Twitter profile to track and interact with users reporting
gun violence events in each region. We requested the team in Rio de Janeiro to
provide a detailed account of the sources for identifying new cases. They
recorded the primary source of 150 events over the period from 2nd Feb 2023 to
15th Mar 2023. Twitter was used in 68% of these events recorded and fact-
checked by the team. WhatsApp was the second most important source, accounting
for 35% of cases. Fogo Cruzado’s mobile application contributed 5% of cases,
while Facebook and personal contacts each played a minor role, accounting for
less than 1% individually. Percentages do not add up to 100% since the same
event can have multiple sources. Although limited, these results confirm the
relevance of Twitter as a data source for identifying gun violence reports in
real-time.

Some of the keywords monitored on Twitter (“(bala voando) OR tiro OR tiroteio
OR baleado"444Verbatim translation: “(bullet flying) OR shot OR shooting OR
[person] shot”) are strongly associated with shooting events, but others are
common terms in Portuguese. While wordings such as “bala voando" (“bullet
flying") are literal representations of firearm events, the word “tiro" not
only means “shot" but is also the first person present tense for the verb
“tirar" (“to take"). Consequently, it is primarily used in various contexts
unrelated to gun violence reports. Furthermore, there are also common
idiomatic expressions in Brazilian Portuguese for words like “tiroteio"
(“shootout") that are unrelated to actual firearm shootings, as in “mais
perdido que cego em tiroteio" (literally, “more lost than a blind person in a
shootout").

Therefore, manually reviewing social media data imposes important limitations.
Analysts use Tweetdeck to browse messages with keywords associated with gun
violence. The analysts search for keywords associated with gun violence on
Tweetdeck555https://tweetdeck.twitter.com/ and filter the results by location,
considerably narrowing down the search scope. Twitter
documentation666https://developer.twitter.com/en/docs/tutorials/advanced-
filtering-for-geo-data estimates that only 1-2% of tweets messages are geo-
tagged, but 30-40% of them contain profile location information. During our
intervention, Tweetdeck geographic search used both fields to filter messages
from a specific location. Still, filtering messages to get only those with
user or post-level location metadata ignores the majority of the data
available on Twitter, and searching without the geographical filter would lead
to an overload of unrelated information.

Once a citizen report of gun violence is identified, the analysts interact
with the user, sending a semi-standardized message to request further details
about the time and location of the event. Below, we define this type of reply
as an interaction from Fogo Cruzado with the users.

Importantly, not all reports receive an interaction from Fogo Cruzado’s team.
Reports of gun violence using the keywords monitored and with location
metadata associated might not receive an interaction due to several reasons.
For example, the analyst may not see a particular message, or the event may
have already been registered. For security reasons, analysts also refrain from
engaging with users who display any indications of association with criminal
activities.

##  4 Methodology

To evaluate a Transformer-based model’s ability to detect gun violence reports
in Portuguese on Twitter, we developed a prototype for real-time tweet
classification and tested it with Fogo Cruzado analysts in Rio de Janeiro.
Next, we describe our mixed method approach, integrating quantitative and
qualitative analysis, to gain a thorough understanding of the challenges and
impacts of AI models in the context of human rights monitoring.

###  4.1 Data description

We collected data using the full-archive search endpoint from the Twitter
Academic API v2 between December 2022 and August 2023.

We treat as labelled examples of the positive class all posts that received a
reply from Fogo Cruzado’s profile in Rio de Janeiro requesting information
about gun violence events. To create this labelled dataset, referred to as
Dataset $L$, We collected all interactions of the user “@fogocruzadorj". We
then collected all the posts to which these messages were replied. The Dataset
$L$ includes 36,241 messages posted between 9 June 2016 and 24 May 2023.

To represent the negative class, we collected posts with the keywords
monitored by Fogo Cruzado using distinct search parameters, resulting in two
datasets. These unlabelled datasets represent posts with geographical and
language metadata and are named $U_{geo}$ and $U_{lang}$, respectively.
Assuming that gun violence is more common in Rio de Janeiro than in other
Portuguese-speaking regions, posts with geographical metadata associated with
Rio de Janeiro ($U_{geo}$) are more likely to have a higher proportion of
reports of gun violence compared to all posts in Portuguese containing the
keywords monitored ($U_{lang}$).

For Dataset $U_{geo}$, we used keywords and geographic filters similar to
those used by Fogo Cruzado’s team to monitor gun violence in Rio de Janeiro.
This dataset emulates the Tweetdeck feed monitored by Fogo Cruzado and
contains 319,114 messages posted between 31st October 2016 and 7th May 2023.
Most of these messages are examples of the negative class (i.e., do not
represent a report of gun violence event). However, as explained in Section 3,
there are also reports that have not received an interaction and, hence, are
not labelled.

For Dataset $U_{lang}$, we used keywords and a language filter (“lang: pt") to
get all tweets in Portuguese with the keywords monitored regardless of the
user’s location; therefore, it is broader than $U_{geo}$. It contains
12,803,338 messages posted between 5 December 2020 and 7 May 2023.

We filtered these datasets with the following preprocessing rules: all tweets
with mentions to the partner organization username were removed, mentions and
links were replaced with special tokens, extra spaces were trimmed, and we
deleted all tweet replies and duplicated messages, ignoring unavailable
tweets. We also converted emojis to text descriptions.

Table 1 shows examples of both classes in our training dataset. Most of the
messages in the training dataset are relatively short. The average number of
words in the training dataset is 16, and the maximum token length in the
training data is 160, with virtually all messages not exceeding the 100-token
mark. This characteristic informed the definition of the standard value of
token length for the Transformer model, as described in Section 4.2.

Table 1: Examples of the positive and negative classes (authors’ translation to English). Positive Examples |  Negative Examples
---|---
Gunshots started going off right when I ordered a milkshake. I hope this man is a brave warrior. |  Sometimes, certain words are like a shot, especially when you’re feeling a little insecure
People here randomly fire off shots out of nowhere. |  Oh so many distracted friends’ photos that I took Jesus I deserved to get shot lol
I already wake up startled, hearing gunshots. |  I’m trying to let my nails grow, but when anxiety attacks, I tear them all off.

###  4.2 Model development

We implemented a dummy classifier, a Naive-Bayes, and a Transformer model. The
dummy classifier is a random baseline that ignores text and generates random
predictions, selecting one of the two classes with equal probabilities. We
used the Naive-Bayes model with a TF-IDF vectoriser as a secondary baseline.

To implement the Transformer model, we leveraged
BERTimbau777https://huggingface.co/neuralmind/bert-large-portuguese-cased, a
pre-trained BERT model with Portuguese text [33]. BERTimbau was trained using
brWaC, a corpus with 3.5 million web pages in Brazilian Portuguese, processed
through a model with 24 layers, 1024 hidden dimensions, 16 attention heads and
330 million parameters [35, 33]. We fine-tuned the model for binary
classification. This process involves incorporating an additional output layer
into the neural network, which represents the probabilities associated with
the positive and negative classes. We employed the Adam optimizer with a
learning rate set to $2.10^{-5}$. To prevent overfitting, we applied a dropout
probability of 5% to the hidden and attention layers and implemented early
stopping to terminate the training process. For computational efficiency, we
reduced BERTimbau’s maximum token length value (from 512 to 128) after
analyzing the training data.

We adopted self-training to address the challenge of making inferences from
partially labelled data. Self-training is a long-standing semi-supervised
learning method [4]. It initially trains the model with a reduced number of
labelled data and then uses it to generate pseudo-labels for unlabelled data.
These pseudo-labels are then combined with the original labelled data, and the
augmented dataset is used to retrain the model to enhance its performance, a
process that can be repeated one or more times in a loop.

We used data prior to 2023 to train the model; messages posted in that year
were reserved for holdout datasets. This criterion yielded 24,353 messages in
Dataset L that were used as labelled examples of the positive class to train
the model. We defined the negative class filtering Dataset $U\\_lang$ by
messages posted before 2023 without location metadata. Then, we deleted those
posts overlapping the Dataset L and down-sampled the data, randomly selecting
a volume of messages three times the number of positive samples (73,059) to
avoid an extreme class imbalance. Therefore, the initial training dataset
combining both classes had 97,412 messages.

After fine-tuning the Transformer model, we used it to generate pseudo-labels
for the Dataset $U_{geo}$, after removing all messages posted in 2023 or used
in the initial training. This approach resulted in 199,015 labelled examples
used to augment the training dataset. To validate the pseudolabels generated,
we categorized the predictions into four quantiles according to the
probability assigned to each positive class. So, we selected ten samples from
each class and quantile, resulting in a validation sample of 80 messages.
Finally, we augmented the original training dataset to include the pseudo
labels and fine-tuned the BERT model again from scratch.

We assessed the performance of the BERT and baseline models using two main
metrics. To compute these, we created two holdout datasets applying distinct
filters to the data described in Section 4.1.

The first metric aims to measure the model performance to predict Twitter
interactions done by Fogo Cruzado’s analysts. For this purpose, we created the
holdout Dataset $H_{interactions}$, which contains 6,724 messages. The
positive class has 1,909 messages resulting from filtering Dataset L for posts
dated 1st January 2023 or later. The negative class was represented by all
4,815 unlabelled messages from Dataset $U_{geo}$. We deleted messages posted
before 2023 and those already in the Dataset $L$.

Notably, some of the messages assigned to the negative class in Dataset
$H_{interactions}$ are actually gun violence reports that have not received an
interaction, potentially increasing False Positive error rates. Yet, we are
interested specifically in the recall of this dataset, which measures the
model’s sensitivity to identify positive instances (messages that were
reviewed by Fogo Cruzado’s analysts and received an interaction). It is worth
noting that, in our setting, having False Negatives is more harmful than
having False Positives. False Negative cases imply potential oversight of a
gun violence report due to model misclassification. In contrast, False
Positives only introduce noise to the messages presented for human review.

Secondly, we aimed to estimate the overall ability to distinguish between gun
violence reports and ordinary texts. For this purpose, we built the holdout
Dataset $H_{reports}$. This is a subset of the previous holdout Dataset
($H_{interactions}$) with 1,211 messages posted in April 2023. These messages
have all been coded by the leading researcher of our team, a Brazilian
Portuguese native speaker who reviewed all records to ensure that these
entries were accurately classified. We coded the labels of 147 (12%) messages
containing gun violence reports that were labelled as negative cases because
they had not received an interaction from Fogo Cruzado. For this Dataset
$H_{reports}$, a key metric to estimate the overall performance is the
F1-score, as it considers both False Positives and False Negatives.

Appendix A summarizes the data collection and transformation process to train
and test the model. Green and red arrows represent, respectively, the positive
and negative classes.

In addition to recall and F1-score, we also report accuracy, precision, the
confusion matrix and the Receiver Operating Characteristic (ROC) curve. All
scores range from 0 to 1. A value of 1 indicates that the model reached the
best performance for a given metric and dataset.

###  4.3 Designing a real-world intervention

We designed an intervention with the Fogo Cruzado team to investigate the
model performance in a real-world setting. We created a prototype, and Fogo
Cruzado has adopted it as part of the standard workflow for live monitoring of
new reports of gun violence in Rio de Janeiro.

The prototype used the Twitter API v1 (search endpoint) to retrieve the latest
tweets and a server for preprocessing and classifying them using the best-
performing model. We developed a Python script to execute this pipeline and
upload the results to a web platform. It was initially configured to run every
15 minutes; later, we reduced this interval to five minutes due to
participants’ requests.

Fogo Cruzado’s team in Rio de Janeiro adopted our AI-powered prototype to
browse tweets from 29 May to 2 July 2023. Five professionals engaged directly
with this intervention.

Our prototype presents live Twitter data aggregated by different tabs.
Messages classified as potential reports in Rio de Janeiro and those
classified as negative cases are aggregated in the first two tabs. The third
tab displays positive cases of messages without location metadata or users’
location descriptions not matched to Rio de Janeiro. The information is
displayed in tables: each row represents a message, and the columns are the
text, timestamp, user location and profile bio. The interface also allowed
analysts to click on a button to view the original post on Twitter or respond
to the user with a standardized follow-up message.

We collected Fogo Cruzado’s interaction on Twitter during the intervention and
gathered information from the participants using surveys and interviews. We
conducted all interviews and survey applications in May and July 2023.

Participants were asked to complete an online survey with closed-ended
questions (Appendix B) before and after the intervention. The first survey
gathered information to aid the interpretation of the training data and the
standard workflow for online report monitoring. The post-intervention survey
evaluated the impact of the model’s adoption for real-time monitoring and
gauged the subjective assessments of the participants.

We conducted five interviews. We interviewed the analyst team leader in Rio de
Janeiro before the intervention. The team leader and three analysts were also
interviewed afterwards, approximately a month after the start of our
intervention. The interviews followed a semi-structured approach, and we did
thematic and content analysis [31] of the answers, aiming to identify
recurring patterns of topics related to the RQ2.

##  5 Results

Next, we compare the performance of the models with both holdout datasets and
present the results of our intervention with the partner organization.

###  5.1 Model performance

Both holdout datasets presented similar trends and confirmed that the
Transformer-based model can effectively identify gun violence reports in
Portuguese (RQ1). As expected, the Transformer model outperformed the baseline
models in both holdout datasets. Our BERTimbau implementation consistently had
over 80% of success with the holdout datasets, a performance better than or on
par with previous research in the field, described in Section 2.

The self-training approach has proven useful. Retraining the model with
pseudo-labels improved BERTimbau’s performance by two percentage points in
recall for the positive class and a one percentage point improvement in
F1-score for $H_{reports}$ compared to training the model without pseudo-
labels. Results for BERTimbau below refer to the model trained with the
augmented dataset.

Our fine-tuned version of BERTimbau correctly predicted nine out of ten Fogo
Cruzado’s interactions in the period analyzed. Table 2 presents the metrics
for the test Dataset $H_{interactions}$. The model reaches 91% in recall of
the positive class.

Table 2: Classification metrics for the holdout dataset $H_{interactions}$. Model | Accuracy | Precision | Recall (pos) | Macro F1-score
---|---|---|---|---
Random baseline | 0.50 | 0.28 | 0.49 | 0.47
Naive-Bayes | 0.85 | 0.70 | 0.82 | 0.82
BERTimbau | 0.87 | 0.89 | 0.91 | 0.85

Figure 1 shows that BERTimbau achieved an AUC score of 0.90 in Dataset
$H_{interactions}$. The dashed line represents the random classifier, with an
AUC of 0.5. Most of the incorrect predictions are False Positives. This result
is expected because there are reports of gun violence that have not received
an interaction from Fogo Cruzado’s team.

Figure 1: Confusion matrix and ROC curve for BERTimbau using
$H_{interactions}$.

Table 3 shows the overall performance of the model to distinguish ordinary
Portuguese text from gun violence reports. BERTImbau reaches an F1-score of
92% in Dataset $H_{reports}$, our human-reviewed holdout dataset.

Table 3: Classification metrics for the holdout dataset $H_{reports}$. Model | Accuracy | Precision | Recall (pos) | Macro F1-score
---|---|---|---|---
Random baseline | 0.51 | 0.41 | 0.52 | 0.51
Naive-Bayes | 0.89 | 0.92 | 0.78 | 0.89
BERTimbau | 0.93 | 0.92 | 0.87 | 0.92

Figure 2 shows the confusion matrix and the ROC curve for $H_{reports}$. The
BERTimbau model inaccurately classified 89 out of 1,211 messages. The model
misclassified a relatively low proportion of gun violence reports, sixty-one
messages representing 5% of all records in this holdout dataset.

Figure 2: Confusion matrix and ROC curve for BERTimbau using $H_{reports}$.

Our error analysis using $H_{reports}$ shows that the best-performing model
struggles with long text and emojis. We found two striking patterns after
reviewing the 89 misclassified messages by BERTimbau in the holdout Dataset
$H_{reports}$. First, there is a considerable proportion of long text: 46% of
misclassified messages have more characters than the average of the training
dataset (80 characters). Secondly, emojis are present in 23% of the messages.

###  5.2 Intervention findings

We implemented a prototype using BERTimbau for the intervention. The solution
was presented as an optional tool for social media listening to Fogo Cruzado’s
analysts in Rio de Janeiro. From 28th May to 2nd July 2023, the prototype
gathered 21,871 messages from Twitter, classified them, and uploaded them to a
web interface. Fogo Cruzado’s team has used the prototype even after the end
of the intervention.

Four participants answered the surveys. All considered that the prototype had
increased their capacity to identify more Twitter reports. Appendix B presents
a summary of participants’ responses.

The interviews and surveys confirmed that our model can accurately identify
reports of gun violence in Brazilian Portuguese and is helpful for real-time
monitoring. The model implementation allowed human rights analysts to track
more messages without being burdened by unrelated information. Overall, they
consider that the advantages in terms of efficiency outweigh the problems
encountered, and the prototype has improved their efficiency and helped them
save time.

Fogo Cruzado’s analysts have highlighted the classifier performance to
accurately distinguish between reports of gun violence and ordinary messages
posted on Twitter. All participants expressed an initial discomfort with the
new workflow and interface for monitoring tweets. However, this adjustment
period was swiftly followed by an understanding of the prototype’s advantages.
The survey reveals that by the end of the intervention, all have adopted it as
part of their daily workflow.888Appendix B. The participants evaluated the
model performance to accurately classify reports of gun violence from other
unrelated messages as “good" and “excellent".

The primary advantages reported in the interviews and indicated in the survey
are outlined as follows:

  * •

Our prototype filters out less relevant social media content and provides a
higher signal-to-noise input for analysts. This enables them to improve the
monitoring and tracking of human rights events. Reviewing messages on Twitter
and searching for potential reports of gun violence became more efficient with
the prototype. Analysts claimed that the interface increases their agility and
helps to streamline their workflow, enabling them to allocate their time more
effectively. This advantage can be illustrated by the following quote from an
interview with one of the participants: “[Now] I do not have to go hunting for
tweets. Sometimes, I missed them [gun violence reports] because there were too
many [unrelated] messages. During the BBB [Big Brother Brasil, an annual TV
show extremely popular on Twitter], it was chaotic […]. It was literally a
treasure hunt"999Authors’ translation to English..

  * •

Instead of using restrictive geolocation filters that limit the number of
messages when collecting data, our prototype allows analysts to expand their
search scope. The platform enabled users to search for more terms or review
messages that were classified by the model as gun violence reports but do not
contain location metadata, thus enabling them to scrutinize more reports. The
prototype allowed analysts to analyze reports without being overwhelmed by the
volume of messages. One of the participants deemed “very important" the tab
showing all tweets with reports of gun violence, regardless of having location
metadata associated to Rio de Janeiro. This participant noted that some of
these tweets have user location information related to Rio de Janeiro that are
not correctly associated to this region by the Twitter search: “People write
[location metadata] the way they want. Today I discovered a shooting because
of this tab. They use details and slangs [that are not filtered by the
standard Twitter geolocation search]".

  * •

Reviewing messages pre-classified by the language model reduces the cognitive
effort required to find relevant information. The table view used in our
prototype was deemed more straight to the point and better organized when
compared to the column-based interface of Tweetdeck. According to the
analysts, the spreadsheet eliminates the need for constant scrolling and
reduces their cognitive effort. One of the participants reported that
reviewing messages on Tweetdeck “gives extra mental fatigue from processing
much information at the same time." The prototype was deemed particularly
useful when the analysts needed to retrospectively review a high volume of
messages. The analyst in the morning shift, who is responsible for reviewing
posts between midnight and 6 am, describes this advantage as follows: “Once I
got the hang of it, it has been a thousand times better to scan [reports] on
the morning shift [because before] I had to scroll down all Tweetdeck
columns."

Conversely, the interviews and surveys also revealed limitations. The use of
our prototype to gather information on events already identified, as opposed
to discovering reports on new events or live track conflicts in real time, was
limited. This happened mostly because of the following drawbacks, according to
the participants:

  * •

Updating new tweets promptly is critical for monitoring ongoing conflicts: The
fact that the prototype only updated the messages in the web interface in
five-minute intervals limited the analysts’ abilities to quickly monitor
events as they happen. They also pointed out that a timestamp indicating the
last successful execution of the pipeline script should be included.

  * •

Search keywords need to be dynamically set: Custom searches are part of
analysts’ standard workflow to monitor events in Tweetdeck. They add terms
related to specific locations to verify and fact-check reports. However, the
prototype did not allow users to change the keywords used to fetch data from
Twitter.

  * •

Further information beyond texts can help analysts interact with users:
Analysts highlighted the absence of user photos as a drawback. According to
them, user photos are important to determine whether there will be a follow-up
interaction requesting further information about the event. Using the
prototype to monitor reports required an extra click to check the user profile
and picture.

Overall, the participants have adopted the platform due to its advantages in
terms of broader search capabilities, time optimization, and interface
improvements. Participants reported that sometimes the model misclassified
long texts or posts reproducing music lyrics. One of the participants also
noted that posts with lyrics might be challenging even for humans, as
classifying them requires contextual knowledge. The interviews revealed that
the participants chose to use both the prototype and Tweetdeck concurrently
instead of replacing one platform with the other. While our prototype was
preferred to discover new cases, Tweetdeck was used mainly to observe events
as they unfold and follow specific Twitter profiles.

The mean number of interactions from Fogo Cruzado’s profile in Rio de Janeiro
(“@FogoCruzadoRJ") with users reporting gun violence before the intervention
was 17 and increased to 24 after adopting the prototype. To test if the AI-
powered search significantly increases the total number of human rights
analysts’ daily interactions with reports per day, we use a difference-in-
difference regression. We defined Rio de Janeiro as a treatment group and Fogo
Cruzado’s team in Bahia as a control group. The diff-in-diff model shows that
our intervention is associated with an increase in nine interactions with
reports of gun violence per day. Appendix C presents the regression models,
confounding variables, and the difference-in-difference analysis.

##  6 Discussion

Our work combines quantitative and qualitative methods to provide the first
systematic assessment of adopting a language model utilizing social media data
to assist human rights monitoring in real-world settings. Our evaluations of
the models’ performance confirm that Transformer models are suitable for
classification tasks in quantitative human rights research and applications.
Next, we interpret the models’ results with the holdout datasets and the
intervention and discuss limitations and directions for further research in
this field.

The performance improvement of Naive-Bayes to BERTimbau is smaller than the
random classifier to Naive-Bayes. This suggests that it is reasonably possible
to distinguish classes by analyzing only vocabulary count without accounting
for the relationship between words. The feature independence assumption of the
Naive Bayes models postulates that words in a given message are independent of
each other, which is certainly not the case. Even so, the Naive Bayes model
could identify correctly over 70% of the reports. On the other hand, while the
F1-score improvement from the Naive-Bayes model to BERTimbau might seem small
(0.89 to 0.92), the increase in performance is substantial for recall scores
(0.78 to 0.87). As explained in Section 4, having high recall scores is
critical for our application.

Our intervention shows that language models can help human rights analysts
filter social media posts in order to find evidence of human rights
violations. It is worth highlighting that participating in our research and
using the prototype were presented as optional alternatives, by no means
mandatory or imposed by supervisors. With an already high number of sources to
monitor, the analysts could simply choose not to use the prototype if they had
not considered it useful. Nevertheless, the team has adopted it as one of
their standard tools and continued to utilize the prototype after the end of
the intervention, expanding its application to monitor regions other than Rio
de Janeiro.

Our research and findings align with contemporary concerns and methods in
computational social science. Describing new data and methods for quantitative
human rights, Murdie and Watson cite the use of automation, experimental
methods, and partnerships between academic researchers and organizations or
governments [21]. Likewise, among other recommendations for an emerging
computational social science, Lazer et al. suggest that academic researchers
should prioritize real-world problems and strengthen collaboration with non-
academic actors but fall short when listing only industry and governments as
potential partners [18]. Non-governmental organizations (NGOs) have developed
the most cutting-edge advances in human rights measurement [22, 11] and are
crucial stakeholders in quantitative human rights research. Having
successfully combined automation, experimental methods and collaboration with
citizen organizations, we encourage academic research projects to cooperate
with NGOs and on-the-ground initiatives in order to improve society’s capacity
to track, report and analyze human rights violations.

While our results show that algorithmic approaches can scale data collection
and increase accuracy in human rights monitoring, they can affect analysts’
capacities to document events in numerous ways. In the best scenario, language
models and AI assistants could help mitigate the underreporting of cases.
Alternatively, they could also lead to over- or under-representation of
certain groups based on linguistic features. They could also lead to the
identification of more reports of the same events already observed without
automation, preserving and reinforcing blind spots in terms of data
collection. It is unclear if and how adopting automated systems for social
media monitoring contributes to the statistical biases and the “information
effect"[12, 8], in which upward trends in event counts occur because human
rights monitors became able to observe more cases.

##  7 Conclusion

As organizations incorporate machine learning models and other data-driven
techniques to analyze human rights violations, addressing potential biases and
fairness concerns is crucial. First, unequal access to resources in terms of
knowledge and financial means can create disparities in the capacity to
leverage machine learning and open-source information, potentially widening
the gap between well-funded and under-resourced organizations. The financial
and human resources needed to train open-source language models and keep them
functional are bottlenecks for human rights organizations. Commercial language
models might be a cheaper alternative, but they often lack transparency,
induce users’ dependency on a third-party service and are general-purpose
solutions, as opposed to models tailored for human rights investigations.
Second, biases related to collecting user-generated content may introduce or
reinforce biases in data, as information from specific demographics or regions
may be overrepresented or underrepresented, potentially leading to inaccurate
conclusions. Finally, the unequal application of machine learning across
different areas of human rights may result in some violations being better
addressed than others, depending on data availability or the suitability of
this technique. Encouraging collaboration between different actors, developing
guidelines and benchmarks, as well as promoting capacity-building initiatives
can help ensure that these modern techniques are employed in a manner that is
both equitable and effective.

## References

  * [1]
  * Alhelbawy et al. [2020] Ayman Alhelbawy, Mark Lattimer, Udo Kruschwitz, Chris Fox, and Massimo Poesio. 2020.  An NLP-Powered Human Rights Monitoring Platform.  _Expert Systems with Applications_ 153 (2020).  https://doi.org/10.1016/j.eswa.2020.113365
  * Alhelbawy et al. [2016] Ayman Alhelbawy, Poesio Massimo, and Udo Kruschwitz. 2016\.  Towards a Corpus of Violence Acts in Arabic Social Media. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_. European Language Resources Association, 1627–1631.  https://aclanthology.org/L16-1257
  * Amini et al. [2022] Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Emilie Devijver, and Yury Maximov. 2022\.  Self-Training: A Survey.  https://doi.org/10.48550/arXiv.2202.12040
  * Arellano et al. [2022] Luis Joaquín Arellano, Hugo Jair Escalante, Luis Villaseñor Pineda, Manuel Montes y Gómez, and Fernando Sanchez-Vega. 2022.  Overview of DA-VINCIS at IberLEF 2022: Detection of Aggressive and Violent Incidents from Social Media in Spanish.  _Procesamiento del Lenguaje Natural_ 69 (2022).  https://doi.org/10.26342/2022-69-18
  * Bauer et al. [2022] Daniel Bauer, Tom Longley, Yueen Ma, and Tony Wilson. 2022\.  NLP in Human Rights Research: Extracting Knowledge Graphs about Police and Army Units and Their Commanders. In _Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022_. European Language Resources Association, 62–69.  https://aclanthology.org/2022.law-1.7
  * Cano [2013] Ignacio Cano. 2013\.  Violence and organized crime in brazil: The case of “militias” in rio de janeiro.  In _Transnational Organized Crime_. Transcript Verlag, 179–210.  https://www.jstor.org/stable/j.ctv1fxh0d.16
  * Clark and Sikkink [2013] Ann Marie Clark and Kathryn Sikkink. 2013.  Information Effects and Human Rights Data: Is the Good News About Increased Human Rights Information Bad News for Human Rights Measures?  _Human Rights Quarterly_ 35, 3 (2013), 539–568.  https://www.jstor.org/stable/24518073
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  https://doi.org/10.48550/arXiv.1810.04805
  * Gokhale and Fasli [2017] Ragini Gokhale and Maria Fasli. 2017.  Deploying a co-training algorithm to classify human-rights abuses. In _2017 International Conference on the Frontiers and Advances in Data Science (FADS)_. 108–113.  https://doi.org/10.1109/FADS.2017.8253206
  * Goodhart [2016] Michael Goodhart. 2016\.  _Human Rights: Politics and Practice_.  Oxford University Press.  https://doi.org/10.1093/hepl/9780198708766.001.0001
  * Greene et al. [2019] Kevin T. Greene, Baekkwan Park, and Michael Colaresi. 2019\.  Machine Learning Human Rights and Wrongs: How the Successes and Failures of Supervised Learning Algorithms Can Inform the Debate About Information Effects.  _Political Analysis_ 27, 2 (2019), 223–230.  https://doi.org/10.1017/pan.2018.11
  * Hirata and Couto [2022] Daniel Hirata and Maria Isabel Couto. 2022.  Mapa Histórico dos Grupos Armados no Rio de Janeiro.  https://geni.uff.br/2022/09/13/mapa-historico-dos-grupos-armados-no-rio-de-janeiro/
  * Hirata and Grillo [2019] Daniel Hirata and Carolina Christoph Grillo. 2019\.  Roubos, proteção patrimonial e letalidade no Rio de Janeiro.  https://geni.uff.br/2021/03/26/roubos-protecao-patrimonial-e-letalidade-no-rio-de-janeiro/
  * Hirata et al. [2023] Daniel Hirata, Carolina Christoph Grillo, Renata Coelho Dirk, and Diego Azevedo Lyra. 2023\.  Chacinas Policiais no Rio de Janeiro: Estatização das mortes, mega chacinas policiais e impunidade.  https://geni.uff.br/2023/05/05/chacinas-policiais-no-rio-de-janeiro-estatizacao-das-mortes-mega-chacinas-policiais-e-impunidade/
  * Hu et al. [2022] Yibo Hu, MohammadSaleh Hosseini, Erick Skorupa Parolin, Javier Osorio, Latifur Khan, Patrick Brandt, and Vito D’Orazio. 2022.  ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, 5469–5482.  https://doi.org/10.18653/v1/2022.naacl-main.400
  * Kolieb and Poblet [2018] Jonathan Kolieb and Marta Poblet. 2018.  Responding to Human Rights Abuses in the Digital Era: New Tools, Old Challenges.  _Stanford Journal of International Law_ 52, 2 (2018).  https://papers.ssrn.com/abstract=3859873
  * Lazer et al. [2020] David M. J. Lazer, Alex Pentland, Duncan J. Watts, Sinan Aral, Susan Athey, Noshir Contractor, Deen Freelon, Sandra Gonzalez-Bailon, Gary King, Helen Margetts, Alondra Nelson, Matthew J. Salganik, Markus Strohmaier, Alessandro Vespignani, and Claudia Wagner. 2020.  Computational social science: Obstacles and opportunities.  _Science_ 369, 6507 (2020), 1060–1062.  https://doi.org/10.1126/science.aaz8170
  * Lemgruber [2022] Julita Lemgruber. 2022\.  Tiros no futuro: Impactos da guerra às drogas na rede municipal de educação do Rio de Janeiro.  https://cesecseguranca.com.br/textodownload/tiros-no-futuro-impactos-da-guerra-as-drogas-na-rede-municipal-de-educacao-do-rio-de-janeiro/
  * Miller et al. [2013] Ben Miller, Ayush Shrestha, Jason Derby, Jennifer Olive, Karthikeyan Umapathy, Fuxin Li, and Yanjun Zhao. 2013.  Digging into human rights violations: Data modelling and collective memory. In _2013 IEEE International Conference on Big Data_. 37–45.  https://doi.org/10.1109/BigData.2013.6691668
  * Murdie and Watson [2021] Amanda M. Murdie and K. Anne Watson. 2021.  Quantitative Human Rights.  _Oxford Research Encyclopedia of International Studies_ (2021).  https://doi.org/10.1093/acrefore/9780190846626.013.603
  * Nations [2013] United Nations. 2013\.  Human Rights Indicators: A Guide to Measurement and Implementation.  https://doi.org/10.18356/58576336-en
  * Nemkova et al. [2023] Poli Nemkova, Solomon Ubani, Suleyman Olcay Polat, Nayeon Kim, and Rodney D. Nielsen. 2023.  Detecting Human Rights Violations on Social Media during Russia-Ukraine War.  https://doi.org/10.48550/arXiv.2306.05370
  * of Disease 2016 Injury Collaborators [2018] The Global Burden of Disease 2016 Injury Collaborators. 2018.  Global Mortality From Firearms, 1990-2016.  _JAMA_ 320, 8 (2018), 792.  https://doi.org/10.1001/jama.2018.10060
  * Ou et al. [2022] Zejin Ou, Yixian Ren, Danping Duan, Shihao Tang, Shaofang Zhu, Kexin Feng, Jinwei Zhang, Jiabin Liang, Yiwei Su, Yuxia Zhang, Jiaxin Cui, Yuquan Chen, Xueqiong Zhou, Chen Mao, and Zhi Wang. 2022.  Global burden and trends of firearm violence in 204 countries/territories from 1990 to 2019.  _Frontiers in Public Health_ 10 (2022).  https://doi.org/10.3389/fpubh.2022.966507
  * Pavlick and Callison-Burch [2016] Ellie Pavlick and Chris Callison-Burch. 2016.  The gun violence database. In _Presented at the Data For Good Exchange 2016_.  https://doi.org/10.48550/arXiv.1610.01670
  * Pilankar et al. [2022] Yash Pilankar, Rejwanul Haque, Mohammed Hasanuzzaman, Paul Stynes, and Pramod Pathak. 2022\.  Detecting Violation of Human Rights via Social Media. In _Proceedings of the First Computing Social Responsibility Workshop within the 13th Language Resources and Evaluation Conference_. European Language Resources Association, 40–45.  https://aclanthology.org/2022.csrnlp-1.6
  * Price and Ball [2015] Megan Price and Patrick Ball. 2015.  The Limits of Observation for Understanding Mass Violence.  _Canadian Journal of Law and Society / La Revue Canadienne Droit et Société_ 30, 2 (2015), 237–257.  https://doi.org/10.1017/cls.2015.24
  * Raleigh et al. [2010] Clionadh Raleigh, rew Linke, Håvard Hegre, and Joakim Karlsen. 2010.  Introducing ACLED: An armed conflict location and event dataset.  _Journal of peace research_ 47, 5 (2010), 651–660.  https://doi.org/10.1177/0022343310378914
  * Ran et al. [2023] Shihao Ran, Di Lu, Joel Tetreault, Aoife Cahill, and Alejandro Jaimes. 2023.  A New Task and Dataset on Detecting Attacks on Human Rights Defenders.  https://doi.org/10.48550/arXiv.2306.17695
  * Rossman and Rallis [2017] Gretchen B. Rossman and Sharon F. Rallis. 2017.  _An Introduction to Qualitative Research: Learning in the Field_.  SAGE Publications.  https://doi.org/10.4135/9781071802694
  * Silva et al. [2021] Mayalu Matos Silva, Fernanda Mendes Lages Ribeiro, Vera Cecília Frossard, Rosane Marques de Souza, Miriam Schenker, and Maria Cecília de Souza Minayo. 2021.  “No meio do fogo cruzado”: reflexões sobre os impactos da violência armada na Atenção Primária em Saúde no município do Rio de Janeiro.  _Ciência & Saúde Coletiva_ 26 (2021), 2109–2118.  https://doi.org/10.1590/1413-81232021266.00632021
  * Souza et al. [2020] Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020\.  BERTimbau: Pretrained BERT Models for Brazilian Portuguese. In _Intelligent Systems_ _(Lecture Notes in Computer Science)_ , Ricardo Cerri and Ronaldo C. Prati (Eds.). Springer International Publishing, 403–417.  https://doi.org/10.1007/978-3-030-61377-8_28
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is all you need. In _Advances in neural information processing systems_ , I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30.  https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Wagner Filho et al. [2018] Jorge A. Wagner Filho, Rodrigo Wilkens, Marco Idiart, and Aline Villavicencio. 2018.  The brWaC corpus: a new open resource for Brazilian Portuguese. In _Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018)_. European Language Resources Association.  https://aclanthology.org/L18-1686

##  Appendix A Data description

Figure 3: Data collection and transformations to create the training and
testing datasets.

##  Appendix B Interview questions and survey results

One month after starting the intervention, we interviewed four participants
from Fogo Cruzado’s team in Rio de Janeiro in July 2023 using the following
questions. The questions sought to explore the participants’ overall
experience with the intervention and evaluate the efficacy of the prototype
developed.

  * •

How do you describe your overall experience using the prototype?

  * •

Have you used to identify non-geotagged messages?

  * •

How often did you check the posts classified as negative cases?

  * •

How have you combined the prototype with your traditional workflow to monitor
tweets in Tweetdeck?

  * •

What were the main advantages of using the prototype?

  * •

What were the main drawbacks?

  * •

Can you identify any pattern in cases where the model misclassifies a report?

  * •

How important are messages that are not potential reports of gun violence but
might contain relevant information for your monitoring work?

We sent the first online survey to Fogo Cruzado’s staff in Rio de Janeiro on
May 26 2023, before the intervention, which started on May 29 2023. The form
to evaluate the intervention was shared on September 4 2023, and the response
rate was good. Four out of the five participants in the core team of analysts
in Rio de Janeiro answered the survey.

Figure 4: Questionnaire prior to the intervention (questions 1 and 2). Figure
5: Questionnaire prior to the intervention (questions 3 and 4). Figure 6:
Questionnaire prior to the intervention (questions 5 and 6). Figure 7:
Questionnaire after the intervention (questions 1 and 2). Figure 8:
Questionnaire after the intervention (questions 4 to 6).

##  Appendix C Difference-in-difference model

This Appendix seeks to answer the following question: Did the intervention
allow analysts to interact with more messages? The intervention involved
changing Fogo Cruzado Institute’s workflow for searching and interacting with
citizen-generated reports of gun violence on Twitter. First, we developed a
Natural Language Processing model to detect gun violence reports and
implemented a prototype for real-time monitoring of Twitter messages. Fogo
Cruzado’s analysts in Rio de Janeiro, Brazil, used this prototype for five
weeks. Then, we analysed difference-in-difference models with control
variables and Fogo Cruzado’s team in Bahia as a control group. We found
evidence suggesting that adopting the model increases analysts’ capacity to
interact with reports.

We defined the pre-intervention start date as March 1 2023 because it was when
the last change in Fogo Cruzado’s analyst staff happened; since then, the team
composition has remained unchanged. The intervention period under analysis
spans from May 29 to July 2 2023. We collected daily data from Fogo Cruzado’s
public API101010https://api.fogocruzado.org.br/ocurrences and Twitter
profiles111111@FogoCruzadoBA, @FogoCruzadoRJ, @FogoCruzadoPE. The former was
used to get records from gun violence events, and the latter measured the
number of interactions with gun violence reports.

Figure 9 presents the total of Fogo Cruzado’s interactions with users in all
metropolitan regions currently monitored: Rio de Janeiro, Bahia and
Pernambuco. Fogo Cruzado uses different Twitter profiles in each region.
Although all teams employ the same methodology in all states, the use of
Twitter to crowdsource gun violence reports varies across the three states.
Several factors can potentially influence the total number of interactions on
Twitter across states, such as the number of gun violence events in a single
day, their lethality, or demographic differences in the Twitter user base in
Brazil.

Figure 9: Fogo Cruzado’s interactions with users reporting gun violence per
day on Twitter in Bahia, Rio de Janeiro, and Pernambuco.

Figure 9 shows that while Rio de Janeiro presents an intense activity, Bahia
has a much smaller but consistent number of daily replies to gun violence
reports. In contrast, Pernambuco has only sporadic activity, with long periods
without interactions with users reporting gun violence on Twitter. This
characteristic justifies the choice of Bahia as a control group.

Then, we estimated the impact of the intervention using a difference-in-
difference approach. We defined Fogo Cruzado’s team in Rio de Janeiro as a
treatment group and Bahia as a control group. This research design is possible
because both groups use the same methodology to track cases, but Bahia did not
have access to the NLP-powered prototype to browse tweets.

The dependent variable is the count of interactions with gun violence reports
on Twitter per day and region/user. The users are @FogoCruzadoRJ and
@FogoCruzadoBA, i.e., the Twitter profiles maintained by Fogo Cruzado
Institute to monitor and interact with reports from the metropolitan region of
two Brazilian states, namely Rio de Janeiro and Bahia. The distribution of the
dependent variable is highly right-skewed, with an average of 12 replies and a
variance of 116.

The difference in means indicates that the intervention allowed analysts from
Rio de Janeiro to interact with nine more reports on average. The mean number
of replies before the intervention in Bahia was found to be 6, while in Rio de
Janeiro, it was 17. After implementing the intervention, the mean number of
replies decreased to 4 (-2) in Bahia, while in Rio de Janeiro, it increased to
24 (+7). Thus, the difference in the means is 9 (+7 - (-2)); thus, the
intervention is associated with an increase of about nine interactions with
reports of gun violence per day. We obtained the same estimation using the
regression models and controlling for confounding variables.

Figure 10 shows the number of interactions per day in the treatment (Rio de
Janeiro) and control group (Bahia). The chart shows a stable trendline for
Bahia and a positive trendline for Rio de Janeiro, reverting the downward
trend observed in the pre-intervention period only, which can be observed in
Figure 12.

Figure 10: Interactions per day of Fogo Cruzado’s team in Bahia and Rio de
Janeiro before and after the intervention.

###  C.1 Regression models

We used two regression models in the difference-in-difference analysis: an
Ordinary Least Square model and a Negative Binomial model. The first offers a
starting point for the analysis and comparison but is not optimal for the task
at hand because some of its assumptions may not hold true for skewed count
data. Conversely, the Negative Binomial is an alternative for Poisson suitable
for cases where the variance is greater than the mean (overdispersed data).

We defined the following variables to run the difference-in-difference
regression. The subscript (t) represents the day.

  * •

$replies_{t}$: the dependent variable is the number of interactions with
reports of gun violence per day on Twitter.

  * •

$\beta_{0}$: the intercept of the model.

  * •

$\beta_{1}\cdot intervention_{t}$: 1 whether the intervention was running in
the day t and 0 otherwise.

  * •

$\beta_{2}\cdot treatment$: 1 for Rio de Janeiro and 0 for Bahia, indicating
the treatment and control group.

  * •

$\beta_{3}\cdot(intervention_{t}\cdot treatment_{t})$: interaction term
between intervention and treatment.

  * •

$\beta_{4}\cdot number\\_cases_{t}$: number of gun violence events in the day
according to Fogo Cruzado’s database.

  * •

$\beta_{5}\cdot number\\_victims_{t}$: number of victims (killed or injured
people) of gun violence in the day t according to Fogo Cruzado’s database.

  * •

$\beta_{6}\cdot avg\\_population_{t}$: population average of the cities
affected in the day according to Fogo Cruzado’s database.

  * •

$\varepsilon_{t}$: error term.

Table 4 summarises the coefficients for each estimate; values inside
parenthesis are the confidence intervals. The formula used for regression was:

$replies_{t}=\beta_{0}+\beta_{1}\cdot intervention_{t}+\beta_{2}\cdot
treatment+\beta_{3}\cdot(intervention_{t}\cdot treatment_{t})+\beta_{4}\cdot
number\\_cases_{t}+\beta_{5}\cdot number\\_victims_{t}+\beta_{6}\cdot
avg\\_population_{t}+\varepsilon_{t}$.

Both models indicate that the intervention for the treatment group
(intervention_treatment) appears to have a statistically significant effect on
increasing the number of interactions with gun violence reports on Twitter.
The coefficient for this variable (intervention_treatment) in the OLS model is
9.7, an estimate close to the simple difference in means reported before.

The treatment variable also presents a high coefficient, indicating that being
from the treatment group increases the number of interactions per day, even in
the absence of the intervention. This finding is coherent with Figure 9, which
shows that Rio de Janeiro has consistently higher interaction than Bahia.
There is no statistically significant effect in any of the control variables.

The observed R-squared values fall within a moderate range, specifically from
0.31 to 0.46. These values suggest that the independent variables in the
models can predict approximately 31% to 46% of the variability in the
dependent variable. In the case of the OLS model, nearly 46% of the
fluctuations in ’replies_t’ can be explained by the model. When we adjust for
the number of predictors, the value marginally decreases to 45%. The Negative
Binomial model does not provide a traditional R-squared value. Instead, we use
the Pseudo R-squared measure, which is tailored to assess goodness-of-fit in
non-linear regression contexts. The Pseudo R-squared value (Cox-Snell) in this
model is 0.31.

| Dependent variable: $replies_{t}$
---|---
| Ordinary Least Square | Negative Binomial
| (1) | (2)
Intercept | 9.430∗∗∗ | 1.986∗∗∗
| (3.914 , 14.946) | (1.269 , 2.703)
avg_population | -0.000∗ | -0.000
| (-0.000 , 0.000) | (-0.000 , 0.000)
intervention | -2.389 | -0.454∗∗
| (-5.572 , 0.793) | (-0.892 , -0.017)
intervention_treatment | 9.693∗∗∗ | 0.832∗∗∗
| (5.180 , 14.207) | (0.234 , 1.429)
number_cases | -0.898 | -0.039
| (-2.373 , 0.576) | (-0.231 , 0.152)
number_victims | 0.213 | 0.012
| (-0.111 , 0.536) | (-0.031 , 0.054)
treatment | 13.666∗∗∗ | 1.175∗∗∗
| (9.295 , 18.037) | (0.607 , 1.743)
Observations | 245 | 245
$R^{2}$ | 0.463 |
Adjusted $R^{2}$ | 0.449 |
Pseudo $R^{2}$ (Cox-Snell) |  | 0.31
Residual Std. Error | 8.024(df = 238) | 1.000(df = 238)
F Statistic | 34.203∗∗∗ (df = 6.0; 238.0) | (df = 6; 238)
Note: |  ∗p$<$0.1; ∗∗p$<$0.05; ∗∗∗p$<$0.01
Table 4: Difference-in-difference regression results.

###  C.2 Regression diagnostics

We used two charts to analyse visually the parallel trend assumption of
difference-in-difference models. The parallel trend assumption requires that
the changes over time in the dependent variable for both groups are parallel
in the absence of the intervention. Figure 12 shows the number of daily
interactions, i.e., replies to gun violence reports. Figure 11 shows the
growth trend for the cumulative sum of interactions on Twitter.

There is a rough parallel trend in Figure 11. On the other hand, in Figure 12,
Rio de Janeiro has a downward trendline, and the trend of daily interactions
of Fogo Cruzado’s team in Bahia seems stable.

Figure 11: Testing the parallel trend assumption before and after the
intervention (cumulative sum of interactions). Figure 12: Testing the parallel
trend assumption before the intervention (number of interactions per day).

We assessed the goodness of fit for the models using Quantile-Quantile (QQ)
plots, represented in Figure 14. The two QQ plots show that points closely
align with the theoretical expectations at the centre, which implies that the
residuals are approximately normally distributed around the mean, but there
are deviations from the theoretical line for extreme quantiles. This pattern
suggests that the models have good accuracy for predicting the dependent
variable around its mean value but might not be as effective when dealing with
the outliers.

Figure 13: QQplot for the Ordinary Least Square model. Figure 14: QQplot for
the regression of the negative binomial model.