# Bots, Elections, and Controversies:
Twitter Insights from Brazil’s Polarised Elections

Diogo Pacheco<EMAIL_ADDRESS>University of Exeter, UK

###### Abstract

From 2018 to 2023, Brazil experienced its most fiercely contested elections in
history, resulting in the election of far-right candidate Jair Bolsonaro
followed by the left-wing, Lula da Silva. This period was marked by a murder
attempt, a coup attempt, the pandemic, and a plethora of conspiracy theories
and controversies. This paper analyses 437 million tweets originating from 13
million accounts associated with Brazilian politics during these two
presidential election cycles. We focus on accounts’ behavioural patterns. We
noted a quasi-monotonic escalation in bot engagement, marked by notable surges
both during COVID-19 and in the aftermath of the 2022 election. The data
revealed a strong correlation between bot engagement and the number of replies
during a single day ($r=0.66$, $p<0.01$). Furthermore, we identified a range
of suspicious activities, including an unusually high number of accounts being
created on the same day, with some days witnessing over 20,000 new accounts
and super-prolific accounts generating close to 100,000 tweets. Lastly, we
uncovered a sprawling network of accounts sharing Twitter handles, with a
select few managing to utilise more than 100 distinct handles. This work can
be instrumental in dismantling coordinated campaigns and offer valuable
insights for the enhancement of bot detection algorithms.

Brazilian Elections, Bots, Twitter, Political Networks, Polarisation

##  I Introduction

Brexit in Europe, Trump in the U.S., and Bolsonaro in Brazil exemplify the
escalating polarisation characterising political discourse worldwide Eger and
Valdez (2015). Simultaneously, the pivotal role of online social platforms as
primary mediums for campaigns, debates, and recruitment has come to the
forefront Badawy et al. (2018); Vosoughi et al. (2018); Yang et al. (2019).

The presence of bots in electoral campaigns has seen a year-on-year increase,
coinciding with a growing academic focus Bessi and Ferrara (2016). This
expanding body of literature delves into elections worldwide, encompassing the
2016 U.S. elections Badawy et al. (2019), the 2017 electoral contests in
Germany Keller and Klinger (2019) and in France Ferrara (2017), Italy in 2018
Radicioni et al. (2021), Spain in 2019 Pastor-Galindo et al. (2020), elections
across numerous African countries during 2017-2018 Ndlela (2020), the Asia-
Pacific region in 2019-2020 Uyheng and Carley (2021), and the 2019 European
Parliament elections Pierri et al. (2020), to name a few. However, a more
pressing concern emerges as the diffusion of misinformation disproportionately
affects accounts depending on their political affiliations Badawy et al.
(2019); Luceri et al. (2019a); Chen et al. (2021).

In 2016, Brazil experienced political turmoil with the impeachment of Dilma
Rousseff. Subsequently, the years 2018 and 2022 witnessed Brazil’s most hotly
contested elections in its history, culminating in the elections of far-right
candidate Jair Bolsonaro, followed by the left-wing figure Lula da Silva. This
era was overshadowed by a murder attempt, a coup attempt, the pandemic, and a
profusion of conspiracy theories and controversies, creating fertile ground
for misinformation. Notably, Brazilian datasets have been at the forefront of
developing computational methods for detecting propaganda Arnaudo (2017),
countering misinformation in advertisements Silva and Benevenuto (2021),
identifying low-credibility Brazilian websites Couto et al. (2022), and fact-
checking images Reis et al. (2020). WhatsApp groups, immensely popular in
Brazil, played a pivotal role in monitoring misinformation spread during the
2018 elections Machado et al. (2019). Furthermore, substantial criticism has
emerged regarding the use of misinformation as a political weapon during the
COVID-19 pandemic Ricard and Medeiros (2020) and culminating in Bolsonaro’s
ineligibility until 2030 Savarese and Jeantet (2023).

In this study, we harness social media data and network analysis to discern
and illuminate population-level political behaviour in Brazil. Our analysis
tracks the evolution of political groups, from contentious competitors during
campaigns to government and opposition blocks after elections. Our findings
illuminate a transition from a pre-election phase marked by numerous polarised
groups to a post-election phase in which these factions coalesce into
government and opposition clusters. Our investigation uncovers a sprawling
network of coordinated accounts that share Twitter handles. We also observe a
pronounced surge in bot engagement, with noteworthy peaks during the pandemic
and in the aftermath of the 2022 election. Furthermore, our data underscores a
strong correlation between bot engagement and the number of replies. Finally,
we identify anomalous days characterised by an unexpectedly high number of
account creations.

We employed the Twitter streaming API to monitor fourteen Brazilian
presidential candidates during the 2018 elections, and thirteen candidates and
twenty-seven political parties during the 2022 cycle. The data collection
spanned from August 30, 2018, to March 14, 2023. The period encompasses 1,657
days, and the collection process remained active for 94% of this time. This
comprehensive effort resulted in the acquisition of a vast dataset comprising
437 million tweets originating from 13 million distinct accounts.

##  II Dissecting Twitter Accounts

###  II.1 Dynamics of political engagement

\begin{overpic}[width=429.28616pt]{images/bra_elections-2018_23-tweets_per_day-
mean} \put(2.0,25.0){{A}}
\end{overpic}\begin{overpic}[width=108.405pt]{images/bra_elections-2018_23-cumulative_users}
\put(5.0,90.0){{B}} \end{overpic}

Figure 1: A Political Tale from Tweets — (A) Timelines spanning two election
cycles (2018/22), covering 1657 days and involving 437 million tweets (in
orange) from 13 million accounts (in purple). Daily tweet counts are
represented by the lighter lines, while the 30-day moving average is depicted
in bold. Key events, such as the 2018 (#1) and 2022 (#7) election days, are
highlighted for reference. (B) The cumulative plot illustrates a continuous
increase in the number of distinct accounts joining the political conversation
for the first time. The red vertical lines (1/July/22) indicate the beginning
of the 2022 election cycle, while the red dashed line marks the election day.

The Twitter timeline depicted in Figure 1A reveals discernible shifts in
political engagement. In 2018, there is a notable surge in activity leading up
to the election day, followed by a decline in the period between the release
of election results and the inauguration day (January 1, 2019). Subsequently,
the volume of tweets and active users stabilises, punctuated by occasional
peaks corresponding to significant events. The volume of tweets remained
relatively low throughout 2020 until the onset of COVID. Subsequently, a
series of peaks emerged, driven by discussions surrounding both the pandemic
and political developments. The most significant surge occurred at the
beginning of 2022, building steadily until the election day. A pattern akin to
2018 repeats as there is a decline between the election and the inauguration.
Notably, 2022 also witnessed an abrupt surge coinciding with the coup attempt
on January 8, 2023.

Despite the somewhat consistent daily number of accounts engaging in the
conversation, Figure 1B reveals an intriguing trend wherein more than three
thousand new accounts join the Brazilian political discourse each day. This
observation hints at an account churn rate of approximately 5%. Importantly,
the introduction of new terms into the data collection on July 1, 2022, does
not appear to have significantly influenced the influx of new accounts. In
forthcoming research endeavours, we intend to delve deeper into the dynamics
of accounts exiting the conversation. One plausible interpretation for this is
that it may be driven by a substantial presence of bots within the Twitter
ecosystem. These bots could potentially be replaced by new ones as they are
suspended by the platform for policy violations. However, it is essential to
note that in our current analysis, we did not assess bot activity among the
incoming accounts.

###  II.2 Accounts’ heterogeneous characteristics

\begin{overpic}[width=390.25534pt]{images/bra_elections-2018_23-users_distributions}
\put(5.0,33.0){{A}} \put(38.0,33.0){{B}} \put(69.0,33.0){{C}} \end{overpic}
Figure 2:  Heterogeneous Behaviour of Accounts — (A) Distribution of the
number of tweets per account, showcasing a long-tail pattern where a small
number of accounts post close to 100k tweets. (B) Distribution of the number
of active days, revealing exceptional accounts that tweet every day. (C)
Distribution of the number of distinct screen names used by accounts, with
some accounts utilizing more than 100 different names.

Consistent with observations in various social systems, our dataset
underscores the presence of accounts exhibiting heavy-tailed properties.
Figure 2 illustrates this phenomenon, wherein the majority of accounts
contribute relatively few tweets, while a select few manage to produce an
exceptionally high volume, nearing 100,000 tweets within the specified
timeframe. It is worth noting that, despite Twitter’s imposed limit of 2,400
tweets per day, some accounts employ strategies to circumvent this
restriction, often through the adoption of abusive deletion behaviours Torres-
Lugo et al. (2022).

The skewness observed in the distribution of tweet volume is mirrored in the
distribution of active days. While the majority of accounts engage for just a
few days, there exists a subset of accounts that remain active on a daily
basis. However, it is imperative to acknowledge that the figures presented
herein may be underrepresented, as they pertain exclusively to the tweets
captured by our data collection. These extremes in user behaviour raise
suspicions of automation.

Prior research has highlighted the association of multiple handles (i.e.,
screen names) used by a single account or shared among multiple accounts with
potentially malicious activities Mariconti et al. (2017) and coordinated
campaigns Pacheco et al. (2021). Figure 2 further elucidates this trend by
illustrating the distribution of the number of distinct names employed by the
accounts within our dataset. It is noteworthy that some accounts exhibit the
use of more than a hundred distinct handles, amplifying concerns of
potentially deceptive practices.

###  II.3 Coordinated accounts

Figure 3:  Screen Name Sharing Network — Each node in the network represents a
Twitter account, and they are connected if they share a common handle
(screen_name). The size of each node is proportional to the number of tweets
posted, and different colours represent various suspicious coordinated groups
(connected components). For clarity, we display only groups consisting of at
least 10 accounts or those responsible for producing more than 10,000 tweets.

Pacheco et al. (2021) introduced a framework for identifying coordinated
campaigns on Twitter, focusing on the presence of shared handles among
multiple accounts . This entails different accounts, signified by distinct
user_id, adopting the same perceived identity, denoted by identical
screen_name. Importantly, this methodology enables the detection of
coordinated groups of accounts, regardless of their automation level,
extending the scope beyond bots.

Figure 3 displays the outcomes of the coordination detection Pacheco et al.
(2021) within our dataset. The figure unveils numerous connected components,
representing the coordinated groups, which vary in size. Notably, the figure
emphasises the most suspicious groups, filtered either due to their size, with
more than ten accounts involved, or because of their prolific engagement in
the conversation, generating over ten thousand tweets.

It is important to highlight that, in this study, we did not uncover groups
involved in name squatting or hijacking, as previously reported in the
literature Mariconti et al. (2017); Pacheco et al. (2021). This discrepancy
can be attributed to the distinctions between our dataset, which is domain-
specific, and the datasets employed in prior research, which were domain-
agnostic. Furthermore, our analysis refrains from delving into the specifics
of the campaigns undertaken by these groups or their overall impact on the
broader discourse. These facets will be addressed in future research.

###  II.4 Bots engagement

\begin{overpic}[width=411.93767pt]{images/bra_elections-2018_23-botscore_per_day-
mean} \put(3.0,78.0){{A}} \put(3.0,40.0){{B}} \end{overpic} Figure 4:
Increasing Bot Activity — (A) The daily average botscore derived from tweets
exhibits a continuous upward trend, reaching peaks in the days following the
2022 election. (B) The percentage of accounts exhibiting bot-like behaviour
remains relatively stable in the dataset, with notable increases observed
after the initial wave of the pandemic and following the 2022 election.

In this section, we explore the temporal evolution of bot engagement by
utilising BotometerLite Yang et al. (2020), a tool designed for the assessment
of bot-like activities within social media data. It is essential to
acknowledge that while bot detection algorithms are valuable, they are far
from infallible Cresci et al. (2023). These algorithms have faced criticism on
various fronts, including concerns about their lack of transparency Ferreira
Dos Santos et al. (2019), the presence of elevated numbers of false negatives
and false positives Hays et al. (2023); Nasim et al. (2018), and issues of
historical data Chowdhury et al. (2020). To mitigate some of these criticisms,
our analysis focuses on bot activity as a broad trend, avoiding specific
account-level scrutiny or rigid threshold definitions.

BotometerLite Yang et al. (2020) operates by assessing a single tweet,
specifically the user profile object within a tweet, to assign a botscore to
the account responsible for that tweet. The botscore, which can range from
zero to one, serves as an indicator of the extent to which an account’s
features resemble those of a human versus automated account (bot) activities.
It is important to note that the user profile features used for this
assessment are subject to change over time, meaning that even two consecutive
tweets from the same account may yield different botscores.

For our analysis, we define the daily botscore of an account as the average
botscore derived from all of its tweets within a given day. Additionally, we
establish the concept of bot engagement or content botscore, denoting the
average botscore calculated from all tweets collectively.

As illustrated in Figure 4A, we present the evolving landscape of bot
engagement within the discourse surrounding Brazilian politics. While the
daily engagement displays noticeable fluctuations, the moving average reveals
a pronounced upward trajectory that has persisted since the commencement of
our data collection in 2018. Notably, we observe a significant surge in bot
engagement commencing in March 2020 during the pandemic, and this trend
further intensifies in the aftermath of the 2022 elections. This observed
trend aligns with findings reported by academics and media outlets, which have
highlighted the escalating dissemination of disinformation in Brazil. Of
particular concern are unsubstantiated claims, often attributed to Bolsonaro,
regarding the e-voting system Rossini et al. (2023); Green (2022); Check
(2022).

The recent acquisition of Twitter has sparked considerable controversy
regarding the prevalence of bots on the platform Knight (2022). In 2017, Varol
et al. (2017) proposed a method for conducting a census of Twitter accounts
and found that approximately 9% to 15% of accounts were likely to be social
bots. In this study, we refrain from providing a specific numerical estimate
and instead examine trends surrounding the presence of bots within our
dataset.

Figure 4B portrays the temporal evolution of the daily percentage of what we
term suspicious users. In our analysis, we categorise suspicious accounts as
those with a botscore exceeding 0.5. The percentage of bots within this
category remains relatively stable, fluctuating between 15% and 20%. It is
worth noting that varying thresholds for suspicious accounts would result in
different quantities of bots, but the overall stable trend persists.
Noteworthy spikes in bot activity occur during COVID and in the aftermath of
the 2022 elections, with specific days registering a particularly high
proportion, exceeding 50% of the total accounts.

The convergence of two key results, namely the escalating content botscore and
the sustained proportion of bots, offers compelling evidence that bots are
progressively intensifying their involvement in the ongoing conversations.
This observation raises important questions about the effectiveness of
existing measures aimed at countering and mitigating bot activities. It hints
at the possibility that current initiatives designed to combat and block bots
may not be sufficient to curtail their presence and influence.

###  II.5 Replying bots

Figure 5:  Relationship Between Retweet and Reply, and Bot Engagement — The
percentage of replies in a day exhibits a significant positive correlation
($r=0.66$) with bot engagement, while the percentage of retweets demonstrates
a negative correlation ($r=-0.55$) with bot activity.  Figure 6:  Evolution of
Tweet Types Over Time — While retweets remain the dominant type of tweets,
there is a noticeable increase in the popularity of replies over time.

Mbona and Eloff (2022) employed a combination of Benford’s Law, Principal
Component Analysis (PCA), and random forest techniques to identify
discriminative features for bot detection. Notably, their findings underscored
that the number of retweets serves as an effective discriminator, whereas the
number of replies did not exhibit the same discriminatory power. In contrast,
Pozzana and Ferrara (2020) demonstrated that both the fraction of retweets and
replies tend to be more prevalent in human interactions compared to bot-driven
activities. Finally, Mazza et al. (2022) distinguished between trolls and
social bots, revealing that the latter tend to employ a higher volume of
replies than human users.

In this section, we delve into the intricate relationships among these
engagement metrics and the overarching content botscore, as defined in Section
II.4. A lower content botscore signifies a scenario in which the majority of
tweets originate from accounts that exhibit human-like characteristics, while
a higher content botscore indicates a greater involvement of automated
accounts in the conversation. Figure 5 showcases the distributions and
correlation patterns among the percentage of retweets, percentage of replies,
and the content botscore. Our results shed light on the distinct tendencies of
bots, particularly their propensity for engaging through replies, offering
valuable insights into the interplay of these engagement metrics.

Figure 6 offers a chronological perspective on the proportions of different
tweet types. Throughout this timeline, retweets consistently dominate the
landscape, maintaining a prominent presence. However, notable shifts in tweet
composition are observed. For instance, there is a discernible 10% decline in
the retweet rate, plummeting from 71% prior to the second round of the 2018
election to 61% following the inauguration day. In stark contrast, the number
of replies more than doubled during the same period, surging from 14% to 30%.
These changes may reflect two distinct behavioural patterns: the prevalence of
propaganda-oriented content during election campaigns, juxtaposed with an
emphasis on discourse and debate in the post-election mandate period. This
phenomenon, characterised by shifts in the composition of tweet types, was not
unique to the 2018 election cycle but recurred during the 2022 cycle as well
at a lower scale.

###  II.6 Uncovering accounts “birthdays”

Figure 7: The distribution of the number of accounts created per day. Figure
8:  Accounts Created on the Same Date (“Birthday Twins”) — This figure
displays the count of accounts created per day, highlighting the age diversity
of Twitter accounts in the dataset. Unusual peaks are also observed, including
accounts created as far back as the early days of Twitter.

Contrary to the zodiac, online “dates of birth” can reveal a wealth of
information beyond just an account’s age. Prior research, such as Tardelli et
al. (2020), has demonstrated that financial social bots often share similar
creation dates. Jones (2019) successfully detected bots exploiting the Gulf
crisis primarily by analysing account creation dates. Similarly, Takacs and
McCulloh (2019) utilised creation dates to identify dormant bots during the
2018 US Senate election. These bots Takacs and McCulloh (2019) were not
particularly active, yet they attempted to exert influence based on their
substantial number of followers.

In our investigation, we embark on a quest for days marked by a substantial
influx of “newborn” accounts. We scrutinised the creation dates of each of the
13 million accounts actively participating in the discourse surrounding
Brazilian politics and categorised them accordingly. Figure 7 visually depicts
the distribution of the number of ”newborns” per day. This distribution
exhibits a fat-tailed pattern, characterised by a decline in the probability
of multiple accounts being created on the same day up to around 1,000
creations daily. Subsequently, the probability rises, peaking at approximately
3,000 accounts per day, before sharply declining once more.

Figure 8 provides a timeline of account creation counts for those accounts
born on the same date. Notably, the plot expectedly reveals that numerous
accounts were established long before our data collection began, with some
dating back to the inception of Twitter itself. However, the plot also unveils
peculiar and anomalous peaks, primarily concentrated in 2020, with a maximum
day in 2022 registering the creation of over 20,000 accounts.

Of particular note is an outlier peak on 1st January 1970, which we chose to
omit from the plot. This anomaly, marked by the creation of 41 accounts on a
date preceding the existence of the Twitter platform itself, is unequivocally
suspicious. The existence of accounts with creation dates preceding the
platform’s inception, as well as those exhibiting multiple creation dates,
presents a puzzling phenomenon. We remain uncertain whether this issue is an
innocuous glitch or a deliberately orchestrated malicious activity. Although
we have not encountered official reports on this matter, it has garnered
attention on social media neeljai.

In future research, we plan to delve deeper into the analysis and
characterisation of these enigmatic accounts, shedding light on their origins
and potential significance.

##  III Discussions

#### Bot Detection Challenges

The significant increase in bot engagement, as evidenced by our findings,
underscores the escalating concerns about our capacity to effectively combat
fringe actors. Our use of BotometerLite, reliant on historical data and a
model trained in 2020, might not fully encapsulate the evolving nature of bot
behaviour, especially during critical events like elections. Research has
illuminated the adaptive tactics of bots during elections and the formidable
challenges in detecting automated accounts Luceri et al. (2019b). Detecting
bots has become a more intricate task, and the proliferation of misinformation
may be greatly exacerbated by innovations like GPT Yang and Menczer (2023);
Ezzeddine et al. (2023).

#### Dataset Bias

While our analysis is grounded in a substantial sample of online users, it
remains uncertain how representative Twitter data is of the broader Brazilian
political spectrum. It is crucial to acknowledge that no dataset is entirely
free from bias. Many research efforts rely on datasets constructed using
dynamic keyword-based approaches, which involve continually updating tracking
terms to adapt to the evolving online environment. For example, some
researchers employ snowball techniques to harvest new hashtags, resulting in
datasets that are tailored to current trends. In contrast, our approach was
distinct. We aimed to minimise changes to tracking terms. For instance, we
retained hashtags primarily associated with campaign periods throughout our
study duration (see Tables 1 and 2). Similarly, we continued tracking all
presidential candidates even after the elections. Notably, a substantial
proportion of these candidates remained actively engaged within the Brazilian
political landscape. A striking 46% of the 2022 candidates were participants
in the 2018 cycle. Some of these candidates joined coalitions to support new
contenders, while others assumed leadership roles in political parties or
government. Maintaining a stable list of tracked individuals allowed us to
consistently monitor Brazilian politics without introducing additional bias
stemming from trending topics.

#### Twitter’s Evolution

The transformation of Twitter has gone beyond a mere re-branding to $X$. The
new API introduces significant limitations on data collection, which have the
potential to hamper the monitoring capacity of academics and open new avenues
for exploitation by malicious actors. However, it is not yet clear whether
certain behaviours observed on “old” Twitter have ceased to exist on $X$.
Consequently, it is imperative to continue exploring datasets from the older
version of Twitter. Furthermore, it’s unlikely that researchers can
reconstruct such a comprehensive dataset as the one presented here. Although
we are unable to directly share our data, we are actively seeking
collaborations to expand and extend this research.

#### Future Research Directions

Future work should delve into the dynamics of accounts joining the political
discourse. Who constitutes the persistent core of participants? Does the churn
rate only capture isolated instances of engagement? Do accounts engage
periodically or based on specific topics? The coordinated suspicious groups
and accounts created on the same day warrant further investigation, including
characterisation efforts to identify who these actors are and the subjects
they discuss. Additionally, it is imperative to measure the impact of their
actions on the overall conversation and trace back groups involved in the coup
attempt. Despite the lingering questions, we anticipate that this work will
play a pivotal role in dismantling coordinated campaigns and offer valuable
insights to enhance bot detection algorithms.

In summary, our research has revealed the alarming growth in bot engagement,
raising concerns about our ability to combat fringe actors effectively. While
our study is not without limitations, such as the evolving nature of bot
behaviour and dataset biases, it has provided valuable insights into the
landscape of Brazilian politics. As we confront the challenges of evolving
social media platforms and advancing technologies, it is imperative to
continue probing these issues and collaborating to develop effective
solutions.

##  IV Data Collection and Context

This paper examines a dataset consisting of 437 million tweets generated by 13
million accounts associated with Brazilian politics between 2018 and 2023.
Before delving into the specifics of data collection, it is essential to
provide a contextual overview of the Brazilian electoral process.

Brazil operates as a federal presidential representative democratic republic
with a multi-party system, comprising 27 federal units (states and a federal
district). Voting in Brazil is mandatory for individuals aged between 18 and
70 years, while it is optional for those under 18, over 16, or over 70.
Elected Brazilian politicians generally serve four-year terms, and the
population is required to select their representatives in general elections
every two years, alternating between federal and local elections. For
instance, the years 2018 and 2022 constituted federal elections for the
positions of president, governor, and federal congressmen, while 2016 and 2020
featured elections for mayor, state deputies, and city councillors. Elections
in Brazil are conducted on a single day, during which all votes must be cast
in person, typically on a Sunday in October, between 8 AM and 5 PM. In cases
where no candidate secures an absolute majority of the valid votes (more than
50%), a second round of voting is held, featuring the two leading candidates.

Since 1996, Brazil has employed electronic voting machines, which have
eliminated paper-based fraud and enabled rapid result tabulation. Despite
increasing concerns regarding the system’s security, it undergoes regular
audits and testing by representatives from all political parties and various
organisations, including cybersecurity experts. So far, there has been no
concrete evidence of corruption in the system Aranha and van de Graaf (2018);
Avgerou et al. (2009); Zucco Jr and Nicolau (2016). There are currently 30
parties registered at the Superior Electoral Court (TSE)Court (2023). Each
party is assigned a unique identification number, which is used as part of a
candidate’s ID. For most positions (e.g., president and governor), each party
can field at most one candidate, and their ID corresponds to the party number
itself. Candidate IDs are prominently featured in campaign materials, as
voters must type them to cast their e-vote.

Our dataset was compiled using the Twitter streaming API. Data collection
commenced in August 2018 and continued until the API’s termination in March
2023. We focused on the presidential elections and, for each candidate,
monitored (i) the official Twitter account, (ii) the official campaign hashtag
(often following the pattern “#¡last name¿+¡candidate ID¿”), and (iii) the
candidate’s full name. We also tracked the Twitter account of the Superior
Electoral Court (TSE). Table 1 provides an overview of the keywords employed
during the 2018 election cycle.

In July 2022, the TSE officially released the updated list of candidates for
the 2022 elections. This event prompted the sole adjustment to the set of
keywords over the five-year period. Table 2 features the revised list of
candidates and associated keywords. Additionally, we initiated monitoring of
the official accounts of Brazilian political parties and the Supreme Court
(STF). Table 3 presents the parties’ accounts added for the 2022 cycle.

Table 1:  2018 Candidates and Government Accounts Keywords — This table lists the keywords used in data collection from August 30, 2018, until June 30, 2022. Highlighted terms were retained for the 2022 election cycle. Terms in italics are descriptive and were not tracked. Name | Account | Hashtag
---|---|---
Alvaro Dias | @alvarodias_ | #AlvaroDias19
Cabo Daciolo | @CaboDaciolo | #Daciolo51
Ciro Gomes | @cirogomes | #Ciro12
Jose Maria Eymael | @Eymaeloficial | #Eymael27
Fernando Haddad | @Haddad_Fernando | #Haddad13
Geraldo Alckmin | @geraldoalckmin | #Alckmin45
Guilherme Boulos | @GuilhermeBoulos | #Boulos50
Henrique Meirelles | @meirelles | #Meirelles15
Jair Bolsonaro | @jairbolsonaro | #Bolsonaro17
Joao Amoedo | @joaoamoedonovo | #Amoedo30
Joao Goulart Filho | @joaogoulart54 | #Goulart54
Luiz Inacio Lula da Silva | @LulaOficial | #Lula13
Marina Silva | @MarinaSilva | #Marina18
Vera Lucia | @verapstu | #Vera16
Superior Electoral Court | @TSEjusbr | #Eleiçoes2018
Table 2:  2022 Candidates and Government Accounts Keywords — This table presents the keywords used in data collection from July 1, 2022, until March 14, 2023. Highlighted terms carried over from the 2018 election cycle. Terms in italics are descriptive and were not tracked. Name | Account | Hashtag
---|---|---
Andre Janones | @AndreJanonesAdv | #Janones70
Ciro Gomes | @cirogomes | #Ciro12
Jose Maria Eymael | @Eymaeloficial | #Eymael27
Felipe Avila | @lfdavilaoficial | #Davila30
Guilherme Boulos | @GuilhermeBoulos | #Boulos50
Jair Bolsonaro | @jairbolsonaro | #Bolsonaro22
Leonardo Pericles | @LeoPericlesUP | #Pericles80
Luciano Bivar | @bivaroficial | #Bivar44
Luiz Inacio Lula da Silva | @LulaOficial | #Lula13
Pablo Marçal | @pablomarcal | #Marcal90
Simone Tebet | @simonetebetbr | #Tebet15
Sofia Manzano | @SofiaManzanoPCB | #Manzano21
Vera Lucia | @verapstu | #Vera16
Superior Electoral Court | @TSEjusbr | #Eleiçoes2022
Supreme Court | @STF_oficial | —
Table 3:  Tracked Political Parties (2022 Cycle) — This table lists the Twitter accounts of Brazilian political parties tracked in the data collection from July 1, 2022, to March 14, 2023, with additional information available on the TSE website Court (2023).  Party | Account
---|---
Brazilian Democratic Movement | @MDB_Nacional
Brazilian Labour Party | @ptb14
Democratic Labour Party | @PDT_Nacional
Workers Party | @ptbrasil
Comunist Party of Brazil | @PCdoB_Oficial
Brazilian Socialist Party | @PSBNacional40
Brazilian Social Democracy Party | @PSDBoficial
Christian Social Party | @pscnacional
Citizenship | @23cidadania
Green Party | @partidoverde
Forward | @somosavante70
Progressives | @Progressistas11
Unified Socialist Workers Party | @pstu
Brazilian Comunist Party | @PCBpartidao
Brazilian Labour Renewal Party | @prtboficial
Party of the Worker’s Cause | @PCO29
We Can | @podemos19
Republicans | @republicanos10
Socialism and Freedom Party | @psol50
Liberal Party | @plnacional_
Democratic Social Party | @PSD_55
Social Order Republican Party | @prosnacional
Solidarity | @solidariedadeBR
New Party | @partidonovo30
Sustainability Network | @REDE_18
Popular Unit | @UP80BR
United Brazil | @uniaobrasil44

## Acknowledgements

I would like to express my gratitude for the valuable insights and fruitful
conversations with Filippo Menczer and Alessandro Flammini during the early
stages of this investigation in 2018-19 at Indiana University Bloomington.
Their guidance and expertise significantly contributed to the development of
this research. I am also indebted to Marcos Oliveira at the University of
Exeter for his thoughtful comments and reviews of the manuscript, which
greatly enhanced the quality of this work.

## References

  * (1)
  * Aranha and van de Graaf (2018) Diego F Aranha and Jeroen van de Graaf. 2018.  The Good, the Bad, and the Ugly: Two decades of E-voting in Brazil.  _IEEE Security & Privacy_ 16, 6 (2018), 22–30.
  * Arnaudo (2017) Dan Arnaudo. 2017\.  Computational propaganda in Brazil: Social bots during elections.  (2017).
  * Avgerou et al. (2009) Chrisanthi Avgerou, Andrea Ganzaroli, Angeliki Poulymenakou, and Nicolau Reinhard. 2009.  Interpreting the trustworthiness of government mediated by information and communication technology: Lessons from electronic voting in Brazil.  _Information technology for development_ 15, 2 (2009), 133–148.
  * Badawy et al. (2018) Adam Badawy, Emilio Ferrara, and Kristina Lerman. 2018\.  Analyzing the digital traces of political manipulation: The 2016 Russian interference Twitter campaign. In _Proceedings of IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_. 258–265.  https://doi.org/10.1109/ASONAM.2018.8508646
  * Badawy et al. (2019) Adam Badawy, Kristina Lerman, and Emilio Ferrara. 2019\.  Who falls for online political manipulation?. In _Companion proceedings of the 2019 world wide web conference_. 162–168.
  * Bessi and Ferrara (2016) Alessandro Bessi and Emilio Ferrara. 2016.  Social bots distort the 2016 US Presidential election online discussion.  _First monday_ 21, 11-7 (2016).
  * Check (2022) Reuters Fact Check. 2022\.  Fact Check-Bolsonaro did not ‘annul’ Brazil’s election and the military did not find evidence of ‘fraud’.  https://www.reuters.com/article/factcheck-bolsonaro-fraud-idUSL1N3341F4
  * Chen et al. (2021) Wen Chen, Diogo Pacheco, Kai-Cheng Yang, and Filippo Menczer. 2021\.  Neutral bots probe political bias on social media.  _Nature communications_ 12, 1 (2021), 5580.
  * Chowdhury et al. (2020) Farhan Asif Chowdhury, Lawrence Allen, Mohammad Yousuf, and Abdullah Mueen. 2020.  On Twitter purge: a retrospective analysis of suspended users. In _Companion proceedings of the web conference 2020_. 371–378.
  * Court (2023) Superior Electoral Court. 2023\.  Partidos políticos registrados no TSE. https://www.tse.jus.br/partidos/partidos-registrados-no-tse.
  * Couto et al. (2022) Joao MM Couto, Julio CS Reis, Ítalo Cunha, Leandro Araújo, and Fabrício Benevenuto. 2022\.  Characterizing Low Credibility Websites in Brazil through Computer Networking Attributes. In _2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)_. IEEE, 42–46.
  * Cresci et al. (2023) Stefano Cresci, Roberto Di Pietro, Angelo Spognardi, Maurizio Tesconi, and Marinella Petrocchi. 2023\.  Demystifying Misconceptions in Social Bots Research.  _arXiv preprint arXiv:2303.17251_ (2023).
  * Eger and Valdez (2015) Maureen A. Eger and Sarah Valdez. 2015.  Neo-nationalism in Western Europe.  _Europ. Sociological Rev._ 31, 1 (2015), 115–130.  https://doi.org/10.1093/esr/jcu087
  * Ezzeddine et al. (2023) Fatima Ezzeddine, Omran Ayoub, Silvia Giordano, Gianluca Nogara, Ihab Sbeity, Emilio Ferrara, and Luca Luceri. 2023.  Exposing influence campaigns in the age of LLMs: a behavioral-based AI approach to detecting state-sponsored trolls.  _EPJ Data Science_ 12, 1 (2023), 1–21.
  * Ferrara (2017) Emilio Ferrara. 2017\.  Disinformation and social bot operations in the run up to the 2017 French presidential election.  _arXiv preprint arXiv:1707.00086_ (2017).
  * Ferreira Dos Santos et al. (2019) Eric Ferreira Dos Santos, Danilo Carvalho, Livia Ruback, and Jonice Oliveira. 2019.  Uncovering social media bots: a transparency-focused approach. In _Companion Proceedings of The 2019 World Wide Web Conference_. 545–552.
  * Green (2022) James N Green. 2022\.  Brazilian Democracy in the Balance: Ahead of the October 2 presidential election, Jair Bolsonaro continues to stoke expectations of fraud, raising fears of a possible January 6-style attack.  _NACLA Report on the Americas_ 54, 3 (2022), 253–257.
  * Hays et al. (2023) Chris Hays, Zachary Schutzman, Manish Raghavan, Erin Walk, and Philipp Zimmer. 2023.  Simplistic Collection and Labeling Practices Limit the Utility of Benchmark Datasets for Twitter Bot Detection. In _Proceedings of the ACM Web Conference 2023_. 3660–3669.
  * Jones (2019) Marc Owen Jones. 2019\.  The gulf information war— propaganda, fake news, and fake trends: The weaponization of twitter bots in the gulf crisis.  _International journal of communication_ 13 (2019), 27.
  * Keller and Klinger (2019) Tobias R Keller and Ulrike Klinger. 2019.  Social bots in election campaigns: Theoretical, empirical, and methodological implications.  _Political Communication_ 36, 1 (2019), 171–189.
  * Knight (2022) Will Knight. 2022\.  Why it’s so hard to count twitter bots?  https://www.wired.com/story/twitter-musk-bots/
  * Luceri et al. (2019a) Luca Luceri, Ashok Deb, Adam Badawy, and Emilio Ferrara. 2019a.  Red bots do it better: Comparative analysis of social bot partisan behavior. In _Companion proceedings of the 2019 world wide web conference_. 1007–1012.
  * Luceri et al. (2019b) Luca Luceri, Ashok Deb, Silvia Giordano, and Emilio Ferrara. 2019b.  Evolution of bot and human behavior during elections.  _First Monday_ (2019).
  * Machado et al. (2019) Caio Machado, Beatriz Kira, Vidya Narayanan, Bence Kollanyi, and Philip Howard. 2019.  A Study of Misinformation in WhatsApp groups with a focus on the Brazilian Presidential Elections.. In _Companion proceedings of the 2019 World Wide Web conference_. 1013–1019.
  * Mariconti et al. (2017) Enrico Mariconti, Jeremiah Onaolapo, Syed Sharique Ahmad, Nicolas Nikiforou, Manuel Egele, Nick Nikiforakis, and Gianluca Stringhini. 2017\.  What’s in a Name? Understanding Profile Name Reuse on Twitter. In _Proceedings of the 26th International Conference on World Wide Web_. 1161–1170.
  * Mazza et al. (2022) Michele Mazza, Marco Avvenuti, Stefano Cresci, and Maurizio Tesconi. 2022.  Investigating the difference between trolls, social bots, and humans on Twitter.  _Computer Communications_ 196 (2022), 23–36.
  * Mbona and Eloff (2022) Innocent Mbona and Jan HP Eloff. 2022.  Feature selection using Benford’s law to support detection of malicious social media bots.  _Information Sciences_ 582 (2022), 369–381.
  * Nasim et al. (2018) Mehwish Nasim, Andrew Nguyen, Nick Lothian, Robert Cope, and Lewis Mitchell. 2018.  Real-time detection of content polluters in partially observable Twitter networks. In _Companion Proceedings of the The Web Conference 2018_. 1331–1339.
  * Ndlela (2020) Martin N Ndlela. 2020\.  Social media algorithms, bots and elections in Africa.  _Social media and elections in Africa, Volume 1: Theoretical perspectives and election campaigns_ (2020), 13–37.
  * neeljai ([n. d.) ]neeljai neeljai. [n. d.].  Is there a bug or is this account creation date for real? joined January 1970.  https://www.reddit.com/r/Twitter/comments/q8f0ng/is_there_a_bug_or_is_this_account_creation_date/
  * Pacheco et al. (2021) Diogo Pacheco, Pik-Mai Hui, Christopher Torres-Lugo, Bao Tran Truong, Alessandro Flammini, and Filippo Menczer. 2021.  Uncovering coordinated networks on social media: methods and case studies. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 15. 455–466.
  * Pastor-Galindo et al. (2020) Javier Pastor-Galindo, Mattia Zago, Pantaleone Nespoli, Sergio López Bernal, Alberto Huertas Celdrán, Manuel Gil Pérez, José A Ruipérez-Valiente, Gregorio Martínez Pérez, and Félix Gómez Mármol. 2020\.  Spotting political social bots in Twitter: A use case of the 2019 Spanish general election.  _IEEE Transactions on Network and Service Management_ 17, 4 (2020), 2156–2170.
  * Pierri et al. (2020) Francesco Pierri, Alessandro Artoni, and Stefano Ceri. 2020\.  Investigating Italian disinformation spreading on Twitter in the context of 2019 European elections.  _PloS one_ 15, 1 (2020), e0227821.
  * Pozzana and Ferrara (2020) Iacopo Pozzana and Emilio Ferrara. 2020.  Measuring bot and human behavioral dynamics.  _Frontiers in Physics_ (2020), 125.
  * Radicioni et al. (2021) Tommaso Radicioni, Fabio Saracco, Elena Pavan, and Tiziano Squartini. 2021.  Analysing Twitter semantic networks: the case of 2018 Italian elections.  _Scientific Reports_ 11, 1 (2021), 13207.
  * Reis et al. (2020) Julio CS Reis, Philipe Melo, Kiran Garimella, Jussara M Almeida, Dean Eckles, and Fabrício Benevenuto. 2020.  A dataset of fact-checked images shared on whatsapp during the brazilian and indian elections. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 14. 903–908.
  * Ricard and Medeiros (2020) Julie Ricard and Juliano Medeiros. 2020.  Using misinformation as a political weapon: COVID-19 and Bolsonaro in Brazil.  _Harvard Kennedy School Misinformation Review_ 1, 3 (2020).
  * Rossini et al. (2023) Patrícia Rossini, Camila Mont’Alverne, and Antonis Kalogeropoulos. 2023.  Explaining beliefs in electoral misinformation in the 2022 Brazilian election: The role of ideology, political trust, social media, and messaging apps.  _Harvard Kennedy School Misinformation Review_ 4, 3 (2023).
  * Savarese and Jeantet (2023) Mauricio Savarese and Diane Jeantet. 2023.  Brazil’s Jair Bolsonaro is barred from running for office until 2030.  https://apnews.com/article/brazil-bolsonaro-ineligible-court-ruling-vote-99dee0fe4b529019ccbb65c9636a9045
  * Silva and Benevenuto (2021) Márcio Silva and Fabrício Benevenuto. 2021.  COVID-19 ads as political weapon. In _Proceedings of the 36th Annual ACM Symposium on Applied Computing_. 1705–1710.
  * Takacs and McCulloh (2019) Richard Takacs and Ian McCulloh. 2019.  Dormant bots in social media: Twitter and the 2018 US senate election. In _Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_. 796–800.
  * Tardelli et al. (2020) Serena Tardelli, Marco Avvenuti, Maurizio Tesconi, and Stefano Cresci. 2020.  Characterizing social bots spreading financial disinformation. In _International conference on human-computer interaction_. Springer, 376–392.
  * Torres-Lugo et al. (2022) Christopher Torres-Lugo, Manita Pote, Alexander C Nwala, and Filippo Menczer. 2022.  Manipulating Twitter Through Deletions. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 16. 1029–1039.
  * Uyheng and Carley (2021) Joshua Uyheng and Kathleen M Carley. 2021.  Computational analysis of bot activity in the Asia-Pacific: A comparative study of four national elections. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 15. 727–738.
  * Varol et al. (2017) Onur Varol, Emilio Ferrara, Clayton Davis, Filippo Menczer, and Alessandro Flammini. 2017\.  Online human-bot interactions: Detection, estimation, and characterization. In _Proceedings of the international AAAI conference on web and social media_ , Vol. 11. 280–289.
  * Vosoughi et al. (2018) Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018.  The spread of true and false news online.  _Science_ 359, 6380 (2018), 1146–1151.  https://doi.org/10.1126/science.aap9559
  * Yang et al. (2019) Kai Cheng Yang, Pik Mai Hui, and Filippo Menczer. 2019\.  Bot Electioneering Volume: Visualizing social bot activity during elections. In _The Web Conference Companion_. 214–217.  https://doi.org/10.1145/3308560.3316499
  * Yang and Menczer (2023) Kai-Cheng Yang and Filippo Menczer. 2023.  Anatomy of an AI-powered malicious social botnet.  _arXiv preprint arXiv:2307.16336_ (2023).
  * Yang et al. (2020) Kai-Cheng Yang, Onur Varol, Pik-Mai Hui, and Filippo Menczer. 2020.  Scalable and Generalizable Social Bot Detection through Data Selection. In _Proc. 34th AAAI Conf. on Artificial Intelligence (AAAI)_.
  * Zucco Jr and Nicolau (2016) Cesar Zucco Jr and Jairo M Nicolau. 2016.  Trading old errors for new errors? The impact of electronic voting technology on party label votes in Brazil.  _Electoral Studies_ 43 (2016), 10–20.


Empirical Software Engineering Syful Islamnnaist Raula Gaikovina Kulannaist
Christoph TreudenUM Bodin Chinthanetnnaist Takashi Ishiomnaist1301152 Kenichi
MATSUMOTOfnaist8925358 [naist]The authors are with the Graduate School of
Science and Technology, Nara Institute of Science and Technology, Ikoma-shi,
630-0192, Japan. [UM]The author is with the University of Melbourne,
Australia. 11 11

# The Brokers in a Dependency Ecosystem: An Empirical Study on using Package
Managers

(2015; 2015)

###### keywords:

Package managers, npm, Stack Overflow

abstract

##  1 Introduction

Package management is crucial to most technology stacks in software
development, especially when building a web or mobile application. Using a
package as a third-party dependency is prominent, with more than 5 million
open source packages available via PMs***According to www.libraries.io.. In
2020, GitHub showed its support for third-party package usage when it acquired
the Node.js PM (i.e., npm), which serves over 1.3 million packages to roughly
12 million end-users, and is constantly growing each day [Web:npmStat].

Figure 1: An example of a PM-related issue that provides an exemplary overview
of the elements analyzed in our study. RQ1 and RQ2 analyze the issue type and
underlying cause from the PM-related post text. Finally, RQ3 and RQ4 analyze
the relationship between the PM-related question and the accepted answer
posts.

A PM acts as an intermediary broker between an application and a third-party
package dependency, to ensure that a verified package is correctly installed,
configured, or removed from an application. PMs were developed as an automated
solution to deal with ‘dependency hell’†††A term made popular by this blog
https://web.archive.org/web/20150708101023/http://archive09.linux.com/feature/155922,
i.e., to avoid compatibility and build issues that arise when an application
adopts numerous dependencies. Diversity in technology stacks has led to a
variety of PMs with different features. For instance, npm brokers packages
that run in the node.js based environment and are written in JavaScript.
Recent studies have mainly investigated dependency management and migration
[Cox-ICSE2015, bogart2016break, kula2018developers, decan2018impact,
decan2019empirical]. While these studies have shown that end-users struggle to
migrate their dependent packages, the common assumption is that PMs broker
dependencies without any issues.

In our previous work [islam2021contrasting], we explored thirteen PMs to
understand whether their package management features correlate with the
experience of their end-users. As part of our previous work, we explore in-
depth the issues end-users face when using PMs and the information needed to
resolve them. Since it is common practice for end-users to use Stack Overflow
(SO) to post questions about issues faced during development, we leverage this
data source for our work. In our empirical study, first we conduct an in-depth
qualitative analysis on 1,131 sample question posts on three PMs, i.e., Maven,
npm, and NuGet to characterize PM issues in term of issue types and their
underlying causes. Next, we perform a mixed method analysis to identify the
information needed to resolve PM issues. Throughout the paper, we used the
words ‘PM Issue’ and ‘PM Question’ interchangeably.

Using Fig. 1 as an exemplary overview of the study, we divide our empirical
study into two parts with four research questions:

  1. 1.

PM Issue Characterization. We take a qualitative approach to characterize the
PM issue types and their underlying causes. The guiding research questions are
as follows:

     * •

RQ1: What types of PM issues do end-users face? Using the question coding
schema of Treude et al. [treude2011programmers], we classify each PM-related
issue. As shown in Fig. 1, we see that the PM question in the example is a
how-to type issue.

     * •

RQ2: What are the underlying causes of PM issues? By reading each PM-related
issue, we are able to identify and categorize common underlying causes which
explain why end-users struggle with using a PM. As shown in Fig. 1, an example
underlying cause could be related to lack of technical depth on general ideas
of dependency practice.

We find that end-users struggle to understand intuitive instructions when they
encounter PM-related issues. The most frequently reported issue types were
how-to guides and error messages. As it turns out, the underlying cause lies
with the PM tool usage rather than an update of a package dependency.

  2. 2.

Information Needed to Resolve PM Issues. We perform a mixed method analysis to
extract useful information needed by end-users to resolve PM issues. As shown
in Fig. 1, we determine whether there are common post attributes that are
associated with resolving a PM-related issues. The guiding research questions
are as follows:

     * •

RQ3: What type of PM issues get resolved?Using the classifications from RQ1
and RQ2, we aim to explore whether the issue types and their underlying cause
influence how an issue will get resolved. As shown in Fig. 1, the PM-related
issue in the example was resolved since it received an accepted answer.

     * •

RQ4: What information is needed to resolve PM issues? Our goal is to identify
common information patterns between question and accepted answer posts to
determine if they relate to a resolved PM issues. As shown in Fig. 1, the
build configuration file is identified as a PM-related post attribute which
contributes to the issue resolution.

We find that PM issues are not getting resolved, with only 40% to 49% having
an accepted answer. For the PM issue resolution, our findings reveal essential
insights into the information needed. We find that end-users post external
links to resolve PM issues.

The rest of the paper is organized as follows. Section 2 describes the dataset
building for the target PMs. Section 3 presents PM issue characterization.
Section 4 presents information needed to resolve PM issues. Section 5 presents
the implication of this work. Section 7 presents the related works. Finally,
section 8 presents the conclusion of this work.

##  2 Dataset building for the target PMs: Maven, npm, and NuGet

Initially, we downloaded the Stack Overflow data dump published on SOTorrent
[DBLP:conf/msr/BaltesDT008]. After parsing the Stack Overflow data dump, the
initial dataset had 39.83% (18,699,426) question posts, and 60.17%
(28,248,207) answer posts from July 2008 to December 2019. Since Stack
Overflow does not provide any fine grained dataset, we needed a way to
identify related posts for the target PMs. To accomplish this goal, we
utilized tag-based question post filtering which was also used by prior
studies [abdellatif2020challenges, rosen2016mobile]. In detail, the PM-related
question-and-answer dataset building was performed through three distinct
steps:

  * •

Step 1: Filter using #package-managers tag. In Step 1, our intention was to
filter posts that are only related to selected 3 PMs i.e., Maven, npm, and
NuGet. To accomplish this, we started with collecting question posts that were
tagged with the keyword #package-managers, which is described in Stack
Overflow, as software that allows administrators (and in some cases also
users) to control the installation and upgrade process of packages on their
systems. The output of Step 1 was 806 question posts.

  * •

Step 2: Discover relevant tags. In Step 2, I extracted a list of further
relevant tags from the 806 question posts (Step 1). A potential risk of
expanding the list of tags was the possibility of introducing noise. For
example, “NPM search remote packages” is a relevant post that contains broad
tags such as javascript and node.js. I used a semi-automatic method to
mitigate this issue and manually removed tags that are too broad. I along with
another authors manually checked 626 unique tags. We observed that some
tags—although having low frequency (i.e., occur only one time) in the initial
posts dataset—were highly related to the 3 PMs. For instance, maven-2 is a tag
that occurs only one time as a co-existing tag in the initial PM post dataset,
but the tag itself is associated with many more posts (5,568 question posts).
The output of Step 2 was a list of 11 relevant tags for the selected 3 PMs as
shown in Table 1.

  * •

Step 3: Collect relevant posts and prepare sample. In Step 3, I used the list
of relevant tags from Step 2 to collect final posts. The output of Step 3 was
a cleaned PM-related question-and-answer dataset with 114,835 questions and
50,696 were accepted answer posts. Afterward, we created representative
sample, maintaining 95% confidence level and a confidence interval of 5 for
each PM. This resulted in a total of 1131 question samples from three package
ecosystem (i.e., Maven:382, npm:379, and NuGet:370 posts). Table 2 shows the
summary of sample posts for PM issue characterization and information needed
to resolve them.

Table 1: List of the 11 manually validated tags discovered from Step 2 of the data preparation. Initial tag |  Identified relevant tags
---|---
package-managers |  npm, nuget, nuget-package, maven, npm-install, npm-scripts, npmignore, pnpm, npm-shrinkwrap, nuget-package-restore, maven-2
Table 2: Prepared PM posts Sample summary. The question samples were created as 95% confidence levels with a 5% interval, and the accepted answers analyzed were associated with the question samples. PM | #Question | # Accepted answers | # Sample questions |  | # Accepted answers
---
of sample questions
Maven | 74,657 | 33,710 | 382 | 173
npm | 30,137 | 11,857 | 379 | 151
NuGet | 10,041 | 5,129 | 370 | 182
Total | 114,835 | 50,693 | 1,131 | 506

##  3 PM Issue Characterization

In this section, We have conducted qualitative analysis of PM issues to get
deeper understanding of the issue types and their underlying causes. To
accomplish the goals, two research questions are formulated. In the following,
we have explained each research approach, and the results.

###  3.1 PM Issue types faced by End-users ($RQ_{1}$)

Approach.  To characterize PM issue types, we performed classification of
sample question posts based on the coding scheme proposed by Treude et al.
[treude2011programmers]. Details of the question coding scheme are described
below:

  * •

How-to: Posts that ask for instructions. For example: “Title: How to get
latest version number of an artifact and replace it in target file? (Id:
26223226)” is a how-to type question asking for instruction to get latest
version number of an artifact and replace it in target file.

  * •

Discrepancy: Some unexpected behavior that the person asking about the PM post
wants explained. For example: “Title: Spring Boot JPA & H2 Records Not
Persisted (Id: 27843682)” is a discrepancy question asked for solution on
persistence issues between spring boot JPA & H2 records.

  * •

Environment: Posts about the environment either during development or after
deployment. For example, “NPM Windows Path Problems (Id: 25120982)” is a
question related to package management environment setting.

  * •

Error: Posts that include a specific error message. For example: “Title: Error
running Google App Engine quick start : POM for com.google.app engine:app
engine-maven-plugin:jar:1.9.24 is missing (Id: 31576681)” is question asking
for solution of a general error message related to google app engine.

  * •

Decision help: Asking for an opinion. For example: “Can I invoke a local bean
into a ear file from a Javax-WS into a war file- apache-tomee-plus-1.7.4 (Id:
51072906)” is question related to decision help.

  * •

Conceptual: Posts that are abstract and do not have a concrete use case. For
example: “Difference between mvn appengine:update and mvn appengine:deploy in
Google App Engine (Id: 40094090)” is a conceptual question.

  * •

Review: Posts that are either implicitly or explicitly asking for a review.
For example: “Is there any possibility of deleting libraries stored maven
central? (Id: 25133985)” is a question where end-user asked for a review on
the possibility of deleting libraries stored maven central.

  * •

Non-functional: Posts about non-functional requirements such as performance or
memory usage. For example: “Java project runs slow from JAR but fast from IDE
(Id: 41861330)” is a question asked for solution on performance of Java
project.

  * •

Novice: Often explicitly states that the person belong PM posts is a novice.
For example: “The mssing package org.spring framework.web (Id: 12603723)” is a
novice question according to the description in the question body.

  * •

Novice/How to: Posts that belong to a novice asking for step by step
tutorials. For example: “Maven2 - POM configure issue in Windows (Id:
5087296)” is novice/how-to question according to the description of the
question body.

To ensure a systematic method and reduce bias in the classification, we
performed a Kappa agreement check using 30 random samples among three authors.
Using the Kappa score calculator [viera2005understanding], we checked the
agreement level and find score 86.67%, which was almost perfect. Confident
with the agreement, two authors then continued to classify all samples through
manual analysis of question title and body.

Figure 2: Percentage of PM issues by question coding compared to Treude et al.
[treude2011programmers]. Result shows that ‘How-to’ and ‘Error’ messages are
the most dominant issue faced by end-users.

Results.  Figure 2, shows that most of the PM posts belong to How-to followed
by Error message for three package ecosystems, confirming that end-users are
suffering from lack of instructions and intuitive error messages during third-
party package management. In Maven, we find that How-to (39.01%) question is
most dominant, followed by Error (26.96%) and Review (14.40%). Similar trend
is also shared by NuGet, where How-to (42.16%) question is most dominant,
followed by Error (30.54%) and Decision help (7.30%). On the other hand, in
npm we find that Error (36.94%) message question is most dominant, followed by
How-to (30.87%) and Review (9.76%).

In comparison of PM posts with the most generic results of Treude et al.
[treude2011programmers], we find that How-to question is most dominant while
Error message question is ranked as being the top type of question asked for
end-users of the npm ecosystem.

Takeaway 1: The most dominant PM issues types are how-to (i.e., around 31 to
42% of question samples) and error messages (i.e., around 27 to 37% of
questions). End-users of Maven and NuGet asked the most how-to type issues
while npm end-users ask mostly error type issues.

###  3.2 Underlying Causes of PM Issues ($RQ_{2}$)

Approach.  To identify the underlying causes of PM issues, we conducted a
qualitative analysis of question post body. Similar to other work in software
engineering by Hata et al. [HataICSE2019], we adopted an open coding strategy.
First, two authors independently sampled 30 questions to establish an initial
set of causes. We then added another 30 samples to make sure that no new
causes appeared. To measure author agreement, we used the Kappa score
[viera2005understanding]. After two rounds manual coding of 30 samples with
three authors, we ended up with a Kappa score of 95.56% (almost perfect). Two
author then coded the rest of the classification of third-party package posts.
The four distinct underlying causes for third-party package posts are
summarized below:

  * •

Specific migration: Question posts related to dependency updates like an
upgrade or downgrade to a specific version of the package, moving to new
environment (language/OS/etc), incompatibility of a packages and other
associated issues. Figure 3 shows that a end-user struggle with an issues to
migrate his project written in Java8 to Java9.

Figure 3: Post (Id: 50262939) that discuss PM issues related to specific
migration

  * •

Package management tool usage: Question posts related to technical details on
package management systems such as installation, configuration of tools, and
their associated costs. Figure 4 shows that a end-user discuss about
configuring the maven build tool to run his multi-module project in a custom
way.

Figure 4: Post (Id: 4811870) that discuss PM issues related to package
management tool usage

  * •

General dependency practices: Question posts related to lack of technical
knowledge on general dependency management practice, bugs, efficiency, etc.
Figure 5 shows that, end-users ask a question on the general ideas of maven
packages dependency management.

Figure 5: Post (Id: 30571) that discuss PM issues related to general idea of
dependency practice.

  * •

Others: Posts that are tagged with a PM but do not fall in the above three
categories.

Results.  Figure 6 shows that Package management tool usage is the most
dominant underlying cause for all three package ecosystems, confirming that
end-users tend to report technical issues on Package management tool usage and
not on specific migrations of dependencies. This finding is consistent with
previous studies [nagaria2020software, mangul2019challenges, argelich2008cnf,
decan2017empirical, abate2015mining]. They reported that the complexity of
software tools [nagaria2020software], installability of tools and packages
[mangul2019challenges, decan2017empirical, abate2015mining, argelich2008cnf]
are the root causes of the end-users struggle during application development
and maintenance. In detail, we find that Package management tool usage
(72.25%) is most dominant underlying cause for Maven, followed by General
dependency practices (10.73%). In npm, Package management tool usage (70.71%)
is most dominant underlying cause, followed by General dependency practices
(5.80%). In NuGet, Package management tool usage (63.51%) is most dominant
underlying cause, followed by General dependency practices (18.65%) and
Specific migration (13.51%).

Figure 6: Percentage of PM issues by their underlying cause. I find that
‘Package management tool usage’ is the most dominant underlying cause.

Takeaway 2: The underlying causes of PM issues can be categorized into package
management tool usage, general dependency practices, specific migration, and
others. End-users tend to ask questions that relate to package management tool
usage, and not specific migration.

##  4 Information Needed to Resolve PM Issues

Similar to previous section, we have used the same dataset (i.e., maven, npm,
and Nuget) as shown in Table 2 to analyze the answers of informatio needed to
resolve a PM issue. We now proceed to answer $RQ_{3}$ and $RQ_{4}$.

###  4.1 PM Issues that get resolved ($RQ_{3}$)

Approach.  To answer RQ3, we conducted a qualitative analysis of the PM sample
posts based on the types of issues (RQ1) and underlying causes (RQ2). We
consider a PM issue being resolved if it receive an accepted answer. According
to SO, accepted answer refers to when an answer is acknowledged by the asker
of the question‡‡‡SO Tour: https://stackoverflow.com/tour. Our analysis is to
examine the proportion of posts as a percentage (i.e., acc= $\frac{\\#posts\
with\ accepted\ answer}{\\#all\ posts}*100$) that get resolved by its issue
type and underlying cause.

Table 3: Proportion of PM issues that get resolved by issue types (RQ1) and underlying cause (RQ2). Overall, we find that PM issues are not getting resolved. | Code |  | Maven
---
Acc. (%)
| npm
---
Acc. (%)
| NuGet
---
Acc. (%)
| How-to | 0.43 [linecolor=gray, filledcolor=black]0.43\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.39 [linecolor=gray, filledcolor=black]0.39\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.51 [linecolor=gray, filledcolor=black]0.51\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Error | 0.39 [linecolor=gray, filledcolor=black]0.39\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.36 [linecolor=gray, filledcolor=black]0.36\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.47 [linecolor=gray, filledcolor=black]0.47\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Review | 0.53 [linecolor=gray, filledcolor=black]0.53\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.41 [linecolor=gray, filledcolor=black]0.41\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.50 [linecolor=gray, filledcolor=black]0.50\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Discrepency | 0.46 [linecolor=gray, filledcolor=black]0.46\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.54 [linecolor=gray, filledcolor=black]0.54\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.70 [linecolor=gray, filledcolor=black]0.70\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Decision help | 0.43 [linecolor=gray, filledcolor=black]0.43\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.31 [linecolor=gray, filledcolor=black]0.31\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.41 [linecolor=gray, filledcolor=black]0.41\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
Issue | Conceptual | 1.00 [linecolor=gray, filledcolor=black]1.0\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.65 [linecolor=gray, filledcolor=black]0.65\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.31 [linecolor=gray, filledcolor=black]0.31\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
types | Environment | 0.80 [linecolor=gray, filledcolor=black]0.80\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.40 [linecolor=gray, filledcolor=black]0.40\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.50 [linecolor=gray, filledcolor=black]0.50\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| How-to/Novice | 0.53 [linecolor=gray, filledcolor=black]0.53\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.35 [linecolor=gray, filledcolor=black]0.35\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.67 [linecolor=gray, filledcolor=black]0.67\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Novice | 0.60 [linecolor=gray, filledcolor=black]0.60\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.20 [linecolor=gray, filledcolor=black]0.20\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.00 [linecolor=gray, filledcolor=black]0.00\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Non-functional | 0.50 [linecolor=gray, filledcolor=black]0.50\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.00 [linecolor=gray, filledcolor=black]0.00\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.00 [linecolor=gray, filledcolor=black]0.00\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
Total |  | 0.45 [linecolor=gray, filledcolor=black]0.45\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.40 [linecolor=gray, filledcolor=black]0.40\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.49 [linecolor=gray, filledcolor=black]0.49\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Package management tool usage | 0.46 [linecolor=gray, filledcolor=black]0.46\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.40 [linecolor=gray, filledcolor=black]0.40\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.50 [linecolor=gray, filledcolor=black]0.50\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
Underlying | General practices | 0.44 [linecolor=gray, filledcolor=black]0.44\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.45 [linecolor=gray, filledcolor=black]0.45\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.43 [linecolor=gray, filledcolor=black]0.43\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
causes | Specific migration | 0.44 [linecolor=gray, filledcolor=black]0.44\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.50 [linecolor=gray, filledcolor=black]0.5\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.54 [linecolor=gray, filledcolor=black]0.54\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
| Others | 0.38 [linecolor=gray, filledcolor=black]0.38\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.38 [linecolor=gray, filledcolor=black]0.38\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.44 [linecolor=gray, filledcolor=black]0.44\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}
Total |  | 0.45 [linecolor=gray, filledcolor=black]0.45\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.40 [linecolor=gray, filledcolor=black]0.40\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture} | 0.49 [linecolor=gray, filledcolor=black]0.49\begin{picture}(0.0,0.0)\put(-35.0,1.5){\tiny} \end{picture}

Results. Table 3 shows that most PM issues are not getting resolved. We make
three additional observations. From the table, we can first see that only the
less frequent types (i.e., in Maven question type Conceptual) received all
accepted answers (i.e, 100% accepted answer), with other ecosystems also
sharing similar trend with npm. Second, we find that in npm question type
Conceptual only receive the highest number of accepted answer (65.00%), while
in NuGet question type Discrepancy is ranked as top (69.57% accepted answer
only). Third, investigating resolution in term of underlying cause report that
end-users are struggling more with PM tool usage, with the Maven PM Tool usage
receiving only a 46.01% accepted answer. A similar trend also exists in npm
(39.55%) and NuGet (50.21%) for the underlying cause of PM tool usage.

Takeaway 3: PM issues are not getting resolved, with only 40% to 49% having an
accepted answer.

###  4.2 Information needed to resolve PM issues ($RQ_{4}$)

Approach.  To answer RQ4, we conducted a mixed method analysis to explore
information patterns between a PM questions and its accepted answer. The
process was completed through two consecutive steps i.e., (i) extract
attributes from the PM question and their accepted answers, and (ii) Mine the
attribute patterns. In the following, we have explained each steps.

  * •

Step 1: Extract attributes from the PM question and their accepted answers.
Similar to RQ2, we used two rounds table discussion with 30 samples of manual
coding and arrive at an almost perfect agreement among three authors (i.e.,
overall kappa score for source code 100%, build configuration file 84.62%,
textual content 100%, cmd/log/output files 84.62%, version information 76.92%)
with six attributes. Then, the first author went on to complete annotation of
all 1,131 PM question and their accepted answers using the coding summarized
below:

    * –

Source code: Post that contain code snippets.

    * –

Build configuration file: Post that contain build configuration files. For
instance: pom.xml is a build configuration file for Maven.

    * –

Textual content only: Post that contain only text as information.

    * –

cmd/log/output file: Post that contains only log files or output from the
program. For instance, this will include error messages or log files.

    * –

External link: Post that contain only external links.

    * –

Version information: Post that contain version information of a package or any
software.

After the annotation process, we performed analysis to identify the most
dominant attribute. The output of this step was the most dominant attribute
from the PM question and their accepted answers.

(a) Questions with accepted answer (b) Accepted answers

Figure 7: Attribute analysis between PM question and accepted answer posts.
External link is the most dominant attribute in the accepted answer of PM
issues.

  * •

Step2: Mine the attribute patterns.  To identify attribute patterns, we mine
the dominant attribute (i.e., output of step 1) between PM question and their
accepted answers by applying the association rule learning. Here, we used the
whole dataset for three PMs as shown in Table 2. We used the apriori algorithm
as implemented in the python package
mlxtend§§§http://rasbt.github.io/mlxtend/api_subpackages/mlxtend.frequent_patterns/.
The attribute is treated as a transaction containing two items: attribute in
the PM question and attribute in the accepted answer post. We then use the
absolute value of 5 as threshold for support and 0.45 as threshold for
confidence, i.e., all rules that we extracted are supported by at least 5 data
points and we have at least a 45% confidence that the left hand side of the
rule implies the right hand side. The output of this step is most significant
rules, based on the 5 absolute support and confidence threshold (50%).

Table 4: Aggregated association rules for the external link attribute extracted from PM questions (Q_domain) and their accepted answers (A_domain). We find that links to GitHub or to i.stack.imgur.com have the highest confidence and support. | Association rule | Confidence | Support
---|---|---|---
| Q_maven.apache.org$\,\to\,$A_maven.apache.org | 49.67% | 300
| Q_maven.apache.org, Q_i.stack.imgur.com$\,\to\,$A_maven.apache.org | 56.00% | 14
| Q_code.google.com, Q_stackoverflow.com$\,\to\,$A_github.com | 50.00% | 9
Maven | Q_docs.gradle.org$\,\to\,$A_docs.gradle.org | 50.00% | 7
| Q_www.oracle.com$\,\to\,$A_stackoverflow.com | 53.85% | 7
| Q_www.jfrog.com$\,\to\,$A_www.jfrog.com | 58.33% | 7
| Q_code.google.com, Q_github.com$\,\to\,$A_github.com | 50.00% | 6
| Q_github.com$\,\to\,$A_github.com | 51.73% | 359
| Q_www.npmjs.com, Q_github.com$\,\to\,$A_github.com | 58.62% | 34
| Q_i.stack.imgur.com, Q_github.com$\,\to\,$A_github.com | 52.83% | 28
| Q_gist.github.com$\,\to\,$A_github.com | 51.72% | 15
| Q_npmjs.org$\,\to\,$A_github.com | 53.85% | 14
| Q_pastebin.com$\,\to\,$A_github.com | 58.33% | 14
npm | Q_www.npmjs.org$\,\to\,$A_github.com | 60.00% | 12
| Q_medium.com$\,\to\,$A_github.com | 66.67% | 10
| Q_yarnpkg.com$\,\to\,$A_github.com | 50.00% | 7
| Q_nodejs.org, Q_github.com$\,\to\,$A_github.com | 75.00% | 6
| Q_www.typescriptlang.org$\,\to\,$A_github.com | 62.50% | 5
| Q_yeoman.io$\,\to\,$A_github.com | 62.50% | 5
| Q_registry.npmjs.org, Q_github.com$\,\to\,$A_github.com | 62.50% | 5
| Q_ionicframework.com$\,\to\,$A_github.com | 71.43% | 5
| Q_travis-ci.org$\,\to\,$A_github.com | 83.33% | 5
| Q_i.stack.imgur.com, Q_docs.microsoft.com$\,\to\,$A_i.stack.imgur.com | 76.47% | 13
| Q_github.com, Q_www.nuget.org$\,\to\,$A_github.com | 50.00% | 11
NuGet | Q_docs.microsoft.com, Q_stackoverflow.com$\,\to\,$A_i.stack.imgur.com | 50.00% | 8
| Q_i.stack.imgur.com, Q_docs.nuget.org$\,\to\,$A_i.stack.imgur.com | 60.00% | 6
| Q_i.stack.imgur.com, Q_docs.microsoft.com, Q_stackoverflow.com$\,\to\,$A_i.stack.imgur.com | 85.71% | 6

Results. Result shows that, external links are useful source of information to
resolve PM issues. This finding is consistent with several previous studies
[gomez2013study, liu2022exploratory], that also showed how external links
connect to the existing information’s and extend the crowd-sourced knowledge.
In detail, Figure 7(b) shows that the external links (i.e., Maven: 45.09% ,
npm: 44.37%, and NuGet: 42.86%) are the most frequent attribute in the
accepted answers. Taking a closer look on the external links as shown in Table
4, we observe that end-users share official website links of PMs followed by
github and image screenshots as part of information to describe PM issues and
accepted answers. We find that the most significant pattern either contain a
link to the github domain, or to the i.stack.imgur.com. Moreover, we observed
that links that refer to useful resources (i.e., official documentation
websites, tutorial etc.) are the mostly shared. This analysis on both PM
questions and their accepted answers reveal the opportunities for tool support
and strategies to find the information needed for end-users to resolve their
issues while dealing with PM.

Takeaway 4: External link sharing is the most common practice to resolve PM
issues. We noticed that links that refer to useful resources (i.e., official
documentation websites, tutorial etc.) are the mostly shared.

##  5 Discussions

We believe that the results of this study can assist the PM community in
better focusing their efforts on the most pressing issues in PM usage. In the
following paragraphs, we describe how the results can be used to better guide
PM end users, PM designers, and researchers.

In terms of PM issues characterisation, we find that most issues are raised by
end-users due to a lack of instructions and error messages from PM tools. In
light of these findings, it is necessary to better understand what technical
background knowledge end users should have concerning PMs. Additionally, PM
Designers need to make it easy for end-users to find the information they need
to resolve problems, for example, by providing good error messages. They
should try to be proactive about issues frequently encountered by end-users,
by providing comprehensive documentation and/or making improvements to PMs
where possible. PM designers can, for example, examine tooling policies for
providing appropriate error messages.

Regarding PM issue resolution, we have discovered that sharing external links
appears to be the most common practice among SO users responding to PM issues.
This finding reveals the opportunities for tool support and strategies to find
useful information for addressing issues. Researchers could develop approaches
that could speed up the issue-resolution process, for example, by
automatically capturing the stack trace and logging information.

##  6 Threats to Validity

This section describes the internal, external, and construct validity threats
of this study.

Internal Validity - Threats to internal validity relate to experimental bias
and error in conducting the analysis. The first threat is the accuracy of the
methods used in this study. We perform manual analysis on random sample since
the dataset size is large. To mitigate this challenge, we prepare
representative samples for three package ecosystems, with a confidence level
of 95% and a interval of 5. Thus, we believe that experimental bias and error
in conducting the analysis were reduced.

External Validity - Threats to external validity relate to the
generalizability of findings. In our study, we focused only on Stack Overflow
which is the largest and most popular question-and-answer platform among end-
users. The findings of our study may not generalize to other question-and-
answer platforms. However, our study is consistent with previous works that
also utilized Stack Overflow data [rosen2016mobile, abdellatif2020challenges,
zhang2018code, chatterjee2020finding, zhang2019empirical].

Construct Validity - Threats to construct validity are related to potential
errors that can occur when extracting data about third-party packages related
discussions for different software packaging ecosystems. The first construct
threat is the validity of the collected data. We used Stack Overflow tags to
identify posts related to the third-party package, but some posts may be
incorrectly labeled (i.e., missing tags or incorrect tags). To reduce this
threat, we created the list of tags by following state-of-the-art approaches
[rosen2016mobile, abdellatif2020challenges].

In my qualitative analysis of classifying question types and underlying
causes, the questions may be miscoded due to the subjective nature of our
coding approach. To mitigate this threat, we took a systematic approach to
validate the taxonomy and the comprehension understanding by the three authors
in several rounds. Only until the Kappa score reaches 0.87 and 0.96,
indicating that the agreement is almost perfect (0.81-1.00), we were able to
complete the rest of the sample dataset.

##  7 Related Works

Complementary related works are presented throughout this paper. This section
describes some additional related works.

Dependency management studies. Prior studies on third-party package management
showed that end-users struggled to manage their software dependency
[bogart2016break, bogart2021and, kikas2017structure, decan2017empirical,
decan2019empirical, abate2020dependency, german2007model, lungu2010recovering,
raemaekers2012measuring, teyton2012mining]. In detail, Bogart et al.
[bogart2016break, bogart2021and], performed multiple case study on a set of
software ecosystems with different tooling policies. They found that end-user
practices differ significantly between ecosystems. All ecosystems share values
such as stability and compatibility, but others are different. In addition,
all communities are interested to invest in tooling to facilitate dependencies
maintenance. Kikas et al. [kikas2017structure], analyzed dependency network of
three software ecosystems (i.e., JavaScript, Ruby, and Rust). They reported
that there exist significant difference in dependency network structure across
language ecosystems. Decan et al. [decan2017empirical, decan2019empirical]
studied several software ecosystem and report that dependency network tend to
grow over time in term of size and number of packages. In addition, they
observed that a minority of packages are mostly responsible for the package
updates. Abate et al. [abate2020dependency] studied state-of-art package
managers to investigate dependency solving capacity. They reported that
solving dependency problem is challenging. German et al. [german2007model] and
Lungu et al. [lungu2010recovering] investigated issues related to dependency
graphs and dependency management specifications. They reported that
dependencies also exist between projects in a software ecosystem. Raemaekers
et al. [raemaekers2012measuring] and eyton et al. [teyton2012mining], have
shown that dependency management involves making cost-benefit decisions
related to keeping package dependencies up to date.

Other studies mined and identified patterns in the migration of third-party
packages [kula2018developers, gaikovina2017modeling, kula2018generalized,
mirhosseini2017can, decan2018impact, dietrich2019dependency], not only between
client applications and libraries, but their effect at the ecosystem
level.Kula et al. [kula2018developers], mined end-user responsiveness to
existing security awareness mechanisms on 850K library dependency migrations
from 4,659 GitHub projects. They found that end-users were particularly
reluctant to update third-party libraries to fix vulnerabilities. Decan et al.
[decan2018impact], mined 400 security reports over 6 years on an npm
dependency network containing over 610k JavaScript packages. They report
findings and provide guidelines for package maintainers and tool end-users to
improve the process of addressing security issues . Mirhosseini and Parnin
[mirhosseini2017can], mined 7,470 GitHub projects that use notification
mechanisms to identify changes in upgrade behavior. On average, projects with
pull request notifications were upgraded 1.6 times more often than non-tool
projects, according to their results. Dietrich et al. [dietrich2019dependency]
studied seventeen different package ecosystems. Their findings reveal that
end-users struggle to find a sweet spot between fixed version dependency
predictability and flexible dependency agility.

While these studies have shown that end-users struggle to migrate their
dependent packages, the common assumption is that PMs broker dependencies
without any issues. Our paper studies the issues confronted when using the PM
tool to manage a third-party package.

Stack Overflow Studies. Q&A platform like Stack Overflow has gained huge
research interest, with topics relating to community dynamics, technical
issues of programmers and human factors [meldrum2017crowdsourced]. Several
empirical case studies were performed using Stack Overflow data such as
improving API documentation and usage scenarios [nasehi2012makes,
uddin2020mining, venkatesh2016client], new programming language (Go, Rust, and
Swift) related discussion [chakraborty2021developers], privacy
[tahaei2020understanding], assess technical debt in software project
[gama2020using], Docker development [haque2020challenges], IOT
[uddin2021empirical], code weakness [zhang2021study], programming errors
[chatterjee2020finding], mobile platforms [wang2013empirical], code reuse
[wu2019developers], deep learning framework [han2020programmers], etc. Some
studies were done on human factors like IT skill [montandon2021skills],
programmers expertise [diyanati2020proposed], etc.

Several tool supports and recommendation models were developed using Stack
Overflow data resources such as PostFinder [rubei2020postfinder], bug severity
prediction model [tan2020bug], code2Que [gao2021code2que], Icsd [ye2018icsd],
etc. These studies reported that Stack Overflow data resources are useful to
solve developers challenges.

In this paper, the same data source (i.e, Stack Overflow) is used but
different from the above mentioned empirical studies. To the best of my
knowledge, there is no prior work that conducted study on PM issues from Stack
Overflow. We extracted PM Q&A posts from Stack Overflow and perform a series
of empirical studies. We believe that this studies highlight the challenges
and information needs associated with using PM for end-users, package manager
designers and researchers.

##  8 Conclusions

In this study, we have explored issues faced by end-users when using PMs
through an empirical study of content on SO question and accepted answer
posts. We carried out a qualitative analysis of 1,131 question and their
accepted answer posts from the Maven, npm, and NuGet dependency ecosystems to
identify issue types, underlying causes, and their resolutions. We observe
that most issues arise from the lack of understanding PM tool usage
information rather than specific version updates and compatibility issues. In
terms of issue resolution, our findings reveal an end-user should be conscious
that the choice of PM will impact their user experience. There are many open
avenues in addition to the investigation of trade-offs between design features
of PM, such as:

  * •

tool support for a newbie PM end-users.

  * •

tool support for end-users who is switching PMs.

  * •

intuitive configuration PM options for end-users.

  * •

intuitive error messages through stack traces and log files.

## Acknowledgment

This work has been supported by JSPS KAKENHI Grant Numbers JP8H04094,
JP20K19774, and JP20H05706.

[Authors_Photo/Syful.JPG]Syful IslamHe received the M.E. degree in Information
Science from Nara Institute of Science and Technology, Japan. He is currently
working toward the P.hD degree in the same institute. At present, he is on
study leave from Noakhali Science and Technology University, Bangladesh. His
research interests include software ecosystem, mining Stack Overflow, etc.

[Authors_Photo/Raula.png]Raula Gaikovina Kulais currently an assistant
professor at Nara Institute of Science and technology. In 2013, he graduated
with a PhD. from Nara Institute of Science and Technology, Japan. He is
currently an active member of the IEEE Computer Society and ACM. His research
interests include repository mining, code review, software libraries and
visualizations.

[Authors_Photo/christoph.JPG]Christoph Treude is a Senior Lecturer in Software
Engineering in the School of Computing and Information Systems at the
University of Melbourne. His research combines empirical studies with the
innovation of tools and approaches that take the wide variety of natural
language artefacts in software repositories into account.

[Authors_Photo/Bodin.JPG]Bodin Chinthanet is currently a Specially Appointed
Assistant Professor in Software Engineering Laboratory under the supervision
of Professor Kenichi Matsumoto, Nara Institute of Science and Technology
(NAIST). His research interests include empirical software engineering and
mining software repositories.

[Authors_Photo/ishio.jpg]Takashi Ishioreceived the Ph.D degree in information
science and technology from Osaka University in 2006. He was a JSPS Research
Fellow from 2006-2007. He was an assistant professor at Osaka University from
2007-2017. He is now an associate professor of Nara Institute of Science and
Technology. His research interests include program analysis, program
comprehension, and software reuse. He is a member of the IEEE, ACM, IPSJ and
JSSST.

[Authors_Photo/matsumoto.png]Kenichi Matsumotoreceived the B.E., M.E., and PhD
degrees in Engineering from Osaka University, Japan, in 1985, 1987, 1990,
respectively. Dr. Matsumoto is currently a professor in the Graduate School of
Information Science at Nara Institute Science and Technology, Japan. His
research interests include software measurement and software process. He is a
senior member of the IEEE and a member of the IPSJ and SPM.