# Incorporating Worker Perspectives into MTurk Annotation Practices for NLP

Olivia Huang
UC Berkeley
Eve Fleisig
UC Berkeley
{efleisig, oliviahuang<EMAIL_ADDRESS>
Dan Klein
UC Berkeley

###### Abstract

Current practices regarding data collection for natural language processing on
Amazon Mechanical Turk (MTurk) often rely on a combination of studies on data
quality and heuristics shared among NLP researchers. However, without
considering the perspectives of MTurk workers, these approaches are
susceptible to issues regarding workers’ rights and poor response quality. We
conducted a critical literature review and a survey of MTurk workers aimed at
addressing open questions regarding best practices for fair payment, worker
privacy, data quality, and considering worker incentives. We found that worker
preferences are often at odds with received wisdom among NLP researchers.
Surveyed workers preferred reliable, reasonable payments over uncertain, very
high payments; reported frequently lying on demographic questions; and
expressed frustration at having work rejected with no explanation. We also
found that workers view some quality control methods, such as requiring
minimum response times or Master’s qualifications, as biased and largely
ineffective. Based on the survey results, we provide recommendations on how
future NLP studies may better account for MTurk workers’ experiences in order
to respect workers’ rights and improve data quality.

##  1 Introduction

Amazon Mechanical Turk (MTurk) is an online survey platform that has become
increasingly popular for NLP annotation tasks Callison-Burch and Dredze
(2010). However, data collection on MTurk also runs the risk of gathering
noisy, low-quality responses Snow et al. (2008) or violating workers’ rights
to adequate pay, privacy, and overall treatment Xia et al. (2017); Hara et al.
(2018); Kim et al. (2021); Lease et al. (2013); Shmueli et al. (2021). We
conducted a critical literature review that identifies key concerns in five
areas related to NLP data collection on MTurk and observed MTurk worker
opinions on public forums such as Reddit, then developed a survey that asked
MTurk workers about open questions in the following areas:

#### Task clarity.

How can requesters ensure that workers fully understand a task and complete it
accurately? Surveyed workers expressed a desire for more examples and more
detailed instructions, and nearly half desired more information about the
downstream context in which an annotation will be used. In addition, they
indicated that task clarity is a major factor in their judgment of how
responsible and reliable a requester is (Section 3).

#### Payment.

What is an appropriate level of pay on MTurk? Are there tradeoffs to different
amounts or methods of pay? Respondents indicated that there is often a
threshold pay below which they will refuse to complete a task, suggesting that
pay rates below minimum wage are not only unethical, but also
counterproductive to worker recruitment. However, very high pay attracts
spammers, which moreover forces attentive workers to complete tasks more
quickly before spammers take them all, meaning that response quality likely
decreases across all workers. Workers expressed mixed opinions over bonus
payments, suggesting that they are most effective if they are added to a
reasonable base pay and workers trust the requester to deliver on the bonus.
(Section 4).

#### Privacy.

What methods best ensure worker privacy on MTurk? How do workers react to
perceived privacy violations? Workers report that they often respond to
questions for personal information untruthfully if they are concerned about
privacy violations. Nearly half of workers reported lying on questions about
demographic information, which means that tasks relying on demographic data
collection should be very careful about collecting such information on MTurk
(if at all). Some of these issues can be partially mitigated if requesters
have good reputations among workers and are clear about the purpose of
personal data collection, since spam requesters are common on MTurk. (Section
4).

#### Response quality.

What methods best ensure high-quality responses and minimize spam? Requesters
must consider how traditional response filters have the side effect of
influencing workers’ perception of the task, and in turn, their response
quality. The rising importance of incorporating worker perspectives is
underscored by the establishment of organizations such as Turkopticon, which
consolidates workers’ reviews of requesters and advocates for MTurk workers’
rights, including petitioning to “end the harm that mass rejections cause”
Turkopticon (2022). We find that workers consider some common quality control
methods ineffective and unfair. For example, workers reported that minimum
response times often make them simply wait and do other things until they hit
the minimum time, while allowing only workers with Masters qualifications to
complete a task excludes all workers who joined after 2019, since the
qualification is no longer offered. In addition, workers are very concerned
about their approval rates because requesters often filter for workers with
very high approval rates. Respondents expressed frustration over having work
rejected, especially automatically or without justification, and receiving no
pay for their labor. Workers thus avoid requesters with low approval rates or
poor communication (Section 6).

#### Sensitive content.

Tasks in which workers annotate sensitive content can pose a psychological
risk, particularly since MTurk workers often face mental health issues Arditte
et al. (2016). Though it is common practice to put general content warnings
before such tasks (e.g. “offensive content”), nearly a third of surveyed
workers expressed a desire for specific content warnings (e.g., “homophobia”)
despite this being uncommon in current research practices (Section 7).

Common themes emerge across these topics regarding the importance of
maintaining a good reputation as a requester, understanding worker incentives,
and communicating clearly with workers. In Section 8, we discuss the
implications of worker preferences for survey design and provide
recommendations for future data collection on MTurk.

##  2 Survey Design

We conducted a literature review and examined posts on r/mturk, a Reddit forum
that MTurk workers use to discuss the platform, to identify the areas of
uncertainty discussed in sections 3-7. Then, to collect data on MTurk workers’
experiences, we conducted a Qualtrics survey posted as a task on MTurk. Our
survey was broadly subdivided into sections on payment, sensitive questions,
response quality, and miscellaneous issues (mainly task clarity and context).
As a preliminary filter for response quality, we required that all respondents
have a $97\%$ HIT approval rate, at least 100 HITs completed, minimum of 18
years of age, and English fluency. This is a lower HIT approval rate than is
typically used, which can increase the rate of spam; however, we aimed to
collect opinions of workers who might be excluded by high approval rate
filters, and therefore manually reviewed the data afterwards to remove spam
responses by examining responses to the required free text fields. We
collected 207 responses from our survey over one week, of which 59 responses
were dropped for spam or incompleteness (we discuss further implications of
high spam rates in Section 8). Each annotator was paid $2.50 to complete the
survey, based on the estimated completion time and a minimum wage of $15/hour.
Appendices A and B contain details on informed consent, the full text of the
survey, and data cleaning details.111This study underwent IRB review and
annotators provided informed consent prior to participation.

The following sections address each area of concern raised in Section 1,
discussing previous research on MTurk annotation practices, the open questions
raised by the literature, the survey questions we included on those topics,
and the results and implications of the survey responses.

##  3 Task Clarity

On MTurk, _requesters_ publish “human intelligence tasks” (or _HITs_) for
MTurk workers to complete in exchange for a monetary incentive. Best practices
for question phrasing and ordering are consistent with overall survey design
guidelines, which encourage including clear instructions for how to use the
platform, clearly outlining requirements for acceptance, including label
definitions and examples, and avoiding ambiguous language Gideon (2012). To
look for strategies for minimizing confusion in survey questions, our survey
asked what additional provided information, such as context and purpose of the
task, would be most helpful for reducing ambiguity or confusion.

To ensure that results are accurate and useful, it is important to carefully
conceptualize the task (define what quality is being measured by the study)
and operationalize it (decide how the study will measure the construct defined
by the conceptualization) along with clear compensation details (discussed
further in Section 4 and Jacobs and Wallach, 2021). In a Reddit thread on
r/mturk, worker u/dgrochester55 (2022) stated that “we are not likely to take
your survey seriously if you as a requester aren’t taking your own survey
seriously” due to issues such as “poor pay, irrelevant subject” or being
“badly put together.” Thus, task clarity not only ensures that the worker can
provide accurate answers, but also serves to increase their effort levels and
willingness to complete the task itself.

Our survey sought to further clarify the importance of task clarity by asking
whether it influences effort level and willingness to start a task. In
addition, we asked a free response question about common areas of confusion to
observe common pitfalls in designing MTurk tasks.

###  3.1 Context

One way to improve task clarity is to provide additional context for the task.
Context is commonly provided in two ways. The first way is to provide
additional words, sentences, or paragraphs in the annotated text itself.
Providing context can increase disagreement among responses, and can be useful
if the goal is to mimic the distribution of differing human judgments (Pavlick
and Kwiatkowski, 2019). The second way is to introduce background information
about the speaker, such as age, gender, race, socioeconomic status, etc. In
tasks such as annotating for the presence of harmful language, such
information can affect an annotator’s perception of text as influenced by
individual biases.

Sometimes, context is inherent in the language itself, instead of explicitly
provided. For example, some annotators may incorrectly interpret African
American English (AAE) as offensive (Sap et al., 2019). Other forms of context
include grammaticality, location, and relative positions of power between the
speaker and the audience.

Thus, it is important to note that even without the presence of additional
context, text is inherently contextual. It is important to identify and
control for such qualities in annotation tasks. To better understand how
context can influence annotation tasks, we asked MTurk workers what additional
information is most useful to them when given a question with an unclear
answer.

###  3.2 Survey Results on Task Clarity

Figure 1: Types of information that respondents felt would have helped them
answer questions they found confusing.

We found that task clarity and fair payment are key factors that determine
both MTurk workers’ willingness to start a task and the amount of effort they
put into it. In our survey, $26.3\%$ of respondents stated that difficulty of
understanding the task influences whether they start the task, and $31.6\%$
indicated that it influences the amount of effort they put into the task. One
respondent suggested providing additional, more comprehensive examples, since
requesters “often skimp on these or only provide super basic, obvious
examples.” When respondents were asked to rank the importance of different
qualities in determining how “reasonable” and “reliable” the requester is
(from one to six), clarity of instructions was ranked second highest with an
average rank of 2.71 (payment was ranked highest; see Section 4). These
results reinforce the importance of task clarity in improving workers’
perceptions of the task and its requester.

Workers often reported being confused by annotation tasks that involve
sentiment analysis and subjective ratings. For such questions, the most
commonly preferred additional information was the purpose of the question
($69.1\%$), context ($50.0\%$), and the platform the annotations would be used
for ($34.9\%$). This suggests that providing an explanation for the downstream
use case of the task significantly aids workers in understanding tasks and
providing quality annotations.

##  4 Payment

MTurk’s platform, which enables researchers to collect large amount of data at
low cost, does not currently regulate compensation beyond a $\$0.01$ minimum
per task that disregards task completion time. The mean and median hourly
wages for MTurk workers are $3.13 and $1.77 per hour, respectively. 95% of
MTurk workers earn below minimum wage requirements in their geographic
location (Hara et al., 2018), and improvements in household income for U.S.
MTurk workers lag behind the general U.S. population Jacques and Kristensson
(2019). Low payment is not only unfair to workers, but can also serve as an
additional sign to workers of an unreliable requester to avoid due to high
potential of having a HIT rejected. u/Bermin299 (2022) claims that “Requestors
that pays cheaply are some of the most trigger happy people when it comes to
handing out rejections.”

Most researchers agree on using minimum wage as a baseline hourly rate for
MTurk work, though exact recommendations vary due to differences in the
minimum wage worldwide (Hara et al., 2018). A common sentiment among MTurk
workers is to pay at a rate of $15 per hour (u/Sharpsilverz, 2022). Whiting et
al. (2019) found that workers are likely to overestimate their work time by
38% of the observed task time. Thus, the amount of time used to determine
minimum wage can roughly account for this overestimation if using workers’
self-reported work times as a metric. Previous research is divided on whether
extra pay beyond minimum wage improves response quality. Callison-Burch and
Dredze (2010) gave anecdotal evidence that unusually high payments, such as $1
for a very short task, may encourage cheating.

However, Snow et al. (2008) found that non-guaranteed payments, which are paid
only after submitted work is approved for work quality, are sometimes
effective at improving response quality. MTurk’s bonus payments can also be
non-guaranteed payments, and they are given to workers after they have
completed the task in addition to the advertised payment rate.

Given the wide range of payment options and formats, along with varying
guidelines on the effectiveness of different payment options, we include
multiple questions in our survey regarding MTurk workers’ perception and
response to payments. These included workers’ perception of normal and bonus
payments, how bonus versus normal payments affect effort, how payment
influences likelihood of starting or completing the task, and how payment
affects the degree to which requesters are perceived as “reasonable” and
“reliable.” We also asked MTurk workers about their MTurk income relative to
the opportunity cost of their work, as well as their own thresholds for the
minimum pay rate at which they would do a task.

###  4.1 Survey Results on Payment

Survey results indicate reasonable payment to be around minimum wage;
significantly higher or lower payment appears to be detrimental to the quality
of responses. Respondents’ minimum pay rate at which they would be willing to
complete a task was $13.72 per hour on average (median of $12 per hour).
$64.5\%$ of respondents stated that the rationale behind their threshold is
that they want to earn a wage on MTurk comparable to what they would make
elsewhere. These numbers are significantly above the current mean and median,
though still below minimum wage. Despite this, the survey surprisingly
indicates that $70.4\%$ of respondents make more on MTurk than they would make
elsewhere, and $22.4\%$ make the same amount. However, payment should not be
set too high, as one worker stated that “[e]xtremely high pay rates cause
rushing in order to complete as many as possible prior to the batch getting
wiped out by other workers.”

Workers responded that both higher normal payments and bonus payments increase
the amount of effort they put into a task. $73.0\%$ of respondents indicated
that bonus payments will increase their effort levels while $49.3\%$ selected
high(er) payments. However, some respondents noted that bonus payments are
only a significant incentive if they are from a reputable requester.
Uncertainty about whether the bonus will be paid is the main concern, with one
worker noting that “regular payments are guaranteed within 30 days, whereas
bonuses can take however long the req decides to take to pay you (assuming
they even do).” Thus, the effectiveness of bonus payments varies with workers’
perception of the requester’s reliability and reputation.

##  5 Privacy

Demographic questions are commonly used in surveys to collect respondents’
information. However, some MTurk workers may view these questions as threats
to their privacy and decide not to complete the survey. To clarify which
questions are more commonly viewed as “sensitive,” our survey asked MTurk
workers to indicate types of questions, including demographic questions and
questions about offensive topics or mental health, that cause them to not
complete a task or answer untruthfully. We also asked for reasons would cause
them to answer a question untruthfully, including whether they believe that
the question is a privacy violation or that it will be used against them.

It is especially important to build trust and the perception of fairness
between the requester and the annotator for surveys that ask more personal
questions. Xia et al. (2017) found that crowdworkers’ major areas of concern
regarding data privacy involved collection of sensitive data, information
processing, information dissemination, invasion into private lives, and
deceptive practices. To minimize these concerns, they proposed providing
information about the requester, being specific about why any private
information is collected and how it will be used, not sharing any personal
information with third parties, and avoiding using provided emails for spam
content outside the context of the survey. Turkopticon’s Guidelines for
Academic Requesters recommend that requesters provide information that clearly
identifies who the requester are and how to communicate with them, which
indicates that the requester is “accountable and responsible”, as well as
avoiding collection of workers’ personally identifying information Turkopticon
(2014). Building on this, our survey asked what additional information, such
as content warnings and mental health resources, would be most helpful in
surveys with sensitive questions.

Best practices on demographic questions include ensuring inclusivity for
questions on gender identity by allowing for open ended responses and
considering that terminology changes over time (e.g. “How do you currently
describe your gender identity?”), distinguishing between ethnicity and race,
and permitting annotators to indicate multiple responses for race, ethnicity,
and gender in case they identify with multiple categories (Hughes et al.,
2016). In our survey, we asked MTurk workers if they encountered questions
where the responses provided did not adequately capture their identity.

Another MTurk worker vulnerability is the use of Worker IDs. Each MTurk worker
has an identifying Worker ID attached to their account, associated with all of
their activities on Amazon’s platform, including responses to MTurk tasks and
activity on Amazon.com (Lease et al., 2013). As a result, any requester with
access to a Worker ID (which is automatically included with survey responses)
can identify the associated MTurk worker’s personal information through a
simple online search. This can return private information such as photographs,
full names, and product reviews (Lease et al., 2013). As this is a
vulnerability in MTurk’s system that cannot be removed by the requester, it is
common for requesters to simply drop the Worker IDs from the dataset before
continuing with any data analysis. In our survey, we ask MTurk workers whether
they are aware of the risk of their Worker IDs being attached to MTurk
responses and if that knowledge is a concern that influences their behavior.

###  5.1 Survey Results on Privacy

Figure 2: Types of personal questions that workers report will cause them to
answer untruthfully. Over half of annotators report that questions about age
will make them answer untruthfully, and over 40% report the same about gender.

Our survey results indicate that concerns about privacy often lead to
untruthful answers. Overall, $79.6\%$ of respondents stated they will answer a
question untruthfully if they feel that it is a privacy violation, and
$17.1\%$ stated they will do so if they have concerns that the questions will
be used against them. In regards to Worker IDs, $51.3\%$ of workers stated
that they are aware that they exist, but it has not caused them to change
their behavior.

In addition, workers often feel that demographic questions do not capture
their identity. This is most often an issue for questions about gender
($38.2\%$), age ($38.2\%$), and sexual orientation ($37.5\%$). Workers also
frequently answer demographic questions untruthfully (Figure 2), especially
those regarding age ($52.0\%$), gender ($41.5\%$), and education ($32.2\%$).
$9.87\%$ of respondents indicated that other questions caused them to answer
untruthfully, and most frequently mentioned questions requesting a phone
number, name, or zipcode. Other respondents were concerned that their
education level could be held against them.

##  6 Response Quality

Response quality can be maximized by ensuring that the task is clear and well-
designed through a preliminary pilot task (annotation tasks released before
the actual task to fine-tune the survey) and filtering respondents to include
those who are more likely to honestly complete the task (Gideon, 2012).
Requesters can also work towards maximizing response quality by using various
functionalities provided by MTurk’s filtering software.

###  6.1 MTurk Qualifications

MTurk provides several mechanisms for improving response quality. These
include Masters Qualifications (which filter for “Masters Workers”, selected
on the basis of successful completion of a wide range of MTurk tasks over
time), System Qualifications (e.g. HIT approval rate, location), and Premium
Qualifications (e.g., income, job function). MTurk also has “Qualifications
Types you have created,” which allow requesters to assign specific workers
with a custom score between 0-100 to determine their eligibility for tasks.
One challenge to filtering MTurk workers is balancing the use of filters to
minimize spam responses while still allowing enough real workers to respond in
order to collect a large enough dataset. Filtering on location is common
because it helps to filter out MTurk workers who may be unqualified to
complete tasks that require proficiency in certain languages or experience
with a certain culture or environment (Karpinska et al., 2021).

To prevent overfiltering respondents at the beginning, some papers suggest
keeping numerical thresholds relatively low. Peer et al. (2014) stated that
high-reputation and high-productivity workers can be filtered using 500
approved HITs and 95% HIT approval rate, while a looser lower bound to avoid
low-reputation and low-productivity workers is 90% approval rate and 100
approved HITs. However, more recent research has found that around 2500
approved HITs and 99% HIT approval rate is a more effective filter for MTurk
surveys, since approval rates below that result in a significant drop in
response quality (Kummerfeld, 2021). Meanwhile, researchers such as Ashida and
Komachi (2022) used a 98% HIT approval rate to allow more workers to attempt
HITs. This aligns with views expressed by MTurk workers that a 99% approval
rate is unreasonably difficult to obtain, as it requires a near perfect record
when HITs are often rejected without a logical reason or explanation
(u/Lushangdewww, 2019). Furthermore, a 99% HIT approval rate largely serves to
unfairly discriminate against newer workers (rather than low quality workers),
due to the relative margin of error based on the total number of HITs
completed (u/ptethesen, 2022). Turkopticon encourages providing clear
conditions for rejection due to the permanent mark that rejection leaves on a
worker’s record Turkopticon (2014).

Many MTurk requesters also use time as a means of filtering responses both as
workers are completing tasks (e.g. through timers) and after receiving the
complete dataset, by removing the top and bottom percentile of response times
(Justo et al., 2017). In response to one such task warning that “speeders”
would be rejected, however, an r/mturk user stated that they “did that survey,
made some lunch, checked my text messages then submitted” u/gturker (2022).
Such forum posts suggest that these filters encourage workers to work on other
tasks to pad time rather than spend more time thinking about their answers.

While several papers outline strategies to ensure high response quality, there
is little work on how such measures are received by MTurk workers themselves.
Thus, our survey asked MTurk workers about how strategies such as limiting
response time ranges and adding time limits affect their response quality. In
addition, we asked if any of these measures are perceived as unreasonable.
Lastly, we asked MTurk workers to provide their own input on reasonable checks
for response quality and whether any existing qualifications are perceived as
unfair or unreasonable.

###  6.2 Recent LLM Developments and Concerns

The rise of large language models (LLMs) poses a significant risk to response
integrity. Veselovsky et al. (2023) found that 33 to 46% of workers used LLMs
to complete a text summarization task on MTurk. Furthermore, these responses
may be difficult to detect, as LLMs such as ChatGPT outperform humans on some
annotation tasks (Gilardi et al., 2023). In our survey, we found that
required, open-ended text responses made it easiest to spot spam respondents
who might pass unnoticed in rating or classification tasks, but such manual
methods may become less effective in the face of better text generation tools.
However, multi-stage filtering procedures, such as Zhang et al. (2022)’s
pipeline that uses a qualification task and a longer endurance task to
identify a high-quality worker list before the main task begins, can help to
identify a trustworthy pool of workers to minimize spam responses. AI-
generated text detectors may also help to flag potential spam responses in
these filtering pipelines Veselovsky et al. (2023).

###  6.3 Survey Results on Response Quality

One common metric to filter for response quality is response time. Overall,
respondents indicated that task timers and time estimates are well-calibrated.
However, we also found that time-based filters are largely unhelpful and
counterproductive. When given minimum required response times, $41.5\%$ of
workers reported that they spend the additional time working on other things
to extend their completion time. $22.4\%$ of survey respondents reported
having had a HIT unfairly rejected due to responding too fast. Meanwhile, time
limits can reduce response quantity and quality. One respondent explained that
“it is frustrating when you have put in a great deal of conscientious effort
only to find the time has expired and you cannot submit a HIT.” These results
align closely with the comments made in the Reddit thread described in the
previous section (u/LaughingAllTheWay83, 2022a).

The filter most commonly seen as unreasonable is the Masters qualification
(55.9%), because MTurk has reportedly stopped giving it out. One respondent
explained that “Masters is a dead qual. Hasnt been issued to anyone since
2019. It is the single biggest gatekeeper to success for newer turkers like
myself.” Thus, MTurk requesters who require a Masters qualification
unknowingly filter out all workers that joined after this time.

On average, respondents indicated that a reasonable HIT approval rate is
$93.6\%$, while the median response was $98\%$. Overall, MTurk workers
reported a strong incentive to maintain a high HIT approval rate, as $40.1\%$
of survey respondents stated that the approval rate of the requester
influences the amount of effort they put into their responses and $36.2\%$
state that it influences whether they even start a task. Thus, a good practice
for rejecting HITs is is to provide a clear rationale for why a HIT is being
rejected as feedback for future work.

##  7 Sensitive Content

MTurk surveys involving more sensitive NLP tasks can pose psychological risks.
MTurk workers are more likely to have mental health issues than the general
population, making them a vulnerable population (Arditte et al., 2016). NLP
tasks that pose a potential risk to workers include labeling hate speech
applicable to the worker’s own background, identifying adult or graphic
content, or questions about workers’ mental health. There is usually higher
risk with tasks that involve language that provides strong emotional stimuli
(e.g. offensive tweets), which can potentially cause trauma (Shmueli et al.,
2021). In addition, work may be more personally damaging when the offensive
content directly applies to the worker (Sap et al., 2020). It may be
beneficial to modify studies being completed by targeted demographic groups
with additional safeguards and warnings in the survey regarding potential harm
or offense.

In addition to exposing workers to potential psychological harm, the presence
and framing of sensitive questions can cause workers to leave tasks incomplete
or answer untruthfully. u/LaughingAllTheWay83 (2022b) described providing
“accurate and thoughtful answers 99% of the time, the exception being in
regards to my own mental health…Requesters have been known to overstep their
lane and contact the police for a welfare check and I’m not opening myself up
to that possibility for an extra $2, ya know?” Thus, it is vital for
researchers to identify potentially sensitive content and take steps to
protect workers from possible harm.

Generally, researchers are aware of the presence of offensive content, as
reflected in content warnings included in the beginning of published papers
(e.g., Sap et al., 2020). However, there is a clear need for standardized best
practices due to the highly vulnerable nature of such MTurk work.

A relevant example is Facebook’s partnership with Sama, a company that pays
workers to identify illegal or banned content on social media (Perrigo, 2022).
An interview with the workers revealed high levels of psychological distress
due to insufficient breaks and subpar in-person resources. In an online
setting such as MTurk, such personalized care is even less accessible, making
it more difficult to provide support for vulnerable workers. These workers’
experiences exemplify the difficulties faced by the large, invisible workforce
behind MTurk and powerful NLP applications, also described by Gray and Suri
(2019) in their discussion of the typically unregulated “ghost work” that
powers machine learning.

Common safeguards include listing clear descriptions of the benefits and risks
of the task, providing mental health resources for workers, using data safety
monitoring boards, adhering to institutional review board policies, and
following confidential data code and informed consent procedures (Kim et al.,
2021). We look to further add to this list of best practices by asking survey
respondents whether there are questions that will immediately deter them from
completing a task or to answer untruthfully and preferred actions or resources
provided by requesters to minimize possible harm.

###  7.1 Survey Results on Sensitive Content

Sensitive survey questions, such as questions that ask workers to annotate
harmful or offensive content (e.g. hate speech), not only expose workers to
potential harm, but also often lead to inaccurate or incomplete responses.
When we asked for specific types of questions that cause workers to decide to
not complete a task, $23.7\%$ of respondents indicated offensive content, and
$19.1\%$ indicated questions concerning mental health. As seen in Figure 2,
$20.4\%$ of workers reported answering questions on offensive questions
untruthfully and $18.4\%$ did so for questions regarding mental health.

In a survey that contains potentially sensitive questions, $43.4\%$ of
respondents most preferred general content warnings (whether the survey
contains sensitive content), $30.9\%$ preferred specific content warnings (the
type of sensitive content the survey contains), and $23.7\%$ preferred mental
health resources (Figure 3). These numbers indicate that a far higher number
of workers prefer specific content warnings than there are currently in MTurk
studies.

Figure 3: Information desired for tasks about sensitive or harmful content.
Nearly a third of annotators desire specific content warnings (e.g.,
“homophobia”), though general content warnings (e.g., “offensive content”) are
more common in current tasks.

##  8 Recommendations

We consolidate our findings on worker perspectives to arrive at several
recommendations.

#### Pay at least minimum wage, but not exceptionally higher.

Payment can influence MTurk workers’ perception of the task as well as their
effort levels. We suggest paying around the minimum wage of $\$15$ per hour
(slightly above the median minimum threshold of $\$12$, and in accordance with
the views expressed by r/MTurk users), but not too much higher, which
decreases response quality.222We note that living wages vary by region and
inflation, so the ideal pay is likely to change over time and may depend on
the region from which MTurk workers are recruited. Additional budget may be
used as bonus payments, which are effective incentives when provided by
reputable accounts.

#### Minimize the collection of private information, and be transparent.

Workers are highly sensitive to questions that they see as violations of
privacy, such as demographic questions. It is thus prudent to minimize their
use, provide a brief explanation for their importance, clarify that the
answers are confidential and will not be held against the respondent, and
include an option for respondents not to answer. To protect worker privacy,
always drop Worker IDs before working with the dataset. Also follow worker-
centered policies such as Turkopticon’s Guidelines for Academic Requesters
Turkopticon (2014) to maintain transparent requester policies and
communication.

#### Avoid time-based filters and the Masters qualification.

Our results indicate that both lower and upper bounds on completion time are
flawed methods of quality filtering. Thus, we discourage rejecting HITs based
on time, barring humanly impossible extremes. In addition, to avoid
unnecessarily filtering out workers who joined MTurk later, do not use the
outdated Masters’ Qualification. For high-quality responses, a $98\%$ HIT
approval rate filter adheres to both worker preferences and research findings,
as it exactly equals the median value proposed in our survey results. Clearly
outline the requirements for task approval to increase trust between the
worker and the requester.

#### Consider effects on worker approval rate when controlling for quality.

Since MTurk spam rates are high (as we saw firsthand), quality filtering is
key to obtaining good results. However, workers are sensitive to the fact that
drops in approval rate bar them from future tasks, and so are both less
willing to accept tasks from requesters with lower approval rates, and
frustrated when work is rejected without explanation. As a result, it is best
to provide explanations when rejecting workers. Paid qualification tasks that
filter and/or train workers before they begin a larger main task, such as the
pipeline proposed by Zhang et al. (2022), can help to minimize spam responses
without needing to reject many responses on the main task. As text generation
tools become more powerful, AI-generated text detectors may also help to flag
potential spam responses Veselovsky et al. (2023).

#### Provide context for subjective tasks.

When possible, preface questions with an explanation of the purpose of the
annotation task, such as the downstream use case. Additional context is also
helpful, such as whether the text is from a longer passage or was sourced from
a specific platform.

#### Provide general and specific content warnings.

If questions in sensitive areas such as offensive content or mental health are
necessary, provide general and specific warnings for sensitive content, an
explanation of the questions’ purpose, and an overview of how the data will be
used.

## Limitations

We surveyed MTurk workers from the United States; the views of these workers
may differ from those of workers in other parts of the world. In addition,
because the survey was completed by the first 207 workers to see the posted
task, there may be sampling bias in the workers who answered (i.e., ones who
go on MTurk more often or are quicker to answer tasks are more likely to have
filled out the survey). Future studies with a larger sample pool of workers
from different parts of the world, and on different crowdworking platforms,
could help to examine the extent to which these results generalize.

## Ethical Considerations

Understanding the factors that make MTurk workers more likely to trust a
requester and thus more likely to provide personal information or accept tasks
with unusual payment schemes (e.g., low normal pay and high bonus pay) could
be misused by spam requesters or phishers to maliciously acquire more
information from MTurk users or extract their labor without fair pay. We
understand that these are potential consequences of our research, but hope
that they are outweighted by the potential benefits of good-faith requesters
understanding how to improve the effects of their tasks on workers.

## Acknowledgments

Many thanks to Nicholas Tomlin and Eric Wallace for their feedback on earlier
drafts of this paper.

## References

  * Arditte et al. (2016) Kimberly A Arditte, Demet Çek, Ashley M Shaw, and Kiara R Timpano. 2016.  The importance of assessing clinical phenomena in Mechanical Turk research.
  * Ashida and Komachi (2022) Mana Ashida and Mamoru Komachi. 2022.  Towards automatic generation of messages countering online hate speech and microaggressions.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 11–23, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Callison-Burch and Dredze (2010) Chris Callison-Burch and Mark Dredze. 2010.  Creating speech and language data with Amazon’s Mechanical Turk.  In _Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , CSLDAMT ’10, page 1–12, USA. Association for Computational Linguistics.
  * Gideon (2012) Lior Gideon. 2012.  The art of question phrasing.  In _Handbook of Survey Methodology for the Social Sciences_ , pages 91–107. Springer.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023.  ChatGPT outperforms crowd-workers for text-annotation tasks.  _ArXiv_ , abs/2303.15056.
  * Gray and Suri (2019) Mary L. Gray and Siddharth Suri. 2019.  _Ghost work how to stop Silicon Valley from building a new global underclass_.  Houghton Mifflin Harcourt.
  * Hara et al. (2018) Kotaro Hara, Abigail Adams, Kristy Milland, Saiph Savage, Chris Callison-Burch, and Jeffrey P. Bigham. 2018.  A data-driven analysis of workers’ earnings on Amazon Mechanical Turk.  In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ , CHI ’18, page 1–14, New York, NY, USA. Association for Computing Machinery.
  * Hughes et al. (2016) Jennifer Hughes, Abigail Camden, and Tenzin Yangchen. 2016.  Rethinking and updating demographic questions: Guidance to improve descriptions of research samples.  _Psi Chi Journal of Psychological Research_ , 21:138–151.
  * Jacobs and Wallach (2021) Abigail Z. Jacobs and Hanna Wallach. 2021.  Measurement and fairness.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 375–385, New York, NY, USA. Association for Computing Machinery.
  * Jacques and Kristensson (2019) Jason T. Jacques and Per Ola Kristensson. 2019.  Crowdworker economics in the gig economy.  In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ , CHI ’19, page 1–10, New York, NY, USA. Association for Computing Machinery.
  * Justo et al. (2017) Raquel Justo, M. Torres, and José Alcaide. 2017.  Measuring the quality of annotations for a subjective crowdsourcing task.  pages 58–68.
  * Karpinska et al. (2021) Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021.  The perils of using Mechanical Turk to evaluate open-ended text generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kim et al. (2021) Jane Paik Kim, Katie Ryan, Tenzin Tsungmey, Max Kasun, Willa A. Roberts, Laura B. Dunn, and Laura Weiss Roberts. 2021.  Perceived protectiveness of research safeguards and influences on willingness to participate in research: A novel mturk pilot study.  _Journal of Psychiatric Research_ , 138:200–206.
  * Kummerfeld (2021) Jonathan K. Kummerfeld. 2021.  Quantifying and avoiding unfair qualification labour in crowdsourcing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 343–349, Online. Association for Computational Linguistics.
  * Lease et al. (2013) Matthew Lease, Jessica Hullman, Jeffrey Bigham, Michael Bernstein, Juho Kim, Walter Lasecki, Saeideh Bakhshi, Tanushree Mitra, and Robert Miller. 2013.  Mechanical Turk is not anonymous.  _SSRN Electronic Journal_.
  * Pavlick and Kwiatkowski (2019) Ellie Pavlick and Tom Kwiatkowski. 2019.  Inherent disagreements in human textual inferences.  _Transactions of the Association for Computational Linguistics_ , 7:677–694.
  * Peer et al. (2014) Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014.  Reputation as a sufficient condition for data quality on Amazon Mechanical Turk.  _Behavior research methods_ , 46:1023–1031.
  * Perrigo (2022) Billy Perrigo. 2022.  Inside Facebook’s African sweatshop.  _TIME_.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _ACL_.
  * Sap et al. (2020) Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020.  Social bias frames: Reasoning about social and power implications of language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5477–5490, Online. Association for Computational Linguistics.
  * Shmueli et al. (2021) Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021.  Beyond fair pay: Ethical implications of NLP crowdsourcing.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3758–3769, Online. Association for Computational Linguistics.
  * Snow et al. (2008) Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Ng. 2008.  Cheap and fast – but is it good? evaluating non-expert annotations for natural language tasks.  In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing_ , pages 254–263, Honolulu, Hawaii. Association for Computational Linguistics.
  * Turkopticon (2014) Turkopticon. 2014.  Guildeines for academic requesters.
  * Turkopticon (2022) Turkopticon. 2022.  End the harm of mass rejections.
  * u/Bermin299 (2022) u/Bermin299. 2022.  Getting to the point you can’t trust any requestor paying less than a $1 (surveys).  Reddit.
  * u/dgrochester55 (2022) u/dgrochester55. 2022.  How seriously do you answer questions? Reddit.
  * u/gturker (2022) u/gturker. 2022.  Gotta love the transparency of a seemingly reasonable requester! Reddit.
  * u/LaughingAllTheWay83 (2022a) u/LaughingAllTheWay83. 2022a.  Gotta love the transparency of a seemingly reasonable requester! Reddit.
  * u/LaughingAllTheWay83 (2022b) u/LaughingAllTheWay83. 2022b.  How seriously do you answer questions? Reddit.
  * u/Lushangdewww (2019) u/Lushangdewww. 2019.  A question for workers with approval rate “greater than 99%”.  Reddit.
  * u/ptethesen (2022) u/ptethesen. 2022.  Rejection from simran g. Reddit.
  * u/Sharpsilverz (2022) u/Sharpsilverz. 2022.  Requesting ma speakers to read out loud for about 7 minutes– what is ethical pay? Reddit.
  * Veselovsky et al. (2023) Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023.  Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks.  _ArXiv_ , abs/2306.07899.
  * Whiting et al. (2019) Mark E. Whiting, Grant Hugh, and Michael S. Bernstein. 2019.  _Fair Work: Crowd Work Minimum Wage with One Line of Code_, volume 7.
  * Xia et al. (2017) Huichuan Xia, Yang Wang, Yun Huang, and Anuj Shah. 2017.  "Our privacy needs to be protected at all costs": Crowd workers’ privacy experiences on Amazon Mechanical Turk.  _Proc. ACM Hum.-Comput. Interact._ , 1(CSCW).
  * Zhang et al. (2022) Lining Zhang, João Sedoc, Simon Mille, Yufang Hou, Sebastian Gehrmann, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Miruna Clinciu, Saad Mahamood, and Khyathi Chandu. 2022.  Needle in a haystack: An analysis of finding qualified workers on mturk for summarization.

##  Appendix A Informed Consent

###  A.1 Key Information

You are being invited to participate in the online version of our research
study on Amazon Mechanical Turk (MTurk). Participation in research is
completely voluntary. The purpose of the study is to examine the best
practices and common pain points involved in conducting annotation tasks on
MTurk. The study will take approximately 10 minutes, and you will be asked to
respond to questions regarding your experience and opinion on conducting
annotation tasks. Risks and/or discomforts may include potentially sensitive
questions that may cause discomfort in the survey respondent. There is no
direct benefit to you. The results from the study may help improve general
MTurk workers’ experiences in future MTurk annotation tasks.
Introduction and Purpose
(Researcher names) We would like to invite you to take part in our research
study, which concerns observing MTurk workers’ experiences on the platform in
order to improve understanding on best practices and common pain points
involved in conducting annotation tasks. The key research areas involve 1)
fair payment methods and its effectiveness in incentivizing workers, 2)
administration of sensitive questions, 3) methods for maximizing response
quality, and 4) miscellaneous questions on survey design and MTurk workers’
experiences.
Procedures
If you agree to participate in this research, we will ask you to complete the
attached Qualtrics survey included in the MTurk task. The survey will involve
questions about payment, administration of sensitive questions, maximizing
response quality, survey design, and MTurk workers’ experiences, and should
take about 10 minutes to complete.
Benefits
There is no direct benefit to you from taking part in this study. It is hoped
that the research will increase knowledge of the risks involved in MTurk
surveys. For example, participants may learn about the fact that Worker IDs
are shared to survey requesters. In addition, the purpose of the study itself
is to improve the MTurk survey experience for MTurk workers by making it
safer, less invasive, and more reasonable. Thus, the MTurk workers completing
the task will benefit from the survey findings.
Risks/Discomforts
Although the survey does not explicitly ask any demographic questions, it asks
about participants’ perceptions and opinions on sensitive questions such as
personal demographic information and offensive content. These questions may be
a possible area of discomfort for the participant. As with all research, there
is a chance that confidentiality could be compromised; however, we are taking
precautions to minimize this risk.
Confidentiality
Your study data will be handled as confidentially as possible. If results of
this study are published or presented, individual names and other personally
identifiable information will not be used. Amazon Mechanical Turk
automatically stores a list containing the Worker IDs of all participants who
work on a survey requester’s task, but these Worker IDs will not be linked to
the anonymous Qualtrics responses, will not be downloaded, and will only be
used to ensure payment. We will delete the Worker IDs on the Mechanical Turk
platform immediately once participants have been paid. We will not obtain any
identifiers linked to individual Qualtrics survey responses and there is no
other potentially identifiable information collected in the survey. The
Qualtrics survey responses will be downloaded immediately after the required
number of responses is reached. After they are downloaded, they will be
deleted from the Qualtrics platform within one week. The survey data will be
secured on two password-protected computers, and it will be retained for no
more than 6 months. After this time period, only the aggregate stats will be
retained. We will not be transferring any identifiable data after results are
downloaded from Amazon Mechanical Turk. When the research is completed, we
will save the data for possible use in future research done by us or others.
We will retain these records for up to 6 months after the study is over. The
same measures described above will be taken to protect confidentiality of this
study data. Your personal information may be released if required by law.
Authorized representatives from the following organizations may review your
research data for purposes such as monitoring or managing the conduct of this
study: University of California. Identifiers might be removed and the de-
identified information used for future research without your additional
informed consent.
Compensation
Participants will be compensated with $2.50 upon completing the survey on
Amazon Mechanical Turk.
Rights
Participation in research is completely voluntary. You are free to decline to
take part in the project. You can decline to answer any questions and are free
to stop taking part in the project at any time. Whether or not you choose to
participate, to answer any particular question, or continue participating in
the project, there will be no penalty to you or loss of benefits to which you
are otherwise entitled.
Questions [Contact Information]
If you agree to take part in the research, please print a copy of this page to
keep for future reference, then check the “Accept” box below.

##  Appendix B Survey and Data Cleaning Details

Figures 4 through 10 contain the full text of the survey.

After data was collected, the mandatory free text responses were manually
reviewed by two authors for spam, including free text responses that were
identical for multiple survey responses, or direct copies of text from the
question with nothing added; numbers in response to questions that do not ask
for numbers; or responses that were incoherent. We also removed incomplete
responses (where the survey was only partially answered).

Figure 4: Full survey (page 1). Figure 5: Full survey (page 2). Figure 6: Full
survey (page 3). Figure 7: Full survey (page 4). Figure 8: Full survey (page
5). Figure 9: Full survey (page 6). Figure 10: Full survey (page 7).


# Avalanche correlations and stress-strain curves in discrete dislocation
plasticity

Henri Salmenjoki1, Lasse Laurson2 and Mikko J. Alava1,3 1Aalto University,
Department of Applied Physics, PO Box 11000, 00076 Aalto, Finland
2Computational Physics Laboratory, Tampere University, P.O. Box 692, FI-33101
Tampere, Finland
3NOMATEN Centre of Excellence, National Centre for Nuclear Research, A.
Soltana 7, 05-400 Otwock-Swierk, Poland

###### Abstract

The sequence of deformation bursts during plastic deformation exhibits scale-
free features. In addition to the burst or avalanche sizes and the rate of
avalanches the process is characterized by correlations in the series which
become manifest in the resulting shape of the stress-strain curve. We analyze
such features of plastic deformation with 2D and 3D simulations of discrete
dislocation dynamics models and we show, that only with severe plastic
deformation the ensuing memory effects become negligible. The role of past
deformation history and dislocation pinning by disorder are studied. In
general, the correlations have the effect of reducing the scatter of the
individual stress-strain curves around the mean one.

##  I Introduction

The fact that plastic deformation takes place by avalanches or bursts makes
for an interesting connection between the theory of avalanches in dislocation
systems and materials science Papanikolaou  _et al._ (2017); Alava  _et al._
(2014); Zaiser (2006). In the case of crystalline solids, plasticity is
mediated by the stress-driven collective dynamics of dislocations, i.e., line-
line topological defects of the crystal lattice. The paradigm of avalanches
implies scale-free features that the statistical properties of the bursts -
such as their sizes and durations and often the intra-avalanche waiting times
- follow. For ensembles of dislocations in plastically deforming crystals,
these signatures of critical-like dynamics are often linked either to the
system being driven to the proximity of a depinning-like phase transition
Ovaska  _et al._ (2015); Salmenjoki  _et al._ (2020), or exhibiting glassy
dislocation dynamics Ispánovity  _et al._ (2014); Lehtinen  _et al._ (2016a).

For a single micron-scale sample undergoing deformation the consequence of a
series of bursts is an irregular stress-strain curve Uchic  _et al._ (2004);
Dimiduk  _et al._ (2005); Ispánovity  _et al._ (2010); Papanikolaou  _et al._
(2012). Single crystals containing assemblies of discrete dislocations have
the property that, both in models Ispánovity  _et al._ (2014); Lehtinen  _et
al._ (2016a); Csikor  _et al._ (2007) and in mechanical tests such as
nano/micropillar compression Dimiduk  _et al._ (2006); Uchic  _et al._ (2009),
the plastic deformation bursts have a broad distribution of sizes. The stress-
strain curve then consists of a sequence of such bursts, separated by
quiescent periods during which the applied stress is increased. The resulting
stress-strain curve has a random appearance, and indeed these processes are
most often characterized simply by probability density functions of burst
sizes and stress increments, a description ignoring possible correlations in
the burst sequence Zaiser (2006).

Here, we go beyond such a simple characterization of fluctuating crystal
plasticity, by studying to what extent stresses, stress increments and bursts
sizes along the stress-strain curves are correlated. This is inherently
coupled to the question how one may exploit the statistics to reconstruct
stress-strain curves Kapetanou  _et al._ (2015); Szabó  _et al._ (2015).
Related questions include how the stress-strain curves compare with the mean
one Ispánovity  _et al._ (2013), in particular it is interesting to ask if
single systems have a tendency to converge to the mean behavior and how this
depends on the degree of plastic deformation. This is in turn dependent on how
random is random-looking plasticity, highlighting the need to measure
correlations in the avalanche activity. To this end, we investigate how the
correlations in avalanche activity influence the plastic deformation by means
of 2D and 3D discrete dislocation dynamics (DDD) simulations. Our work extends
previous studies of avalanche correlations in plasticity Kapetanou  _et al._
(2015); Weiss and Marsan (2003) and in the related problem of interface
depinning in disordered media Le Doussal and Thiery (2020). The main issues we
address are four. First, as a function of plastic strain, we quantify the
variation of individual samples around the average (stress-strain curve)
behaviour. We then measure the correlations in the series of bursts, focusing
on subsequent avalanches. These correlations are important but diminish along
the stress-strain curve. This results in a tendency to approach the average
response. In the initial stages of plastic deformation these correlations
leave an imprint on both the yield stress and on its variation from sample to
sample. An important question is how these memory effects work in general. To
investigate this, we also study the influence of past deformation by ”pre-
strained” systems and that of the presence of a competing dislocation pinning
mechanism by a field of precipitates. This is related to the general question
of how to classify memory effects in physical systems Keim  _et al._ (2019),
in particular in systems undergoing deformation Keim  _et al._ (2020); Mungan
_et al._ (2019); Pashine  _et al._ (2019). Our results tackle the subtle
effects that arise from the exact configuration of dislocations at the
beginning of a deformation experiment or trial: what are the consequences, and
how does that depend on the past history.

The structure of this paper follows the usual paths: first the methods
(Section II), then the results (Section III), and finally the conclusions
(Section IV).

##  II DDD simulations

To study the correlations between dislocation avalanches, we collect four
datasets: We perform standard DDD simulations in both two and three
dimensions, pre-strained systems (deformation history) in two dimensions and
systems with quenched pinning points/precipitates in three dimensions. For the
two-dimensional (2D) systems, we use an in-house developed code to model a
square-shaped cross-section of a crystal with infinitely long, parallel edge
dislocations. The model is similar to the one studied in several previous
works Ispánovity  _et al._ (2014); Ovaska  _et al._ (2015); Laurson  _et al._
(2010); Laurson and Alava (2012); Salmenjoki  _et al._ (2018). The dislocation
are randomly initialized with either positive or negative Burgers vectors with
magnitude $b$. Inside the simulation box, the dislocations are restricted to
move in their glide planes in $x$-direction with an equation of motion

$\frac{1}{\chi b}v_{i}=s_{i}b\left[\sigma_{ext}+\sum_{i\neq
j}s_{j}\sigma_{d}(\mathbf{r}_{j}-\mathbf{r}_{i})\right],$ (1)

where $s_{i}$ is the sign of the Burgers vector, $\chi$ is the mobility,
$\sigma_{ext}$ is the external stress and the sum is over all other
dislocations in the system. Moreover, the interaction arising from the shear
stress field is given by

$\sigma_{d}(\mathbf{r})=\frac{\mu
b}{2\pi(1-\nu)}\frac{x(x^{2}-y^{2})}{(x^{2}+y^{2})^{2}},$ (2)

where $\nu$ is the Poisson ratio and $\mu$ is the shear modulus. The system is
implemented with periodic boundaries and parameters are set to correspond to
the simulations with the largest system size found in Salmenjoki  _et al._
(2018), i.e. we initialize the system with 400 dislocations in a box with size
$L=100b$ and model parameters are chosen so that
$\frac{\mu}{2\pi(1-\nu)}=b=\chi=1$ and timestep is set to $0.1$. Therefore,
stress and strain are measured in dimensionless units. An example of a 2D DDD
system is illustrated in Fig. 1a.

Figure 1:  (a), (b) Examples of initial 2D and 3D dislocation structures,
respectively.  Figure 2:  Stress-strain curves (left) and strain-resolved
avalanche size distributions (right) for (a) 2D (b) 3D (c) Pre-strained 2D and
(d) 3D systems with precipitates. The average stress-strain curve (solid black
line) is accompanied by single system stress-strain curves (colored lines) and
the standard deviation around the average (shaded region). The size
distributions for avalanches in different strain bins (color coding) are
plotted with the fitting results to Eq. 4 (solid lines). The solid black lines
are added as guide for the eye.

The three-dimensional (3D) DDD simulations are conducted with our version of
the ParaDiS code Arsenlis  _et al._ (2007). In ParaDiS, rectangular systems
with dislocations are simulated and the program discretizes dislocation lines
into a set of nodes and straight segments. Interaction stresses between the
segments are derived from linear elasticity and the long-range nature of these
stresses is taken into account by considering periodic images of the system.
At the dislocation core, interactions are computed with output from MD
simulations. For the standard 3D DDD simulations (Fig. 1b), we set the
simulation parameters to the values of FCC aluminium ($b=0.286\,\mathrm{nm}$,
$\nu=0.35$, $\mu=26\,\mathrm{GPa}$) in a cubic system with size
$L_{3D}=1.43\,\mu\mathrm{m}$ and 40 initial straight mixed dislocations
Lehtinen  _et al._ (2016a). Unlike 2D DDD, we measure the 3D DDD results in
SI-units. On the other hand, the disordered systems (i.e. ones with quenched
pinning points included Lehtinen  _et al._ (2016b)) are performed along
similar lines as in Ref. Salmenjoki  _et al._ (2020). The disorder is
implemented as spherical coherent precipitates (with radius
$r_{p}=28.6\,\mathrm{nm}$) that form Gaussian barriers for dislocation motion,
i.e. the (radial) precipitate-dislocation interaction force is given by

$F(r)=-\nabla U(r)=\frac{2Ab^{3}re^{-\frac{r^{2}}{r_{p}^{2}}}}{r_{p}^{2}},$
(3)

where $A$ is a parameter characterizing precipitate strength. The materials
parameters used are those of aluminium (like in the standard case), but the
system size is $L_{depinning}=4\,\mu\mathrm{m}$ and initial number of
dislocations is 24. The precipitate parameters are chosen so –
$A=10^{10}\,\mathrm{Pa}$ and density $\rho_{p}=10^{20}\,\mathrm{m}^{-3}$ –
that the system dynamics is dominated by dislocation (de)pinning Salmenjoki
_et al._ (2020).

All systems are driven with the quasistatic stress-controlled loading scheme.
This means that, once we have initialized the systems with randomly placed
dislocations (2D) or generated structures using paradisgen which is
distributed with ParaDiS (3D), and these dislocations have found a meta-stable
state after relaxation with $\sigma_{ext}=0$, we start to increase the
external stress with a rate $\dot{\sigma}$. Only exception is the pre-strained
2D case, where the initial states are prepared by first running a stress ramp
until a pre-strain of $\varepsilon_{ID}=0.2$, after which the system is let to
relax again at zero stress. In 2D simulations (both basic and pre-strained),
we set $\dot{\sigma}_{2D}=2.5\cdot 10^{-7}$, in 3D $\dot{\sigma}_{3D}=2.5\cdot
10^{13}\mathrm{Pa}/\mathrm{s}$ in the $[010]$ direction, and in 3D with
precipitates $\dot{\sigma}_{Depinning}=1.0\cdot 10^{14}\mathrm{Pa}/\mathrm{s}$
in the $[100]$ direction. As $\sigma_{ext}$ increases, we measure the velocity
of dislocations inside the system (in 2D simply the sum of the $|v_{i}|$’s,
while in 3D we measure the absolute value of the extensive velocity
$V(t)=\sum_{i}l_{i}v_{\perp,i}$ where $l_{i}$ is the segment length and
$v_{\perp,i}$ velocity perpendicular to the line direction of the segment). If
the velocity signal then passes a preset threshold $V_{0}$, an avalanche
starts and the stress increments are stopped until the dislocations again
reach a jammed state and motion ceases. This way the systems produce stress-
strain curves with staircase-shape, as can be seen in Fig. 2 Szabó  _et al._
(2015). The avalanches are visible as the constant $\sigma$ plateaus and we
measure their sizes $s$ by the strain accumulated, i.e. change in
$\varepsilon$ (except for the systems with precipitates, for which the
avalanche size is the integral of the velocity signal, i.e.
$s^{\prime}=\int_{0}^{T}V(t)-V_{0}\,\mathrm{d}t$).

##  III Results

###  III.1 Avalanche analysis

We then proceed to analyze the avalanches that come in many sizes as seen in
Fig. 2 which also shows the strain-resolved size distributions for the studied
systems. In principle, the systems exhibit three types of avalanches: First,
there are the small avalanches arising from numerical noise and velocity
signal oscillating above and below the avalanche threshold - these are already
cropped out from the figures. Second, there are the power-law avalanches
which, as the name suggests, follow the distribution closely and are mostly
uncorrelated. Finally, the largest avalanches are the so called cut-off
avalanches as there the distributions start waning from the pure power-law.

Because the largest avalanches have also the largest impact on the stress-
strain curves, in what follows we restrict our analysis to them. However as we
search for correlations between avalanches and only few cut-off avalanches
occur per system, we take this into account as we define the threshold for the
large avalanches. To define large avalanches systematically, we first fit the
(strain-dependent) distributions with the typical form

$P(s)=s^{-\tau}e^{-s/s_{0}},$ (4)

where both $\tau$ (power-law exponent) and $s_{0}$ (the cutoff avalanche size)
are fitting parameters. Then we set the threshold of large avalanches to
$c\cdot s_{0}$ with $c$ some constant smaller than unity. Finally as we have
obtained values for $s_{0}$ in the different strain bins, we can interpolate
$c\cdot s_{0}(\varepsilon)$ to obtain an estimate if any specific avalanche
belongs to the large avalanches depending on its starting strain.

From the obtained set of large avalanches, we collect subsequent avalanches
that occur in the same system to study correlations between them. We focus on
parameters introduced in Fig. 3a, namely the starting strains of the preceding
and following avalanches, $\varepsilon_{1}$ and $\varepsilon_{2}$, the sizes
of the preceding and following avalanches $s_{1}$ and $s_{2}$, and the stress
increment between the avalanches $\Delta\sigma=\sigma_{2}-\sigma_{1}$. To
measure the correlations, we use the Spearman rank correlation coefficient
$\rho$. Unlike the more commonly used Pearson correlation coefficient, which
measures the linear relationship between the values of two variables, Spearman
correlation coefficient considers the rank of the variables, thus resulting in
+1 (or -1) if the variables form a perfect monotonously increasing (or
decreasing) curve, and 0 if no correlation between the ranks exists Spearman
(1904). For the case of correlations between (power-law distributed)
avalanches, assuming linear dependencies seems unnecessarily restrictive and,
thus, Spearman correlation coefficient is preferred here.

Figure 3: (a) Illustration of the parameters of the two subsequent avalanches.
(b), (c) Spearman correlation coefficient between $(\sigma_{1},\Delta\sigma)$
and $(\Delta\sigma,s_{2})$ as a function of the first avalanche starting
strain for avalanches from 2D and 3D systems, respectively.

###  III.2 Avalanche correlations in 2D and 3D DDD

We continue to study the correlations between two subsequent large avalanches.
The observations made here come with avalanches collected from 5000 2D DDD and
1000 3D DDD (from which 250 have been driven to $\varepsilon\sim 1\%$ and the
rest to much smaller strains) systems. In Figs. 3b-c we have the Spearman
correlation coefficient for the 2D and 3D cases, respectively, when $\rho$ are
computed between the starting stress of the previous avalanche $\sigma_{1}$
and the stress increment $\Delta\sigma$, and between the stress increment and
the size of the following avalanche $s_{2}$. Moreover,
$\rho_{\sigma_{1},\Delta\sigma}$ and $\rho_{\Delta\sigma,s_{2}}$ are plotted
as a function of the starting strain of the preceding avalanche,
$\varepsilon_{1}$, to see how the correlations change along the stress-strain
curve.

Starting from $\rho_{\Delta\sigma,s_{2}}$, we see that with small
$\varepsilon_{1}$ in both 2D and 3D there is a strong monotonously increasing
dependence between the variables as $\rho>0.5$ in both cases (for reference,
the inset of Fig. 3b shows the scatter plot of the variables). As
$\varepsilon_{1}$ increases and the systems approach the plateau of the
stress-strain response, the correlations disappear. Then with
$\rho_{\sigma_{1},\Delta\sigma}$ there is an opposite, monotonously decreasing
dependence between $\sigma_{1}$ and $\Delta\sigma$ which similarly weakens
with strain, although at small strains the correlation is much weaker in 2D
($\sim-0.2$ at best) than in 3D ($<-0.6$). The correlations measured here are
obtained with large avalanche thresholds $c=0.25$ and $c=0.1$ for 2D and 3D,
respectively, and we have tested that the results seem robust with respect to
the value of $c$ used or the number of strain bins. Other avalanche parameter
pairs (e.g. $s_{1}$ and $s_{2}$) show no similar, notable correlations.

Figure 4: A randomly chosen set of stress-strain curves shifted with the
average curve, for 2D (a) and 3D (b) systems. The shaded region corresponds to
the standard deviation.

The correlations have a significant impact on the systems’ stress-strain
response with small strains. This is because both
$\rho_{\sigma_{1},\Delta\sigma}$ and $\rho_{\Delta\sigma,s_{2}}$ have the same
effect of pushing the stress-strain curve towards the system average: If the
preceding avalanche starts with a larger stress $\sigma_{1}$, then there is a
smaller increment in stress before the next avalanche and vice versa, due to
the negative $\rho_{\sigma_{1},\Delta\sigma}$. Similarly if the stress
increment between the avalanches is large (small), next avalanche will be
large (small) as $\rho_{\Delta\sigma,s_{2}}$ is positive. The push towards the
average curve is visualized in Figs. 4a-b which show randomly chosen stress-
strain curves of single 2D and 3D systems shifted by the average
$\sigma(\varepsilon)$. In the figure, intersections of the single curves with
the average ($\sigma-\langle\sigma\rangle=0$) are most frequent with small
strains where the correlations are stronger, and cease as the correlations
vanish.

Figure 5: The rate of intersections with the average stress-strain curves
$\frac{N(\sigma-\langle\sigma\rangle=0)}{\mathrm{d}\varepsilon}$ and the
distribution of avalanche starting strains in (a) 2D and (b) 3D systems. For
2D systems, the dashed line also shows the goodness of a neural network (NN)
prediction of the stress-strain curve as a function of strain as obtained in
Salmenjoki  _et al._ (2018).

To elaborate, Fig. 5 shows the average rate
$\frac{N(\sigma-\langle\sigma\rangle=0)}{\mathrm{d}\varepsilon}$ along with
avalanche starting strain distributions. Indeed the rate decreases with large
strains and it follows $P(\varepsilon_{aval})$ in both 2D and 3D. But there
are some fundamental differences between the two sets. In 2D, the rate of
intersections and avalanche activity is confined to a section of strains,
while in 3D the rate and avalanche activity are in an ongoing decrease. This
difference in the small strain behaviour is mostly explained by considering
the simulation protocols: the 2D system are relaxed before the loading to a
state where all the motion stops in the limit of numerical error. However the
3D systems reach no such state as there remain some oscillation and slow decay
of dislocation motion, thus resulting in (small) bursts already in the start
of the stress ramp.

The eventual decrease of
$\frac{N(\sigma-\langle\sigma\rangle=0)}{\mathrm{d}\varepsilon}$ at large
strains has a connection to the ’yield stress’, i.e. the system-specific
stress required to enter plastic flow. One reason for the dropping rate is the
less and less frequent avalanches but additionally, the stress increments
between the avalanches get smaller and the standard deviation from the average
stress-strain curve increases compared to $\langle\Delta\sigma\rangle$ as seen
in Fig. 6 which shows the strain-dependent complementary cumulative
distributions of $\Delta\sigma$. Therefore, the single systems ’freeze’ their
relative behavior with respect to $\langle\sigma\rangle$ and decide between
being a ”stronger” or a ”weaker” sample. This then affects the predictability
of single system stress-strain curves interestingly: Fig. 5a illustrates also
the goodness of a neural network (NN) fit of the 2D initial dislocation
structures to the ensuing stress-strain curves computed in Ref. Salmenjoki
_et al._ (2018). The NN fit is best at small strains and large strains while
at intermediate strains, where
$\frac{N(\sigma-\langle\sigma\rangle=0)}{\mathrm{d}\varepsilon}$ is high and
most of the avalanche activity happens, the score of the fit drops to a clear
minimum. Thus, predicting the stress-strain curve accurately is impossible at
strains, where the systems oscillate below and above the average behaviour.

Figure 6:  Strain-dependent complementary cumulative distributions of
$\Delta\sigma$ in (a) 2D and (b) 3D systems for avalanches starting with
varying strain level (color coding). In (a), the dashed lines illustrate the
standard deviation of stress-strain curves with the corresponding strain.

###  III.3 Avalanche correlations in pre-strained and precipitate-dominated
systems

What then happens to these correlations, that push the system response towards
the average, if the system has some deformation history or includes disorder
in the form of quenched pinning points? To study the effect of deformation
history, we consider 5000 2D DDD systems that have been first pre-strained up
to a strain $\varepsilon_{ID}=0.2$ and then relaxed to a new, meta-stable
initial state Salmenjoki  _et al._ (2018). Avalanches that then occur during a
quasistatic stress ramp in these pre-strained systems are analyzed with the
same scheme as applied above on the basic systems.

Fig. 7a shows the Spearman correlation coefficients between relevant
parameters of two subsequent avalanches in pre-strained systems.

Figure 7:  (a) Spearman correlation coefficient between subsequent avalanche
parameters in pre-strained 2D systems along the stress-strain curve. For
reference, the color-coded dashed lines show $\rho_{\sigma_{1},\Delta\sigma}$
and $\rho_{\Delta\sigma,s_{2}}$ measured in the systems without pre-strain.
(b) The rate of intersections with the average stress-strain curve and
distribution of avalanche starting strains and NN score for pre-strained
systems. (c) Spearman correlation coefficient between subsequent avalanche
parameters in 3D systems with precipitates (depinning). The used large
avalanche thresholds are (a) $c=0.15$ and (b) $c=0.2$

Generally, the correlations are similar compared to the 2D case:
$\rho_{\sigma_{1},\Delta\sigma}$ is negative for small strains and
$\rho_{\Delta\sigma,s_{2}}$ is positive, both steering the single system
response towards the average. But there are two notable differences too:
First, the correlations seem to decline already with smaller strains than in
the 2D case. Although this is not that clear in the rate of intersections with
the average curve in Fig. 7b which resembles that of Fig. 5a, the
predictability in the form of the score of the NN fit increases more sharply
in pre-strained systems. Second, correlation between first avalanche size and
the following stress increment, which was negligible in the basic 2D case, is
significant in pre-strained systems.

The final set of avalanches is collected from 100 3D DDD simulations where now
the dislocations are accompanied by precipitates. Moreover, the precipitate
strength is chosen to ensure the dislocation dynamics is governed by
dislocation depinning with a distinct critical stress $\sigma_{c}=4.4\cdot
10^{7}\,\mathrm{Pa}$ of dislocation flow Salmenjoki  _et al._ (2020). Opposed
to pure systems following glassy dislocation dynamics, pinning force of
disorder becomes the dominating interaction and it changes the subsequent
avalanche correlations entirely as is seen in Fig. 7c. With precipitates, the
correlations are insignificant at small strains but start to increase at
strains close to the plateau of the stress-strain curve, although at larger
strains the number of avalanches decreases causing larger error estimates on
the magnitude of the correlation. Thus at small strains, the avalanches occur
randomly as dislocation segments break away from pinning points. At large
strains, the (possible) correlations follow from the approaching $\sigma_{c}$:
$\rho_{\sigma_{1},\Delta\sigma}\approx-0.5$ means that, as a larger avalanche
occurs with large stress, the following avalanche starts after a smaller
stress increment. Similarly $\rho_{\Delta\sigma,s_{2}}\approx 0.5$ after
larger stress increments between the avalanches, the following avalanche will
be larger. Therefore with depinning, the correlations between avalanches start
to push single curves towards the average curve (i.e. $\sigma_{c}$) at large
strains as opposed to ’pure’ samples where the correlations were observed in
the small strain region.

###  III.4 Randomly generated stress-strain curves

To further elaborate on the dislocation systems’ tendency to have mechanical
response follow the average, we refine the idea of randomly drawing stress-
strain curves as introduced in Kapetanou  _et al._ (2015). In our simulations
with quasistatic stress ramp, the resulting system response is a stress-strain
curve consisting of two recurring building blocks with three parameters: there
are the avalanches with sizes $s$, which are separated by (near) linear
increase of stress $\Delta\sigma$ during strain increment of
$\Delta\varepsilon$ (Fig. 3a). Thus, the idea is to resolve the independent
distributions of the three variables – $p(s)$, $p(\Delta\sigma)$ and
$p(\Delta\varepsilon)$ – and draw samples from the distributions to mimic
stress-strain curves of the simulations.

We approximate the distributions by utilizing a Markov chain Monte Carlo
(MCMC) method called Metropolis-Hastings Gelman  _et al._ (2013). Because the
distributions evolve as the simulations progress, we divide the avalanches to
eight strain bins and build separate samplers for each variable in each strain
bin. We neglect all the small noise avalanches as we fit the samplers. To
ensure the convergence of our MCMC samplers, we measure the potential scale
reduction factor and compare the distributions of the drawn samples to the
results of the simulations Gelman  _et al._ (2013). For comparison, we also
test a multivariate distribution, $p(s,\Delta\sigma,\Delta\varepsilon)$, which
should capture part of the inter-avalanche correlations observed in this
paper, but unfortunately this converges only for the 2D and pre-strained
cases.

The resulting random stress-strain curves are illustrated in Fig. 8. The
different rows of the figure correspond to example stress-strain curves (top
row), average stress-strain curve, standard deviation from the average curve
and the rate of intersecting the average curve (bottom row) for the different
systems (2D, 3D, pre-strained, depinning) separated in distinct columns.
Starting from the top row, the randomly drawn curves have the same staircase-
like shape as the simulated curves as expected. The average shape of the
randomly drawn curves also follows the simulation results nicely at least with
small and intermediate strains. However, there is a clear stress overshoot
which emerges for every case, thus hinting at a possible systematic error. But
instead of MCMC sampling, the error arises already from the simulations: the
avalanche data that we use to build the samplers does not include those
avalanches that are unfinished as the simulation finishes. Therefore, large
avalanches are underrepresented in the fitting data and the randomly drawn
curves have larger slope at large strains than required.

Figure 8:  MCMC sampled random stress-strain curves. Different columns from
left to right correspond to different datasets of basic 2D, basic 3D, pre-
strained (2D) and depinning (3D) simulations. In the top row we have examples
of randomly drawn stress-strain curves (dashed lines for curves from separate
distributions, dash-dotted for multivariate distributions where applicable;
see text for more information) compared to simulated curves (solid lines). The
second row from top shows the average stress-strain curves of 1000 randomly
drawn samples along with the simulation results and the third row from the top
has the standard deviation from the average curve. The bottom row has the rate
of intersections with the average curve.

The standard deviation from the average (second row from the bottom) shows a
clear distinction between the simulated and the random stress-strain curves.
Using independent separate distributions for samplers yields curves that have
a significantly wider spread at all strains than the simulated curves which,
again, highlights the tendency of the dislocation systems to follow the
average response. In 2D and pre-strained cases where we are able to draw
curves also from the converged multivariate distribution
$p(s,\Delta\sigma,\Delta\varepsilon)$, standard deviation is slightly closer
to the simulated than the separate distributions. As the multivariate
distribution contains some information of the correlations between
$\Delta\sigma$ and $s_{2}$ and $\sigma_{1},\Delta\sigma$ (through the
different samplers along the stress-strain curve), the wider spread of the
curves could imply some longer-range correlations in the avalanche time
series.

Finally the bottom row shows the rate of intersections with the average curve
compared to the simulations. With 2D and pre-strained data, the multivariate
random curves show more similar magnitude and shape of
$\frac{N(\sigma-\langle\sigma\rangle=0)}{\mathrm{d}\varepsilon}$ then the
separate distribution random curves as expected from the smaller standard
deviation. In 3D and depinning, the rate magnitude is quite close to the
simulation results even though the curves are drawn from separate
distributions. However, there the small strain behaviour is not captured by
the sampler because the initial state is not entirely stable and many small
avalanches occur in simulations with ParaDiS (as mentioned above), and our
samplers are fitted without the noisy avalanche data.

The comparisons of reconstructed and simulated stress-strain curve sets lead
us to the final question of the averaging of stress with increasing strain.
This relates directly to the question of what strain values are physically
justified for an effective definition of a yield stress. Figure 9 shows for
the 3D cases how the standard deviation of sample-dependent stress values
decays with $\varepsilon$. We find without and with precipitates similar
behaviors even though as noted the strain ranges are different. The standard
deviation decays quite fast with strain. It is notable that reconstruction
leads to much larger variation.

Figure 9: Standard deviation of stress values divided by the average stress as
a function of strain in (a) 3D and (b) depinning simulations (solid line)
compared to the case of randomly drawn curves from separate distributions
(dashed line).

##  IV Conclusions

In this work, we have studied the coupling of avalanche dynamics in plastic
deformation with the sample response. We show that the process is
fundamentally different from usual avalanching systems, where the presence of
a dynamical phase transition between active (plastic flow) and passive phases
makes only the proximity of the critical point (here, yield stress)
interesting due to the mechanism of a diverging correlation length. Instead,
in the absence of pinning points (precipitates) interfering with dislocation
motion the subsequent avalanches exhibit correlations, which decay along the
stress-strain curve, and result in a reduced scatter of the individual stress-
strain curves around the average response. The main features of our results
are independent of dimension (2D vs 3D), and are present also both in pre-
strained samples or dislocation systems with quenched pinning points. The case
with a true depinning transition has qualitatively quite similar behavior to
all the other cases as Fig. 8 shows in particular in stress-strain
reconstruction, even if the correlations tend to increase as the critical
stress of the depinning is approached. A signature of the depinning critical
point is also flattening of the stress-strain curve as the critical stress is
approached, due to divergent size of dislocation avalanches at the critical
point.

These issues are best explored in the context of studies of plasticity on the
level of small, micron-size samples. The classical case would be the
compression of micropillars, where as noted already fluctuations and
avalanches are omnipresent Uchic  _et al._ (2004); Dimiduk  _et al._ (2005);
Ispánovity  _et al._ (2010); Papanikolaou  _et al._ (2012). The role of big
avalanches and the tendency to follow (or not follow) the average behaviour at
a given strain may be investigated by gathering enough statistics and by
paying attention to the correlations as we have done here. An additional
feature is that the presence of self-averaging at larger strains seems to
imply that the extra disorder (precipitates here) controls the dislocation
dynamics. Precipitation strengthening is thus coupled to this feature.

One should also point out the fact that in the 3D case, if one defines the
yield stress to correspond, e.g., to 0.1 % plastic strain our simulations show
that the width of the yield stress distribution is significantly narrower than
one might expect assuming uncorrelated avalanches (Fig. 9). It would be
interesting to explore such correlations and their effect on the shape of the
stress-strain curve in deformation experiments, and extend our study to the
case of polycrystalline samples. Finally, avalanche correlations could be
looked for also in amorphous plasticity Budrikis  _et al._ (2017).

###### Acknowledgements.

LL acknowledges the support of the Academy of Finland via the Academy Project
COPLAST (project no. 322405). HS acknowledges the support from Finnish
Foundation for Technology Promotion. MA acknowledges support from the European
Union Horizon 2020 research and innovation programme under grant agreement No
857470 and from European Regional Development Fund via Foundation for Polish
Science International Research Agenda PLUS programme grant No MAB PLUS/2018/8.
The authors acknowledge the computational resources provided by the Aalto
University School of Science “Science-IT” project, as well as those provided
by CSC (Finland).

## References

  * Papanikolaou  _et al._ (2017) S. Papanikolaou, Y. Cui, and N. Ghoniem, Modelling and Simulation in Materials Science and Engineering 26, 013001 (2017).
  * Alava  _et al._ (2014) M. J. Alava, L. Laurson, and S. Zapperi, The European Physical Journal Special Topics 223, 2353 (2014).
  * Zaiser (2006) M. Zaiser, Advances in Physics 55, 185 (2006).
  * Ovaska  _et al._ (2015) M. Ovaska, L. Laurson, and M. J. Alava, Scientific Reports 5, 1 (2015).
  * Salmenjoki  _et al._ (2020) H. Salmenjoki, A. Lehtinen, L. Laurson, and M. J. Alava, Physical Review Materials 4, 083602 (2020).
  * Ispánovity  _et al._ (2014) P. D. Ispánovity, L. Laurson, M. Zaiser, I. Groma, S. Zapperi, and M. J. Alava, Physical Review Letters 112, 235501 (2014).
  * Lehtinen  _et al._ (2016a) A. Lehtinen, G. Costantini, M. J. Alava, S. Zapperi, and L. Laurson, Physical Review B 94, 064101 (2016a).
  * Uchic  _et al._ (2004) M. D. Uchic, D. M. Dimiduk, J. N. Florando, and W. D. Nix, Science 305, 986 (2004).
  * Dimiduk  _et al._ (2005) D. Dimiduk, M. Uchic, and T. Parthasarathy, Acta Materialia 53, 4065 (2005).
  * Ispánovity  _et al._ (2010) P. D. Ispánovity, I. Groma, G. Györgyi, F. F. Csikor, and D. Weygand, Physical Review Letters 105, 085503 (2010).
  * Papanikolaou  _et al._ (2012) S. Papanikolaou, D. M. Dimiduk, W. Choi, J. P. Sethna, M. D. Uchic, C. F. Woodward, and S. Zapperi, Nature 490, 517 (2012).
  * Csikor  _et al._ (2007) F. F. Csikor, C. Motz, D. Weygand, M. Zaiser, and S. Zapperi, Science 318, 251 (2007).
  * Dimiduk  _et al._ (2006) D. M. Dimiduk, C. Woodward, R. LeSar, and M. D. Uchic, Science 312, 1188 (2006).
  * Uchic  _et al._ (2009) M. D. Uchic, P. A. Shade, and D. M. Dimiduk, Annual Review of Materials Research 39, 361 (2009).
  * Kapetanou  _et al._ (2015) O. Kapetanou, V. Koutsos, E. Theotokoglou, D. Weygand, and M. Zaiser, Journal of the Mechanical Behavior of Materials 24, 105 (2015).
  * Szabó  _et al._ (2015) P. Szabó, P. D. Ispánovity, and I. Groma, Physical Review B 91, 054106 (2015).
  * Ispánovity  _et al._ (2013) P. D. Ispánovity, Á. Hegyi, I. Groma, G. Györgyi, K. Ratter, and D. Weygand, Acta Materialia 61, 6234 (2013).
  * Weiss and Marsan (2003) J. Weiss and D. Marsan, Science 299, 89 (2003).
  * Le Doussal and Thiery (2020) P. Le Doussal and T. Thiery, Physical Review E 101, 032108 (2020).
  * Keim  _et al._ (2019) N. Keim, J. Paulsen, Z. Zeravcic, S. Sastry, and N. SR, Reviews of Modern Physics 91, 035002 (2019).
  * Keim  _et al._ (2020) N. Keim, J. Hass, B. Kroger, and W. D, Physical Review Research 2, 012004(R) (2020).
  * Mungan  _et al._ (2019) M. Mungan, S. Sastry, K. Dahmen, and I. Regev, Physical Review Letters 123, 178002 (2019).
  * Pashine  _et al._ (2019) N. Pashine, D. Hexner, A. Liu, and S. Nagel, Science Advances 5, 4215 (2019).
  * Laurson  _et al._ (2010) L. Laurson, M.-C. Miguel, and M. J. Alava, Physical Review Letters 105, 015501 (2010).
  * Laurson and Alava (2012) L. Laurson and M. J. Alava, Physical Review Letters 109, 155504 (2012).
  * Salmenjoki  _et al._ (2018) H. Salmenjoki, M. J. Alava, and L. Laurson, Nature Communications 9, 1 (2018).
  * Arsenlis  _et al._ (2007) A. Arsenlis, W. Cai, M. Tang, M. Rhee, T. Oppelstrup, G. Hommes, T. G. Pierce, and V. V. Bulatov, Modelling and Simulation in Materials Science and Engineering 15, 553 (2007).
  * Lehtinen  _et al._ (2016b) A. Lehtinen, F. Granberg, L. Laurson, K. Nordlund, and M. J. Alava, Physical Review E 93, 013309 (2016b).
  * Spearman (1904) C. Spearman, The American Journal of Psychology 15, 72 (1904).
  * Gelman  _et al._ (2013) A. Gelman, J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin,  _Bayesian Data Analysis_ (CRC press, 2013).
  * Budrikis  _et al._ (2017) Z. Budrikis, D. F. Castellanos, S. Sandfeld, M. Zaiser, and S. Zapperi, Nature Communications 8, 1 (2017).