# GPT Assisted Annotation of Rhetorical and Linguistic Features for
Interpretable Propaganda Technique Detection in News Text.

Kyle Hamilton<EMAIL_ADDRESS>0000-0003-0809-0664 School of
Computer Science and SFI Centre for Research Training in Machine
LearningTechnological University DublinDublinIreland ,  Luca Longo
<EMAIL_ADDRESS>0000-0002-2718-5426 Artificial Intelligence and
Cognitive Load Research Lab, School of Computer Science and SFI Centre for
Research Training in Machine LearningTechnological University
DublinDublinIreland and  Bojan Božić<EMAIL_ADDRESS>0000-0002-4420-1029 School of Computer Science and SFI Centre for Research
Training in Machine LearningTechnological University DublinDublinIreland

(2024)

###### Abstract.

While the use of machine learning for the detection of propaganda techniques
in text has garnered considerable attention, most approaches focus on “black-
box” solutions with opaque inner workings. Interpretable approaches provide a
solution, however, they depend on careful feature engineering and costly
expert annotated data. Additionally, language features specific to
propagandistic text are generally the focus of rhetoricians or linguists, and
there is no data set labeled with such features suitable for machine learning.
This study codifies 22 rhetorical and linguistic features identified in
literature related to the language of persuasion for the purpose of annotating
an existing data set labeled with propaganda techniques. To help human experts
annotate natural language sentences with these features, RhetAnn, a web
application, was specifically designed to minimize an otherwise considerable
mental effort. Finally, a small set of annotated data was used to fine-tune
GPT-3.5, a generative large language model (LLM), to annotate the remaining
data while optimizing for financial cost and classification accuracy. This
study demonstrates how combining a small number of human annotated examples
with GPT can be an effective strategy for scaling the annotation process at a
fraction of the cost of traditional annotation relying solely on human
experts. The results are on par with the best performing model at the time of
writing, namely GPT-4, at 10x less the cost. Our contribution is a set of
features, their properties, definitions, and examples in a machine-readable
format, along with the code for RhetAnn and the GPT prompts and fine-tuning
procedures for advancing state-of-the-art interpretable propaganda technique
detection.

Natural Language Processing, Large Language Models, Annotation, Rhetorical
Devices, Propaganda Technique Detection

††journalyear: 2024††copyright: rightsretained††conference: Companion
Proceedings of the ACM Web Conference 2024; May 13–17, 2024; Singapore,
Singapore††booktitle: Companion Proceedings of the ACM Web Conference 2024
(WWW ’24 Companion), May 13–17, 2024, Singapore, Singapore††doi:
10.1145/3589335.3651909††isbn: 979-8-4007-0172-6/24/05††ccs: Computing
methodologies Information extraction

##  1\. Introduction

The study of rhetoric has a rich history spanning thousands of years,
traditionally falling within the purview of the humanities and relying on
manual analysis methods. Given the limitations in the scalability of manual
analysis, we naturally turn to exploiting machine learning approaches. In
turn, the challenge for the machine learning community is a shortage of
labelled data. Hence, most machine learning-based approaches have been
executed on small data sets annotated with specific rhetorical figures, such
as, for example, “chiasmus” (Dubremetz and Nivre, 2018), “litotes” (Mitrović
et al., 2020), or “doubt” (Ward et al., 2022), to name a few.

Both propagandistic techniques and rhetorical devices can be considered tools
of persuasion and have certain overlapping characteristics. Indeed, “the major
techniques of propaganda are long known rhetorical techniques gone wrong”
(Bryant, 1953). To our knowledge, the most relevant data set related to
rhetoric in news text is the Propaganda Technique Classification (PTC) corpus
developed by Da San Martino et al. (2019). The PTC corpus is annotated with 18
propaganda techniques divided into “logical fallacies” and “emotion and
sentiment mechanisms”. These can be considered logos and pathos in the
nomenclature of rhetoric.111Notably, “appeals to authority” is categorized
under “logical fallacies”, whereas, in the rhetoric tradition, this would
belong to a third category, namely “ethos”.  Furthermore, some propaganda
techniques, such as “doubt”, have equivalent counterparts in rhetorical
devices. In contrast, others tend to have a broader scope, such as “flag-
waving,” which includes appeals to nationalism as a means of persuasion.

We propose that propaganda techniques and rhetorical devices can be further
deconstructed into grammatical and word choice components as developed by
Fahnestock (2011) in Rhetorical Style: The Uses of Language in Persuasion. If
true, these components, or “features” as we will refer to them from here on
in, should correlate with propaganda techniques. However, to test this
assumption, we need to be able to extract the features from natural language
in order to use them for model training and inference. No such extraction
procedure exists for these specific features. Thus, the focus of this study is
on constructing such a procedure.

We were further constrained by a very limited budget. This is not an unusual
position to be in - cost-saving techniques can be critical to a researcher’s
ability to conduct their work. We propose an annotation strategy which takes
advantage of GPT-4’s explanations to aid in the human annotation phase, where
a small subset of the data is annotated, and utilizes an iterative prompt
engineering approach to annotate the remaining data using GPT-3.5. We show
that even noisy human-labeled data can be utilized to fine-tune GPT-3.5 to
achieve the performance of its state-of-the-art but 10x more expensive
counterpart, GPT-4. Our process is illustrated in Figure 1.

Figure 1. Three stages of GPT assisted annotation of rhetorical and linguistic
features for interpretable propaganda technique detection. Code available in:
https://github.com/kyleiwaniec/rhetoric-annotation

Even though our long-term goal is specific to propaganda detection, we believe
that the features and annotation procedure we propose can be used in any NLP
task that aims to leverage the characteristics of the language of persuasion.
In addition, this effort could contribute to developing a linguistic
rhetorical representation akin to existing representations such as Abstract
Meaning Representation (AMR) or Rhetorical Structure Theory (RST).

##  2\. Methodology

###  2.1. Data

We utilize the PTC corpus, introduced at the NLP4IF workshop at EMNLP-IJCNLP
2019, which consists of 451 news articles annotated with 18 propaganda
techniques at the sub-sentence fragment level. This means a sentence can have
multiple fragments labelled with multiple propaganda techniques. Figure 2
illustrates an example sentence from PTC. Details about this data set and
associated tasks can be found in Da San Martino et al. (2019). We roll up the
labels to the sentence level as per Yu et al. (2021), which, at the time of
this writing, represents the state-of-the-art interpretable propaganda
technique detection in news text. Thus, fragment location information is
discarded, and each sentence is labelled with zero or more propaganda
techniques. This aligns with similar open problems devoted to creating
interpretation techniques for opaque, black-box generative and large language
models (Longo et al., 2024), both at the global and local levels.

Figure 2. A sentence from the PTC corpus annotated at the fragment level with
three occurrences of propaganda techniques. Fragments are highlighted in
white.

###  2.2. Features

We use Fahnestock (2011) as a manual to codify 22 features, each with anywhere
from 3 to 14 possible properties, summarized in Table 5 of the appendix.
Fahnestock (2011)’s book is organized into four parts: I) Word Choice, II)
Sentences, III) Interactive dimensions, and IV) Passage construction. We
utilize only parts I and II as they can be applied at the sentence level and
leave III and IV for future work. For each property we generate a brief
definition by summarizing the corresponding section in the book and extracting
one or more examples. See Table 1 for the hyperbole property of the Tropes
feature.

I. Word Choice
---
Tropes
hyperbole - An overstatement. An exaggerated statement or claim not meant to
be taken literally. Example: “I’m so hungry I could eat a horse.”
Table 1. Definition and example of the hyperbole property of the Tropes
feature.

###  2.3. Human annotations

We engaged a professional annotation firm to annotate a subset of the PTC
corpus with our features. This task was undertaken by three independent human
experts, each with a background in linguistics.

To estimate the task’s difficulty, we annotated 15 sentences, randomly
selected from the PTC corpus, using a spreadsheet where each sentence is a
row, and each feature is a column with drop-down menus containing the
properties. Definitions and examples were stored in separate sheets for
reference as needed. This process proved cumbersome and time-consuming as it
required multiple clicks, moving back and forth between sheets, finding the
relevant row, and scrolling to the appropriate column at each move from one
sheet to another. With so many redundant actions and the necessity of keeping
track of the current sentence and feature in memory, completing the annotation
process for one thousand sentences was estimated at 1 year, working 8 hours a
day, 40 hours a week. To reduce the time needed to complete the task, we built
a web-based user interface called RhetAnn.

Figure 3. RhetAnn - Rhetoric Annotation web-based application. The Verb Phrase
(VP) node is selected in the parse tree. The current feature is Aspect, and
the selected property is progressive.

###  2.4. Annotation user interface – RhetAnn

The RhetAnn user interface - Figure 3 \- contains all of the information
needed for a given sentence and feature, including all property definitions
and examples, and allows the annotator to move easily between features and
sentences without having to keep anything in memory. In addition to reducing
the complexity associated with the above-mentioned actions, we have added a
GPT-4 assistant. During our initial estimation efforts, we found that the
explanations provided by GPT-4 were sometimes helpful and could speed up the
annotation process. This is similar to the finding in Wei et al. (2023).
Annotators were instructed to use the assistant only as an aid and that the
final annotation must always be verified. The prompt to GPT-4 consists of the
same instructions and definitions/examples as given to the human annotators. A
sample prompt is given in Table 2. Each time an annotator submits an
annotation, the GPT-4 API is called, and the response is stored.

Prompt
---
You are a rhetorician and linguist specializing in news text.
Your task is to identify which, if any, of the following properties of Aspect
are used in the example text. You may select multiple properties. Each line
contains a property followed by a colon, followed by a brief definition and
example(s):
simple: Expressing a simple fact. Examples: ’Rover eats bones.’, ’Rover ate a
bone.’
perfect: Expressing a completed action. Example: ’Rover has eaten a bone.’
progressive: Expressing an ongoing, habitual, or incomplete action. Example:
’Rover is eating a bone.’
perfect progressive: Expressing the end of an ongoing action. Example: ’Rover
has been eating a bone.’
Format your response as a JSON object with “Properties” and “Explanation” as
the keys. The value of “Properties” should be a list. If none of the
properties are present, return an empty list. Explain your choice in the
“Explanation”. Make your response as short as possible. The example text is
delimited with triple backticks.
Example text: `‘‘‘`While it earlier made attempts to balance its shoddier side
with some interesting reporting, it is now solidly mainstream in the worst
sense.`‘‘‘`
Table 2. Example prompt posted to GPT-4 API for the Aspect feature.

For properties that apply to single words or sentence fragments, we want our
annotators to select the appropriate part of the sentence along with the
property. To that end, we provide a constituency parse tree for the annotator
to select a node which contains the desired fragment, illustrated in Figure 3.
To generate the parse trees, we utilise the SpaCy python library, which is
open source and makes data preparation easily reproducible. Annotating the
data at this level of granularity creates opportunities for a wide range of
NLP tasks which aim to enrich such parse trees for downstream tasks, and in
particular, for architectures involving graphs (Liu and Wu, 2022; Wu et al.,
2023). The parse tree structure does not always allow the annotator to capture
the fragment exactly as they would have done if this limitation wasn’t
imposed. However, utilizing parse trees will make it easier to use our data to
complement research on language representations, which also utilize this
structure, which we believe justifies the above limitation.

To initiate the annotation process, we invited the annotation firm to a
tutorial on the UI, data, and labels. Next, we had three annotators
independently annotate the first 10 sentences. We then reconvened in a virtual
meeting to address inconsistencies and answer any clarifying questions that
may have come up in this phase. Most of the inconsistencies involved assigning
nodes in the parse tree. Some annotators tended to assign nodes closer to the
leaves, while others chose nodes higher up the tree, thus selecting a larger
sentence fragment. We resolved this by instructing the annotators to choose
nodes as close as possible to the leaves, or in other words, the smallest
possible fragment that captures the feature/property under consideration.
Inconsistencies arising from different annotators’ interpretations were
discussed until consensus was reached to help annotators align their
understanding moving forward. We discuss the inter-annotator agreement in
detail in Section 3.1. Annotators were also encouraged to refer to the
Fahnestock (2011) book for additional clarification.

Our budget necessitated a trade-off between quantity and reliability. Using
our alignment exercise described above as an indicator, we anticipated that
inter-annotator agreement would be low, and as such, we would have preferred
to have more annotators. However, this would have significantly reduced the
number of sentences we could have annotated. We believe that three annotators
are an acceptable minimum in our case, and thus, the number of sentences is
driven by this assumption. Consequently, we chose to have three annotators
independently annotate 350 sentences. We took a stratified sample balanced
across the propaganda techniques in the full training data set. As there are
three annotators, GPT-4 will be called three times for each sentence/feature
combination, adding to our cost calculation. For this small subset of
sentences, we chose to use the most powerful zero-shot GPT-4 model available
at the time of writing. To annotate the remainder of the data, we will utilise
the orders of magnitude less expensive GPT-3.5-turbo model.

###  2.5. Inter-annotator agreement

Each sentence can be annotated with one or more of 22 features, where each
feature can have between 3 and 14 properties. Annotators can assign multiple
properties for a given feature. This makes the space of possibilities very
large, and the expectation of all annotators assigning the same combination of
features/properties and selecting the same node in the parse tree would be
unreasonable. Instead, we measure agreement on a per-feature basis, and we
disregard the node selection. An illustrated example is given in Figure 4. In
this example, both annotators assign humans for the Subject choices feature
but select different nodes in the parse tree. For the Verb choices feature,
one annotator assigns nominalization, while the other assigns personification.
We consider Subject choices to be in agreement and Verb choices not in
agreement.

Figure 4. For the Subject choices feature two annotators assign humans but
select different nodes. For the Verb choices feature, one annotator assigns
nominalization, while the other assigns personification. We consider the first
case in agreement and the second one not.

We calculate Krippendorff’s alpha for each feature among all three annotators.
Krippendorff’s alpha is a statistical measure of inter-coder agreement which
considers disagreement one would expect to get by chance. Features which do
not have any labels assigned by any of the annotators are ignored. It’s
important to note that each feature can have only one label in the
Krippendorff statistic. Since our setting allows for multiple labels, we
cannot use this metric to measure partial agreement. For example, an annotator
who assigns both negation and modality to Verb choices is considered in
disagreement with an annotator who only assigns modality. In order to address
this scenario, we also calculated the Jaccard score for each feature. This
gives us a measure of partial agreement. We sum these scores and normalize
them by the number of examples for the given feature. Not all features have
the same number of examples, as not all sentences are annotated with all
features. For example, the Tropes feature may not occur in all 350 sentences
because not all sentences make use of tropes.

For completeness, we also calculate the joint probabilities of exact
agreement, and again, we normalize by the number of examples for the given
feature. We can use these measures to compare the inter-annotator agreement to
intra-GPT consistency, the extent to which the GPT output is consistent given
the same input. The GPT APIs are called three times for each prompt.
Consistency is considered exact when the set of properties for a given feature
is exactly the same in all three responses. For example, given the feature
Verb choices, response 1 = {negation, modality}, response 2 = {negation,
modality}, and response 3 = {negation, modality}. All scores are shown in
Table 3.

Feature | K. | J. | E. | GPT-4 | GPT-3.5
---|---|---|---|---|---
Aspect | 0.364 | 0.573 | 0.320 | 0.980 | 0.568
Emphasis | -0.023 | 0.325 | 0.314 | 0.588 | 0.389
Fig. of argument | 0.330 | 0.101 | 0.094 | 0.925 | 0.429
Fig. of word choice | 0.171 | 0.067 | 0.047 | 0.873 | 0.342
Language of origin | -0.101 | 0.282 | 0.042 | 0.421 | 0.353
Language varieties | 0.074 | 0.613 | 0.126 | 0.059 | 0.157
Lexical fields | 0.057 | 0.269 | 0.036 | 0.364 | 0.129
Modifying clauses | 0.496 | 0.355 | 0.255 | 0.759 | 0.322
Modifying phrases | 0.294 | 0.543 | 0.370 | 0.160 | 0.375
Mood | 0.458 | 0.870 | 0.846 | 0.935 | 0.734
New words | 0.285 | 0.203 | 0.094 | 0.866 | 0.241
Parallelism | 0.035 | 0.009 | 0.009 | 0.957 | 0.384
Phrases b.o. nouns | 0.520 | 0.250 | 0.237 | 0.953 | 0.476
Phrases b.o. verbs | 0.240 | 0.104 | 0.095 | 0.976 | 0.431
Predication | 0.331 | 0.556 | 0.324 | 0.769 | 0.459
Prosody & punct. | -0.003 | 0.000 | 0.000 | 0.000 | 0.067
Sentence arch. | 0.197 | 0.183 | 0.106 | 0.575 | 0.412
Series | 0.332 | 0.089 | 0.089 | 1.000 | 0.412
Subject choices | 0.351 | 0.426 | 0.245 | 0.652 | 0.331
Tense | 0.477 | 0.648 | 0.447 | 0.900 | 0.353
Tropes | 0.217 | 0.107 | 0.067 | 0.853 | 0.255
Verb choices | 0.407 | 0.281 | 0.182 | 0.678 | 0.476
Table 3.  Inter-annotator agreement scores for all features, where
K=Krippendorff’s alpha, J=Jaccard, E=Joint probability of exact agreement,
GPT-4=Joint probability of intra-GPT-4 consistency, and GPT-3.5=Joint
probability of intra-GPT-3.5 consistency. Figure 5. Distribution of intra-
GPT/inter-annotator consistency scores across all features. GPT-3.5 was
parameterized with a temperature setting of 1.0 and 0.2.

##  3\. Experiments & Results

Interestingly, both intra-GPT-3.5 and intra-GPT-4 consistency is higher than
inter-annotator agreement, illustrated in Figure 5. Consistency alone,
however, does not tell us anything about accuracy. To measure accuracy, we
sample 30 examples in each feature where all three annotators agree on the
same list of properties and manually produce ground truth labels for those
examples. Human accuracy, excluding a few exceptions, ranges from 70-100%,
whereas GPT-3.5 performs very poorly. GPT-4 performs five times better than
GPT-3.5 but at ten times the cost. The results are shown in Table 4 in the V1
columns. In addition, reducing the temperature
parameter222https://platform.openai.com/docs/api-reference/chat/create#chat-
create-temperature significantly increases the consistency of GPT-3.5. This is
as expected. Notably, consistency in the GPT responses is not always
correlated with accuracy. Thus, simply taking a majority vote is not a useful
strategy in our case. To improve accuracy, we perform the following
experiments.

###  3.1. Prompt engineering

Our initial zero-shot prompts, which were simply the same instructions
provided to the human annotators, are ineffective. We asked GPT to provide a
list of applicable properties in its response and an explanation of its
choices. We scrutinize each of the approximately 600 (30 sentences * 22
features) GPT-generated explanations to understand where GPT is going wrong
and thereby aim to improve the prompts. We categorise the GPT errors into the
following types:

  * •

confounding \- GPT returns some property but uses a definition of a different
property in its explanation.

  * •

over-generalizing \- GPT returns some property but makes up its own definition
of that property rather than the definition provided. Usually, this GPT
definition is broader than the one provided and is out of context. This is
despite being instructed to use the definitions and context provided.

  * •

hallucinating \- GPT returns an explanation based on elements not present in
the given sentence.

  * •

greedy answering \- GPT returns properties when none of the properties are
applicable. This is despite being instructed to return an empty list in such
cases.

  * •

other errors \- GPT returns properties and explanations that are not plausible
or are simply incorrect.

We can solve for confounding and greedy answering by modifying the prompts to
ask about one property at a time. Instead of providing a list of properties
and definitions for GPT to choose from, we provide a single property and its
definition and ask GPT if it is present. For each property we ask for a yes/no
answer and an explanation. Given that majority voting does not improve
accuracy, we set the temperature to 0.0 for maximum consistency and only
prompt GPT once to reduce cost. This modification improves performance of
GPT-3.5 on some features but does not change the overall performance of GPT-4.
Results are listed in Table 4 in the V2 columns. We do not anticipate any
significant improvements with further prompt engineering as per Webson and
Pavlick (2022), who show that “models learn just as fast with many prompts
that are intentionally irrelevant or even pathologically misleading as they do
with instructively ‘good’ prompts. […] these results contradict a hypothesis
commonly assumed in the literature that prompts serve as semantically
meaningful task instructions.” That is to say, LLMs do not “understand”
instructions in the same way that people do. Thus, prompt engineering efforts
should focus on deconstructing the LLM rather than writing “better”
instructions as perceived by humans. How to deconstruct the LLM is outside the
scope of this study.

###  3.2. Fine-tuning

We construct three types of data sets for fine-tuning GPT-3.5 as follows:

  1. (1)

Small size, high quality, single data-set \- For each property of a given
feature, we find one positive and one negative example from the sub-set of the
data where we had previously generated ground truth labels. The pairings
comprise the prompt (SYSTEM and USER message as per V2) and an answer
(ASSISTANT message), a JSON object with ‘Answer’ and ‘Explanation’ keys and
their values. This is the most labour-intensive data set as it requires
verifying that the answer is correct and providing an explanation.

  2. (2)

Medium size, medium/high quality, single data-set \- For each property of a
given feature we find 25 positive and 25 negative examples from the sub-set of
the data which was annotated by humans. The pairings comprise the prompt
(SYSTEM and USER message as per V2), and an answer (ASSISTANT message) which
is a JSON object. We do not include an explanation. While omitting the
explanations reduces the labour intensity, the number of examples nearly
cancels this benefit.

  3. (3)

Large size, low quality, multiple data sets \- We utilize all human-annotated
data minus the examples for which we generated ground-truth labels. For each
sentence, feature and property, we generate positive examples by taking the
majority vote of the human annotators and negative examples using sentences
where none of the annotators applied the property under consideration. Each
feature ends up with its own data set of size approx. 300 x #properties. For
example, the Aspect feature has 4 possible properties, thus the size of this
data-set is approx. 1,200 examples. This is the least labour-intensive effort
as it is completely automated. The downside is that the majority vote is not
always correct.

To keep costs and labour down, we experiment with 2-4 features before
committing to generating the full data sets. We fine-tune GPT-3.5 using the
OpenAI web interface and evaluate the results on the held-out ground-truth
data. The small and medium data sets produce rather poor results. This runs
contrary to the recommendation from OpenAI of 50-100 high-quality
examples.333https://platform.openai.com/docs/guides/fine-tuning/preparing-
your-dataset We abandon these experiments to avoid incurring unnecessary
costs. The accuracy of the fine-tuned GPT-3.5 on the large, albeit noisy data
sets, is, overall, on par with GPT-4, and even slightly higher for some
features. Results are shown in the FT column of Table 4.

| GPT V1 | GPT V2 | FT
---|---|---|---
Feature | H. | 3.5 | 4 | 3.5 | 4 | 3.5
Aspect | 0.94 | 0.14 | 0.46 | 0.31 | 0.44 | 0.5
Emphasis | 0.6 | 0.4 | – | 0.0 | 0.2 | 0.13
Fig. of argument | 0.83 | 0.17 | 0.67 | 0.53 | 0.57 | 0.47
Fig. of word choice | 0.3 | 0.07 | 0.4 | 0.03 | 0.17 | 0.37
Language of origin* | – | – | – | – | – | –
Language varieties | 0.74 | 0.01 | 0.54 | 0.04 | 0.37 | 0.7
Lexical fields | 0.38 | 0.2 | 0.2 | 0.17 | 0.1 | 0.07
Modifying clauses | 0.69 | 0.10 | 0.62 | 0.52 | 0.66 | 0.62
Modifying phrases* | – | – | – | – | – | –
Mood | 1.0 | 0.76 | 0.48 | 0.21 | 0.48 | 0.21
New words | 0.84 | 0.16 | 0.08 | 0.16 | 0.4 | 0.36
Parallelism | 0.95 | 0.1 | 0.81 | 0.81 | 0.84 | 0.84
Phrases b.o. nouns | 1.0 | 0.03 | 0.97 | 0.74 | 0.84 | 0.9
Phrases b.o. verbs | 0.72 | 0.0 | 0.67 | 0.8 | 0.67 | 0.97
Predication | 0.9 | 0.13 | 0.4 | 0.17 | 0.2 | 0.57
Prosody & punct.** | – | – | – | – | – | –
Sentence arch. | 0.97 | 0.0 | 0.73 | 0.53 | 0.73 | 0.6
Series | 1.0 | 0.0 | 0.83 | 0.87 | 0.96 | 0.83
Subject choices | 0.87 | 0.13 | 0.23 | 0.5 | 0.47 | 0.1
Tense | 0.73 | 0.1 | 0.4 | 0.5 |  0.63 | 0.6
Tropes | 0.7 | 0.09 | 0.31 | 0.16 | 0.44 | 0.47
Verb choices | 0.73 | 0.23 | 0.2 | 0.27 | 0.53 | 0.77
All features | 0.83 | 0.11 | 0.56 | 0.39 | 0.51 | 0.53
Table 4.  Accuracy of annotations where all annotators agree. The highest
scores, excluding human annotators, are highlighted in bold font. In GPT
Version 1 (V1) the prompts to GPT include all properties of a given feature
and we ask GPT to choose zero or more applicable properties. In GPT Version 2
(V2) columns, each property gets its own prompt, and GPT is asked to provide a
yes/no answer as to whether the property is applicable. The FT column contains
scores, where GPT-3.5 was fine-tuned on the majority, vote human annotations,
and each property receives its own prompt. Features marked with an asterisk
are extracted using existing NLP techniques and do not need to be annotated.
Features marked with double asterisks are too sparse to be useful.

##  4\. Discussion

We aim to make the best of the noisy data obtained through human expert
annotations by utilizing the state-of-the-art large language models, which
have recently dominated NLP research. Identifying linguistic features and
rhetorical devices in natural language is not trivial for humans, as is borne
out of the low inter-annotator agreement on our task. However, it seems this
task would be well suited to an LLM for the following reasons: it does not
require reasoning, it does not require any highly specialized domain knowledge
that wouldn’t have been available in the data the LLM was trained on, nor does
it require access to facts or events which would have occurred post the
training date. The one thing that LLMs are particularly good at, pattern
recognition and general ‘knowledge’ (arising from the vast amounts of publicly
available data they were trained on) would suggest that they are ideally
suited to our task. This aligns with the generally accepted premise that
people are good at logical reasoning while machines are good at finding
patterns in large volumes of data.

Our research highlights another interesting, perhaps less conspicuous,
difference between human annotators and GPT. What we found was that expert
human inter-annotator agreement is low when it comes to non-trivial linguistic
feature identification tasks. Conversely, GPT, when called with a low-
temperature setting, achieves consistency levels around 80% (see Figure 5).
However, when humans do agree, they are correct, overall, 83% of the time,
while the maximum GPT accuracy we were able to achieve is only 56% (see Table
4). This suggests that we can confidently rely on the established majority
voting mechanism, or ‘wisdom of the crowd’ approach, when it comes to
traditional human annotations, but not when it comes to GPT.

This brings us to our next point, which is that every setting appears to
demand its own exploration, and the techniques that work in one scenario do
not always translate to another. In particular, we found that providing a
small set of carefully crafted high-quality data points for fine-tuning does
not necessarily increase the accuracy of GPT-3.5, as discussed in the previous
section. In addition to different settings demanding different approaches, on
a more granular level, not all features respond similarly to the same
techniques. For example, ‘Mood’ and ‘Emphasis’ accuracy was highest using the
original prompts, which performed poorly for all the other features. The
‘Aspect’, ‘New words’, and ‘Subject choices’ features are relatively easy for
humans but particularly difficult for GPT. And fine-tuned GPT outperforms
humans on ‘Phrases built on nouns’. These insights enable us to make
cost/benefit decisions about utilizing human annotators vs GPT, prompting
technique, or fine-tuning.

We have run our experiments on GPT-3.5 and GPT-4 for comparison purposes, and
no doubt the next version of GPT will be even better. However, our goal is to
find ways to optimize not only for accuracy but also for cost. Our fine-tuned
(FT) models perform on par with GPT-4 at 10x less the price. Since we intend
to annotate the PTC corpus, which comprises approx. 20,000 sentences, we can
estimate that using our FT models would cost approx. $2,000444This is a rough
estimate based on our spending to date. A more precise estimate can be
calculated by estimating the number of tokens input and output to and from
GPT., GPT-4 would cost $20,000, and human annotators would cost $90,000 (at
$1.50 per sentence, per expert, using three experts). These orders of
magnitude differences can be critical to a researcher’s budget and ability to
proceed.

##  5\. Related Work

Since the release of ChatGPT in November 2022, researchers (Gilardi et al.,
2023; He et al., 2023; Yu et al., 2023; Pangakis et al., 2023; Törnberg, 2023;
Kuzman et al., 2023; Thapa et al., 2023; Ding et al., 2023; Bansal and Sharma,
2023; Feng and Narayanan, 2023; Alizadeh et al., 2023; Savelka, 2023; Huang et
al., 2023; Tekumalla and Banda, 2023; Zhang et al., 2023) have explored using
it as a text annotation tool to reduce reliance on expensive human annotated
data for downstream NLP tasks. Various prompt engineering strategies have been
proposed including zero-shot, few-shot, chain-of-thought (CoT), and active
learning.

Wang et al. (2021) first explore using GPT-3 (the model upon which ChatGPT was
built) for text annotation and show that data labelled by GPT-3 achieves the
same performance on several NLP and NLU tasks as human labelled data at a
fraction of the cost. Furthermore, models trained using the GPT-3 labelled
data outperform GPT-3 on their respective tasks. However, the authors suggest
that labeling by GPT-3 should only be used in low stakes scenarios due to its
now well known tendencies to “hallucinate” (Ji et al., 2023), and/or
perpetuate undesirable biases. Reiss (2023) also cautions against using
ChatGPT-3.5 for zero-shot annotation tasks, pointing out the intra-coder
inconsistencies (e.g. the inconsistencies among ChatGPT-3.5 responses given
identical prompts). To improve the accuracy of ChatGPT annotations, Reiss
(2023) recommends using majority voting based on three, or ideally more,
repetitions of each single input. We show that this does not improve results
in our setting. Taking this a step further, Davani et al. (2022) show that
taking a majority vote before training is not as effective as training a
multi-task model using all annotators (ChatGPT repetitions) and subsequently
predicting the final label as an aggregation of each task output. Similar to
Reiss (2023), Pangakis et al. (2023) point out that while “LLM performance for
text annotation is promising, it is highly contingent on both the data set and
the type of annotation task, which reinforces the necessity to validate on a
task-by-task basis.” The authors propose an iterative strategy that evaluates
the LLM annotations on a small subset of human expert annotated data until
acceptable performance is reached. We follow a similar strategy.

Despite these potential pitfalls, others have shown promising results. Kuzman
et al. (2023) compare a fine-tuned XLM-RoBERTa model against ChatGPT-3 for
genre identification, a task deemed difficult for human annotators, where the
inter-annotator agreement is low. While this is not strictly an annotation
comparison, we can think of the GPT zero-shot classifier as an annotator where
GPT achieved an accuracy score of 72% against the human-annotated test set.
GPT outperformed the fine-tuned model by 5 points on an out-of-distribution
data set given a similar task. In contrast, He et al. (2023) propose a method
which surpasses human annotators on some tasks and reaches near parity on
others. Unlike the zero-shot scenario, the proposed method involves few-shot
chain-of-thought (CoT) (Wei et al., 2023) prompting. Uniquely, GPT-3.5 is used
to generate the explanations, which are then used as few-shot examples in the
final prompt. Having GPT generate the explanations is justified by attributing
an emergent reasoning capability to GPT. In contrast, we provide expertly
written definitions and real-world examples in our prompts. However, when we
examine the GPT-3.5 explanations in our setting, we cannot identify any
emergent reasoning capabilities - see section 3.1 for details.

Alizadeh et al. (2023) extend the research into LLM annotation beyond ChatGPT
to open-source (OS) models. They show that while ChatGPT generally outperforms
human annotators and OS models on most tasks, OS models outperform ChatGPT on
some tasks and approach ChatGPT performance on others. OS models also
outperform human annotators at a lower cost, with all the open-source benefits
such as transparency, reproducibility, and data privacy.

Bansal and Sharma (2023) take an active learning approach, where the LLM is
utilized to annotate the most informative inputs based on a novel conditional
informativeness sampling strategy, thereby reducing the cost of annotation
even further while improving the performance of downstream NLP models trained
on the LLM annotated data.

It’s clear from the literature that the ideal prompt engineering methods and
hyper-parameters vary across tasks, especially when those tasks are complex or
particularly difficult for human annotators. Our work focuses on the scenario
where the task is unique and complex, and human annotator agreement is low,
and therefore merits its own study.

##  6\. Conclusion

In this study, we have identified linguistic features based on the premise
that propaganda techniques can be decomposed into rhetorical devices and
linguistic characteristics, as seen in the use of language for persuasion.
Given our limited budget, we employed three expert annotators to label 350
sentences from the PTC corpus, to the best of our knowledge, the most granular
propaganda technique corpus available. With the help of our small human-
annotated data set, we investigated whether a large language model like GPT
can be utilized to annotate the remaining 2̃0,000 sentences. The impetus for
our experiments and solutions is driven by an acutely limited budget, thus the
solutions we propose are necessarily a trade-off between accuracy and cost.

In addition, we find certain unexpected differences between human annotators
and GPT. In particular, we discovered that majority voting of GPT responses
does not improve accuracy, and fine-tuning on a small but high-quality data-
set is also not an effective strategy. Both of these findings are contrary to
usual practices in machine learning, and we must emphasize again that each
setting demands its own exploration.

Finally, we fine-tune GPT-3.5 using the human-annotated data set by generating
examples for each feature/property combination by taking a majority vote for
the positive examples and the absence of a given label from all annotators as
negative examples. Even with this noisy data, we achieve an accuracy on par
with GPT-4 at 10x less the cost. We make all our code, including our web-based
annotation application, the properties and their definitions and examples, and
the GPT prompts available in
GitHub.555https://github.com/kyleiwaniec/rhetoric-annotation

##  7\. Limitations and Future Work

Our biggest limitation is our budget. However, even without this constraint,
the task is not trivial, even for expert annotators. Majority voting by itself
still leads to a lot of noise, and we intend to perform several more
iterations on the 350 sentences with our human annotators to reach a higher
agreement. We performed our experiments on GPT as it is the subject of
considerable attention. In the future, we plan to test other models. In
particular, LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023) are
open source and available in different sizes, making them especially suitable
for analyzing the trade-off between cost and accuracy.

Furthermore, the feature set we created is based on literature, but this, too,
could benefit from additional research. We have yet to see if these features
improve the detection of propaganda techniques. While this is outside the
scope of this paper, it is the next natural step. By making these available,
we hope to aid other researchers in pursuing the goal of interpretable
propaganda detection.

###### Acknowledgements.

This publication has emanated from research conducted with the financial
support of Science Foundation Ireland under Grant number 18/CRT/6183. For the
purpose of Open Access, the author has applied a CC BY public copyright
licence to any Author Accepted Manuscript version arising from this
submission.

## References

  * (1)
  * Alizadeh et al. (2023) Meysam Alizadeh, Maël Kubli, Zeynab Samei, Shirin Dehghani, Juan Diego Bermeo, Maria Korobeynikova, and Fabrizio Gilardi. 2023.  Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text-Annotation Tasks.  arXiv:2307.02179 (July 2023).  https://doi.org/10.48550/arXiv.2307.02179 arXiv:2307.02179 [cs].
  * Bansal and Sharma (2023) Parikshit Bansal and Amit Sharma. 2023.  Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost.  arXiv:2306.15766 (June 2023).  https://doi.org/10.48550/arXiv.2306.15766 arXiv:2306.15766 [cs].
  * Bryant (1953) Donald C. Bryant. 1953.  Rhetoric: Its functions and its scope.  _Quarterly Journal of Speech_ 39, 4 (1953), 401–424.  https://doi.org/10.1080/00335635309381908
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-Grained Analysis of Propaganda in News Articles. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ _(EMNLP-IJCNLP 2019)_. Hong Kong, China.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022.  Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations.  _Transactions of the Association for Computational Linguistics_ 10 (Dec. 2022), 92–110.  https://doi.org/10.1162/tacl_a_00449
  * Ding et al. (2023) Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023.  Is GPT-3 a Good Data Annotator?. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 11173–11195.  https://doi.org/10.18653/v1/2023.acl-long.626
  * Dubremetz and Nivre (2018) Marie Dubremetz and Joakim Nivre. 2018.  Rhetorical Figure Detection: Chiasmus, Epanaphora, Epiphora.  _Frontiers in Digital Humanities_ 5 (2018).  https://www.frontiersin.org/articles/10.3389/fdigh.2018.00010
  * Fahnestock (2011) Jeanne Fahnestock. 2011.  _Rhetorical style: The uses of language in persuasion_.  OUP USA.
  * Feng and Narayanan (2023) Tiantian Feng and Shrikanth Narayanan. 2023.  Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting.  arXiv:2309.08108 (Sept. 2023).  https://doi.org/10.48550/arXiv.2309.08108 arXiv:2309.08108 [cs, eess].
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ 120, 30 (July 2023), e2305016120.  https://doi.org/10.1073/pnas.2305016120
  * He et al. (2023) Xingwei He, Zhenghao Lin, Yeyun Gong, A.-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023.  AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators.  https://arxiv.org/abs/2303.16854v1
  * Huang et al. (2023) Fan Huang, Haewoon Kwak, and Jisun An. 2023.  Is ChatGPT Better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech. In _Companion Proceedings of the ACM Web Conference 2023_ (Austin, TX, USA) _(WWW ’23 Companion)_. Association for Computing Machinery, New York, NY, USA, 294–297.  https://doi.org/10.1145/3543873.3587368
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of Hallucination in Natural Language Generation.  _ACM Comput. Surv._ 55, 12, Article 248 (mar 2023), 38 pages.  https://doi.org/10.1145/3571730
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  arXiv:2310.06825 [cs.CL]
  * Kuzman et al. (2023) Taja Kuzman, Igor Mozetič, and Nikola Ljubešić. 2023.  ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification.  https://arxiv.org/abs/2303.03953v2
  * Liu and Wu (2022) Bang Liu and Lingfei Wu. 2022.  _Graph Neural Networks in Natural Language Processing_.  Springer Nature, Singapore, 463–481.  https://doi.org/10.1007/978-981-16-6054-2_21
  * Longo et al. (2024) Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, Javier Del Ser, Riccardo Guidotti, Yoichi Hayashi, Francisco Herrera, Andreas Holzinger, Richard Jiang, Hassan Khosravi, Freddy Lecue, Gianclaudio Malgieri, Andrés Páez, Wojciech Samek, Johannes Schneider, Timo Speith, and Simone Stumpf. 2024.  Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions.  _Information Fusion_ 106 (2024), 102301.  https://doi.org/10.1016/j.inffus.2024.102301
  * Mitrović et al. (2020) Jelena Mitrović, Cliff O’Reilly, Randy Harris, and Michael Granitzer. 2020.  Cognitive Modeling in Computational Rhetoric: Litotes, Containment and the Unexcluded Middle:. In _Proceedings of the 12th International Conference on Agents and Artificial Intelligence_. SCITEPRESS - Science and Technology Publications, Valletta, Malta, 806–813.  https://doi.org/10.5220/0009104908060813
  * Pangakis et al. (2023) Nicholas Pangakis, Samuel Wolken, and Neil Fasching. 2023.  Automated Annotation with Generative AI Requires Validation.  https://arxiv.org/abs/2306.00176v1
  * Reiss (2023) Michael V. Reiss. 2023.  Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark.  arXiv:2304.11085 (April 2023).  https://doi.org/10.48550/arXiv.2304.11085 arXiv:2304.11085 [cs].
  * Savelka (2023) Jaromir Savelka. 2023.  Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts. In _Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law_ (¡conf-loc¿, ¡city¿Braga¡/city¿, ¡country¿Portugal¡/country¿, ¡/conf-loc¿) _(ICAIL ’23)_. Association for Computing Machinery, New York, NY, USA, 447–451.  https://doi.org/10.1145/3594536.3595161
  * Tekumalla and Banda (2023) Ramya Tekumalla and Juan M Banda. 2023.  Leveraging Large Language Models and Weak Supervision for Social Media data annotation: an evaluation using COVID-19 self-reported vaccination tweets. In _International Conference on Human-Computer Interaction_. Springer, 356–366.
  * Thapa et al. (2023) Surendrabikram Thapa, Usman Naseem, and Mehwish Nasim. 2023.  From humans to machines: can ChatGPT-like LLMs effectively replace human annotators in NLP tasks. In _Workshop Proceedings of the 17th International AAAI Conference on Web and Social Media_. Association for the Advancement of Artificial Intelligence (AAAI).  https://doi.org/10.36190/2023.15
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Törnberg (2023) Petter Törnberg. 2023.  ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning.  https://arxiv.org/abs/2304.06588v1
  * Wang et al. (2021) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021.  Want To Reduce Labeling Cost? GPT-3 Can Help. In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 4195–4205.  https://doi.org/10.18653/v1/2021.findings-emnlp.354
  * Ward et al. (2022) Katrina J. Ward, Hamilton Link, Kiril Avramov, and Jean Goodwin. 2022.  _Identifying Disinformation Using Rhetorical Devices in Natural Language Models_.  Number SAND2022-13730.  https://doi.org/10.2172/1891194
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do Prompt-Based Models Really Understand the Meaning of Their Prompts?. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 2300–2344.  https://doi.org/10.18653/v1/2022.naacl-main.167
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  arXiv:2201.11903 (Jan. 2023).  https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs].
  * Wu et al. (2023) Lingfei Wu, Yu Chen, Kai Shen, Xiaojie Guo, Hanning Gao, Shucheng Li, Jian Pei, and Bo Long. 2023.  Graph Neural Networks for Natural Language Processing: A Survey.  _Foundations and Trends® in Machine Learning_ 16, 2 (2023), 119–328.  https://doi.org/10.1561/2200000096
  * Yu et al. (2023) Danni Yu, Luyang Li, Hang Su, and Matteo Fuoli. 2023.  Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apologies.  _International Journal of Corpus Linguistics_ (Dec. 2023).  https://www.benjamins.com/catalog/ijcl
  * Yu et al. (2021) Seunghak Yu, Giovanni Da San Martino, Mitra Mohtarami, James Glass, and Preslav Nakov. 2021.  Interpretable Propaganda Detection in News Articles. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)_ , Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., Held Online, 1597–1605.  https://aclanthology.org/2021.ranlp-1.179
  * Zhang et al. (2023) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023.  LLMaAA: Making Large Language Models as Active Annotators.  arXiv:2310.19596 (Oct. 2023).  https://doi.org/10.48550/arXiv.2310.19596 arXiv:2310.19596 [cs].

##  Appendix A Features

Features | Properties
---|---
Word choices |
Figures of word choice |  [’agnominatio’, ’metaplasms’, ’polyptoton’, ’ploce’, ’anatanaclasis’, ’synonyms’, ’rhetorical conditional’, ’correctio’, ’emphasis’]
Language of origin |  [’Old English Core’, ’Norman French’, ’Latin/Greek’]
Language varieties |  [’correctness’, ’clarity’, ’forcefulness’, ’low’, ’middle’, ’high’, ’dialects/registers’, ’register shift’, ’cliches/idioms’, ’maxims/proverbs’, ’allusions’]
Lexical and semantic fields |  [’prototype’, ’abstract’, ’concrete’, ’indefinite/thesis’, ’definite/hypothesis’]
New words and changing uses |  [’foreign borrowing’, ’compounds’, ’prefixes/suffixes’, ’clipping’, ’blends’, ’conversions’, ’catachresis’, ’acronyms’, ’proper nouns to common nouns’, ’analogy’, ’fabrication’, ’onomatopoeia’, ’taboo deformation’, ’doubling’]
Tropes |  [’synecdoche’, ’metonymy’, ’antonomasia’, ’metaphor’, ’allegory’, ’simile’, ’full analogies’, ’irony’, ’hyperbole’, ’litotes’, ’amphiboly’, ’paradox’, ’paralepsis/praeteritio’, ’euphemism’]
Sentences |
Aspect |  [‘simple’, ‘perfect’, ‘progressive’, ‘perfect progressive’]
Emphasis |  [‘by position’, ‘by sentence role’, ‘from inversions’, ‘combinations’]
Figures of argument |  [‘strategic repetition’, ‘antithesis ’, ‘antimetabole’, ‘defintion as figure’, ‘induction’, ‘eduction’]
Modifying clauses |  [‘subordinate’, ‘conditional’, ‘comparitive’, ‘adjective’, ‘noun’]
Modifying phrases |  [‘prepositional phrases’, ‘single word modifiers’, ‘multiplying and embedding modifiers’]
Mood |  [‘indicative’, ‘subjunctive’, ‘exclamatory’, ‘interrogative’, ‘imperative’, ‘optative’]
Parallelism |  [‘in syllables (isocolon)’, ‘in stress patterns (iambic parameter)’, ‘grammatical structure (parison)’, ‘comparison (parallelism in argument)’]
Phrases built on nouns |  [‘appositives’, ‘absolute construction’, ‘resumptive modifier’, ‘summative modifier’]
Phrases built on verbs |  [‘participal phrases’, ‘inifinitive phrases’]
Predication |  [‘active’, ‘stative’, ‘compound’]
Prosody and punctuation |  [‘innotation’, ‘stress’, ‘rhythm’]
Sentence architecture |  [‘left branching’, ‘middle branching’, ‘right branching’, ‘Periodic sentences’, ‘loose sentences’]
Series |  [‘bracketing’, ‘order (incrementum)’, ‘gradatio’, ‘polysyndenton’, ‘asyndenton’]
Subject choices |  [‘humans’, ‘rhetorical participants’, ‘things’, ‘abstractions’, ‘concepts’, ‘slot fillers’]
Tense |  [‘present’, ‘past’, ‘future’, ‘progression’]
Verb choices |  [‘negation’, ‘modality’, ‘nominalization’, ‘personification’, ‘multiplication’]
Table 5.  Features and their properties extracted from Rhetorical Style: The
Uses of Language in Persuasion (Fahnestock, 2011).


[1,2]Douglas J. Jerolmack

1]Department of Earth and Environmental Sciences, University of Pennsylvania,
Philadelphia, 19104, PA, USA

2]Department of Mechanical Engineering and Applied Mechanics, University of
Pennsylvania, Philadelphia, 19104, PA, USA

# Origins of complexity in the rheology of Soft Earth suspensions

Shravan Pradeep<EMAIL_ADDRESS>Paulo E. Arratia
<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [

###### Abstract

When wet soil becomes fully saturated by intense rainfall, or is shaken by an
earthquake, it may fluidize catastrophically [1]. Sand-rich slurries are
treated as granular suspensions, where failure is related to an unjamming
transition and friction is controlled by particle concentration [2] and pore
pressure [3]. Mud flows are modeled as complex fluids akin to gels, where
yielding and shear-thinning behaviors arise from inter-particle attraction and
clustering [4, 5]. Here we show that the full range of complex flow behaviors
previously reported for natural debris flows – dense slurries of soil and
water – can be reproduced with suspensions made only of water, sand and clay.
Going from sand-rich to clay-rich suspensions, we observe a continuous
transition from brittle (Coulomb-like) to ductile (perfect plastic) yielding
that is characterized by (i) a decrease in the stress drop associated with
yielding, (ii) an increase in the shear-thinning exponent from zero to 1/2,
and (iii) a decrease in the normal force. We propose a general constitutive
relation for soil suspensions, with a particle rearrangement time that is
controlled by yield stress. Our experimental results are supported by models
for yielding of idealized amorphous solids [6, 7], suggesting that the
paradigm of non-equilibrium phase transitions can help to understand and even
predict complex behaviors of Soft Earth suspensions.

###### keywords:

dense suspensions, soft matter, geophysical flows, Soft Earth Geophysics

Earth’s surface is covered in heterogeneous and soft particulate material that
we call soil. To first order, we may consider Soft Earth suspensions – and
their associated geophysical flows – to be composed of three ingredients:
frictional particles (sand/silt), cohesive particles (clay, fine silt, organic
material), and water [8]. Apparently solid soil can suddenly lose rigidity
when stressed; for example, earthquake-induced liquefaction, the collapse of
earth dams, or the failure of water-soaked hillsides that forms landslides and
debris flows [1, 9]. Whether and how soil yields and flows under environmental
loads is exquisitely sensitive to the relative proportions of the three main
ingredients [10]. Here we focus on debris flows: dense, viscous and fast-
flowing suspensions of soil. With the increasing frequency and intensity of
extreme weather and wildfire due to climate change, the hazard posed by debris
flows is growing [11]. Debris flows are complex and transient phenomena, where
shear stress, flow speed, particle size and water content co-evolve [3].
Controlled laboratory tests are useful for taming this complexity by fixing
variables, allowing determination of the constitutive relations among shear
stress ($\tau$), solids volume fraction ($\phi$) and shear rate
($\dot{\gamma}$). Measured ‘flow curves’ are typically fit with the
phenomenological Hershel-Bulkley equation, $\tau=\tau_{y}+k\dot{\gamma}^{n}$,
where $\tau_{y}$ is a yield stress below which flow cannot occur, $n$ is the
flow index and $k$ is an empirical constant. Previous studies have established
that debris flows are typically shear thinning ($n<1)$ and have a significant
yield stress ($\tau_{y}>0$), and that these properties are sensitive to
particle concentration [12, 13]. Even under idealized conditions, however,
laboratory studies have not provided a clear picture on how debris-flow
material composition controls flow behavior. The physical meaning of $k$ and
$n$ are ambiguous, which limits extrapolation of results beyond the
laboratory.

Some physical insight has been gained by classifying debris flows into two
end-member types based on soil composition: “Granular” and “Cohesive” [14,
15]. Granular debris flows are composed mostly of frictional, cohesionless
soil – i.e, sand and larger particles. Our previous work demonstrated that
weakly cohesive debris-flow materials from the California Coast Range could be
described by constitutive relations developed for idealized granular
suspensions [16], by accounting for two material-specific quantities
associated with jamming of granular media: the jamming fraction ($\phi_{m}$)
above which flow cannot occur, and the yield stress [2]. Using these
quantities to normalize the data, we found the resultant flow behavior of
debris-flow materials was well described by a dimensionless Bingham model:
elastic-like behavior at low shear rate/stress, viscous flow behavior (i.e.,
$n=1$) at high shear rate/stress, and a rapid transition between them
consistent with discontinuous, Coulomb-like failure (Fig. 1) This picture is
incompatible, however, with the Cohesive debris-flow end member. Clay-rich
debris-flow materials from the French Alps were shown to yield gradually and
continuously, with $n$ much smaller than one [4] – flow curves cannot be fit
with the same Bingham model (Fig. 1). Yielding dynamics of clay suspensions
are strongly related to attractive interaction among particles, and these
suspensions may exhibit solid-like behavior even at low $\phi$ [17, 18, 19].
Clay suspensions also exhibit hysteresis and thixotropy, that change with the
strength of inter-particle attraction (e.g., particle surface charge and
solution ionic strength) [20, 17]. All of these behaviors are reminiscent of
gels, in which inter-particle attraction and clustering – rather than friction
and jamming – determine yield and flow properties [20].

Figure 1: Debris flows and rheological phase space. a, Debris flow deposit at
Montecito, in the California Coast Range (2018; Photo Credit: United States
Geological Survey). b, Non-dimensionalized flow curves of debris-flow
materials from the French Alps (blue circles [4]) and the California Coast
Range (red circles [2]) on a scaled shear stress ($\tau^{\star}$) - shear rate
($\Gamma$) plot. A simplified Bingham model, $\tau^{\star}=1+\Gamma$, is fit
to the California Coast Range data but cannot fit the French Alps data. We
hypothesize that stress transmission in the former is dominated by granular
friction, while in the latter it is governed by cohesion. Cohesive particles
are expected to impose a rotational constraint on motion, while frictional
particles impart a sliding constraint [21]. c, Proposed phase space for Soft
Earth suspensions defined using scaled variables: interparticle attraction
strength ($\chi$) on the y-axis, and non-dimensional shear rate ($\Gamma$) on
the x-axis. Low $\chi$ suspensions are expected to exhibit brittle yielding;
as $\chi$ increases, a transitional plastic regime emerges and grows as
yielding becomes increasingly ductile.

Motivated by industrial applications, there is increasing interest in
examining the rheologic behaviors of granular materials suspended in complex
fluids such as gels [22, 23, 24, 25] and emulsions [26]. While progress has
been made in extending constitutive models to such complex mixtures [26, 25],
no study has probed the transition from granular suspension to gel. Some
insight on the possible nature of this transition can be gained by examining
yielding in other amorphous solids. A numerical study discovered that a
transition from brittle yielding typical of “hard” glass materials, to ductile
yielding common in “soft” glassy materials, can arise within an amorphous
solid by changing its degree of annealing – i.e., the initial stability of the
material [6]. Theoretical analysis of elasto-plastic models – which are
commonly used to describe soft glassy materials like emulsions, gels and foams
– has constrained the value $n\approx 1/2$ in the vicinity of yielding [27,
7]. The reported rheologies of debris-flow materials summarized earlier
indicate that there must be a transition from frictional to cohesive control,
as the relative proportion of clay to sand is increased in a suspension. Flow
curves suggest that this corresponds to a transition from brittle, Coulomb-
type failure to ductile, plastic failure (Fig. 1b,c). Here we build a minimal
experimental system to examine this transition: suspensions of water,
frictional sand and cohesive kaolin clay. Carefully controlled rheologic tests
reveal a continuous transition from brittle to ductile yielding as the
relative proportion of clay to sand is increased. The transition appears to be
related to percolation of cohesive-particle networks. Our model Soft Earth
suspensions capture the full range of previously reported debris-flow
rheologies, and are used to build a physically-based constitutive relation
with parameters that are related to material properties.

## Developing a generic constitutive relation

The dimensionless Bingham model that we previously developed for weakly
cohesive debris-flow slurries [2] has the form $\tau^{\star}=1+\Gamma$, where
the dimensionless shear stress is $\tau^{\star}\equiv\tau/\tau_{y}$, and the
dimensionless viscous stress $\Gamma\equiv\eta\dot{\gamma}/\tau_{y}$ is the
inverse Bingham number. Note that $\Gamma$ can also be interpreted as a
dimensionless shear rate $\Gamma=t_{\mu}\dot{\gamma}$, where
$t_{\mu}\equiv\eta/\tau_{y}$ is a material relaxation timescale [4]. In this
model, elastic deformation produces a constant $\tau=\tau_{y}$ at low
$\dot{\gamma}$, while viscous dissipation $\eta\dot{\gamma}$ becomes dominant
for large $\dot{\gamma}$. The abrupt transition from solid to viscous-flow
states in this model corresponds to a brittle yielding behavior expected for
frictional or highly annealed materials (Fig. 1c). We found that the effective
viscosity $\eta$ followed the rheology of dense granular suspensions, i.e.,
$\eta$ depends on the distance from jamming, $\eta=\eta(\Delta\phi)$, where
$\Delta\phi\equiv\phi_{m}-\phi$ and $\phi_{m}$ is a material-dependent jamming
fraction [2]. The yield stress was also found to be a function of the distance
from jamming. This model cannot describe clay-rich suspensions, however, that
exhibit ductile yielding (Fig. 1). Caggioni suggested a generic constitutive
relation for soft glassy materials:
$\tau^{\star}=1+(t_{\mu}\dot{\gamma})^{1/2}+\eta\dot{\gamma}/\tau_{y}$, where
the three terms on the right hand side are elastic, plastic and viscous
dissipation, respectively [28]. The value $n=1/2$ was chosen as the typical
plastic yielding exponent for amorphous solids, and $t_{\mu}$ is a material-
specific relaxation time or, equivalently, $1/t_{\mu}$ is a critical strain
rate for yielding [29, 30]. Their model was validated against experiments of
various yield-stress fluids.

While the value for the plastic yielding exponent is commonly taken to be
$n=1/2$, theoretical studies indicate that $n$ may vary depending on material
properties [7]. More broadly, if brittle materials correspond to $n=0$ and
ductile materials have $n\approx 1/2$, what is the proper description for
intermediate materials? For granular suspensions we posit the following: as
the proportion of attractive particles (clay) $\chi$ increases from $\chi=0$
(pure sand suspensions), there is a percolation-like transition where the
yielding behavior becomes progressively more ductile once stress transmission
is dominated by cohesive elements. This means that, beyond a critical value of
$\chi$, the transitional (yielding) region of the flow curve grows with
increasing $\chi$ (Fig. 1c). In terms of the constitutive equation, this
gradual transition from brittle to increasingly ductile behavior would
correspond to increasing $n$. To describe this behavior, we propose a generic
dimensionless constitutive relation that builds on the models above:

$\tau^{\star}=1+\alpha{\Gamma}^{n}+\Gamma.$ (1)

In this expression the dimensionless shear rate
$\Gamma\equiv\dot{\gamma}t_{\mu}$. The coefficient $\alpha$ describes the
onset of plastic yielding of the suspension. For sand-rich granular
suspensions we expect $n=\alpha=0$, and that $t_{\mu}$ is a timescale
associated with the viscous dissipation of particle motion as described by the
model of Boyer et al. [16]. For clay-rich suspensions, however, $t_{\mu}$
should be a material relaxation time that increases with the strength of
inter-particle attraction [4] – as observed for soft microgel pastes [31].
More, clay-rich suspensions are expected to have a limiting value of $n\approx
1/2$ for ductile materials. Below we use $t_{\mu}$, $\alpha$ and $n$ as
fitting parameters, and examine how they change across a transition from sand-
rich (frictional) to clay-rich (cohesive) suspensions.

## Measuring and understanding Soft Earth suspension rheologies

We perform rheological experiments in a stress-controlled, parallel-plate
rheometer (Fig. 2a). This setup allows us to vary the gap height, and measure
both shear and normal stresses. Following previous protocol [2], we were
careful to avoid the effects of suspension wall slip, jamming/shear banding,
particle sedimentation, and particle ejection at the air-suspension interface.
These factors place limits on the experimentally accessible ranges of shear
rates, volume fractions, and clay concentrations, as discussed below. Pre-
shear protocols ensured that steady-shear flow curves were reproducible;
detailed information about sample preparation and rheological measurements
appears in the “Methods” section.

We create model Soft Earth suspensions by varying the proportions of de-
ionized water, silica sand and kaolin clay (see Fig. 2b for particle size
distributions). The total solids volume fraction (sand + clay) is denoted by
$\phi$, and the relative clay fraction is defined $\xi\equiv\phi_{clay}/\phi$.
Previous zeta potential ($\zeta$) measurements of the same kaolin particles in
de-ionized water found $\zeta\approx-33$mV, which corresponds to moderately
strong aggregation [32]. We find that below $\xi=0.2$ and $\phi\approx 0.2$,
the yield stress of the suspension is insufficient to prevent sedimentation of
sand particles; this sets an experimental lower limit for our sand-rich
suspensions. The upper limit of $\phi=0.60$ for sand-rich ($\xi=0.2$)
suspensions is the point at which the sample becomes difficult to process –
i.e., it jams. For pure clay suspensions ($\xi=1$), $\phi$ varies from a lower
limit $\phi=0.05$ to an upper achievable limit $\phi=0.40$. Steady-shear flow
curves ($\tau$ vs. $\dot{\gamma}$) are generated for each suspension (Fig. 2),
following a protocol we established previously for debris-flow slurries [2].

The yield stress $\tau_{y}$ for each suspension is obtained from a fit of the
Herschel-Bulkley equation to the steady-shear flow curve, whereas $t_{\mu}$
for each suspension is chosen such that all curves of differing $\phi$ values
for a given value of $\xi$ collapse and yielding is centered around
$\Gamma\approx 1$. We also perform transient tests to examine yielding, where
strain is applied at a constant rate $\Gamma$. In the quasi-static regime
($\Gamma\ll 1$), the stress build-up with applied strain, $\gamma$, provides a
measure of elasticity, while the nature and magnitude of the stress drop at
yielding indicates how brittle or ductile yielding is [6].

Sand-rich ($\xi=0.2$) suspensions show the same flow behavior as debris-flow
slurries from the California Coast Range, which had comparable clay and sand
content [2]. Cohesion contributes to a yield stress at low $\Gamma$
(dimensionless shear-rate), while for large $\Gamma$ the samples behave like a
granular (frictional) suspension (Fig 2e). The sand-rich suspensions for all
values of $\phi$ can be collapsed onto a dimensionless Bingham curve, i.e.,
$\tau^{\star}=1+\Gamma$ (Fig 2f). As expected, yield stress increases with
$\phi$, and there is no appreciable plastic dissipation stress ($\alpha=n=0$).
More, we find that $t_{\mu}\approx\eta(\Delta\phi)/\tau_{y}$ (Fig. 2f inset);
that is, the experimentally determined timescale for collapsing the flow
curves corresponds to the particle rearrangement timescale in a yield-stress
fluid. Transient tests for nearly-jammed suspensions ($\Delta\phi=0.03$) show
a sharp stress drop (Fig. 3c) that is characteristic of brittle yielding. Our
sand-rich suspensions jam around $\phi_{m}\approx 0.55$, consistent with
typical granular materials [16] and the California Coast Range materials [2].
For volume fractions $\phi\geq 0.40$ we observe a positive normal force (Fig.
4b), consistent with force-chain formation and dilatancy in granular
suspensions [33]; the importance of this will be discussed below.
Qualitatively and quantitatively, dense sand-rich suspensions exhibit brittle
yielding behavior.

Pure clay ($\xi=1$) suspensions are markedly different from sand-rich ones,
and exhibit flow curves similar to the clay-rich French Alps debris-flow
materials. In the dilute regime ($\phi=0.05$) the suspension flow curve
behaves like a Newtonian fluid (Fig 2d). As $\phi$ is increased, the yield
stress rapidly increases and the material jams at roughly $\phi=0.40$. Clay
suspensions are known to jam at significantly lower volume fractions than
granular suspensions, due to attractive particle networks that percolate
across the sample [20, 4]. Similar to the sand-rich suspensions, for all clay
suspensions with a yield stress ($\phi>0.05$), we find that
$t_{\mu}\approx\eta(\Delta\phi)/\tau_{y}$ (Fig 2f inset). When non-
dimensionlized, the pure clay data for all $\phi$ can be collapsed onto a
single curve; this curve, however, is distinct from the sand-rich data. In
particular, there is the marked emergence of an intermediate yielding regime –
spanning almost three orders of magnitude in $\Gamma$ – indicating ductile
behavior (Fig 2f). Fitting the collapsed data to Eq. 1, we find $n=1/2$
consistent with soft glassy materials, and $\alpha\approx 10$. We suspect that
the exponent $n\approx 1/2$ is related to the mechanism of plastic
dissipation, while the coefficient $\alpha$ is determined by the strength of
inter-particle attraction. To test this idea, we conduct additional
experiments with identical clay and water content, but varying ionic strength
through the addition of 0.001 M and 0.1 M NaCl. Salt screens the repulsive net
charge of clay particles and facilitates stronger aggregation [32, 34]. As
expected, we find that $\alpha$ increases systematically with ionic strength,
but $n$ remains constant (Extended Data Fig. 1, Table 1). Quasi-static strain
tests of nearly-jammed clay-rich suspensions ($\Delta\phi\approx 0.03$) show a
very small stress drop after yielding (Fig. 3c), and negligible normal force
(Fig. 4b). In sum, clay suspensions exhibit all of the hallmarks of ductile
yielding, and do not show any effects of granular suspensions. Our results
support the characterization of clay suspensions as a gel.

Figure 2: Experimental Soft Earth suspensions, and steady-shear flow curves of
sand-rich and pure-clay suspensions. a, Parallel-plate rheology setup; arrow
indicates shear direction. b, Particle size distributions for kaolin clay and
silica sand particles. c, Scanning Electron Microscopy images of kaolin clay
particles (left) and silica sand (right) particles. Insets are zoomed-in
images for each case; note the stacked clay sheets for kaolin.d, Steady-shear
flow curves (shear stress-shear rate) of pure clay suspensions ($\xi=1$) and
e, sand-rich suspensions ($\xi=0.2$). Respective volume fractions ($\phi$) are
indicated by different shapes values are indicated on the side of the plots
(diamonds, $\phi$ = 0.10; invert triangle with right-half black, $\phi$ =
0.15; invert triangles, $\phi$ = 0.20; triangles, $\phi$ = 0.30; circle with
left-half black, $\phi$ = 0.35; circles, $\phi$ = 0.40; squares, $\phi$ =
0.50; squares with top-half black, $\phi$ = 0.55). f, Dimensionless flow
curves for the data shown in (d) and (e); note similarity of the sand-rich and
pure clay suspensions to California Coast Range and French Alps data,
respectively (Fig 1b). Inset shows that empirically determined microscopic
timescale, $t_{\mu}$, scales with the viscous relaxation time of a yield
stress fluid, $\eta(\Delta\phi)/\tau_{y}$.

We can now examine the transition from brittle to ductile behavior as relative
clay content is systematically changed between $\xi=0.2$ and $\xi=1$.
Transient shear tests show that as $\xi$ increases, the stress drop associated
with yielding systematically decreases (Fig. 3c, Extended Data Fig. 2). For
steady-shear flow curves, we observe that the failure envelope – defined by
the range in $\Gamma$ associated with the transitional regime between purely
elastic and pure viscous flow (Methods) – increases systematically with
increasing $\xi$ (Fig. 4a). Fitting Eq. 1 to the data, we observe that $n$
increases with $\xi$ from a lower limit of $n=0$ at $\xi=0.20$ to a maximum of
$n=1/2$ at $\xi=1$ (Fig. 3). We also observe that $\alpha$ increases
systematically with clay content (Extended Data Fig. 3, Table. 1), consistent
with our proposal that this parameter is related to the strength of cohesive
bonds. Previous studies have mapped gelation in colloid and polymer systems to
percolation [35, 36, 37]. If the percolation interpretation is relevant for
our system, it implies that increasing $\xi$ corresponds to increasing the
cluster size of the attractive clay network – and that the brittle-ductile
transition occurs when these cohesive particle networks percolate across the
sample. Normal forces become negligible for $\xi>0.6$, suggesting that
granular force chains are unable to percolate the sample for larger clay
concentrations (Fig. 4). A different take on criticality is to examine the
behavior of the “yielding exponent”, defined in elasto-plastic models as
$\beta\equiv 1/n$ [7]. In this view, $\beta=2$ in the fully ductile regime
corresponding to pure clay suspensions ($\xi=1$). As $\xi$ decreases toward a
critical value $\xi_{c}$ associated with sand-rich suspensions, $\beta$
diverges as yielding becomes increasingly discontinuous (Fig. 3b inset). In
the limit that $\beta\rightarrow\infty$, which corresponds to roughly
$\xi_{c}\approx 0.2$ for our suspensions, suspension yielding is discontinuous
and reduces to classical Coulomb failure.

Figure 3: The brittle-ductile transition in Soft Earth suspensions. a, Non-
dimensionalized flow curves for all values of $\xi$. Note changes in the
vicinity of yielding (inset), corresponding to increasing plasticity with
increasing $\xi$. b, The exponent $n$ in Eq. 1 increases gradually from a
minimum $n=0$ at $\xi=0.20$ associated with brittle failure, to a maximum
$n=1/2$ at $\xi=1$ corresponding to plastic failure. Increasing the attraction
strength of clay did not change $n$. Inset: the yielding exponent $\beta=1/n$
diverges as $\xi\rightarrow\xi_{c}$ from above, where the critical relative
clay fraction $\xi_{c}=0.2$. This is another way to show the continuous nature
of ductile to brittle transition in Soft Earth suspensions. c, Stress-strain
curves for suspensions at all values of $\xi$, where the oscillatory stress
$\tau_{osc}$ is normalized by the steady-state stress after yielding
$\tau_{ss}$ is presented as a function of strain. The stress overshoot
systematically decreases as $\xi$ increases; this behavior is characteristic
of a brittle to ductile transition. (compare to Fig. 2a in ref. [6]).

## Discussion and Implications

The origins of complex flow behaviors for Soft Earth suspensions arise from
two critical points. First is the well known jamming transition, in which
viscosity diverges with increasing volume fraction as
$\eta(\phi)\propto(\Delta\phi)^{-2}$ [16, 2, 38]. The second, revealed here,
is the ductile to brittle transition wherein $\beta\rightarrow\infty$ as
$\xi\rightarrow\xi_{c}$ from above. Sand-rich slurries match the behavior of
granular debris flows; they exhibit brittle yielding corresponding to a rapid
transition from frictional to viscous stress dissipation. These suspensions
are compressible, and effects like Reynolds dilation and transient pore
pressure must be considered for simulating debris flows [3]. Even suspensions
with moderate clay content ($\xi=0.20$) still behave as granular suspensions;
however, the yield stress arising from cohesion must be taken into account
[2]. Clay-rich slurries reproduce the observed rheology of Cohesive debris
flows; ductile yielding arises due to strong plastic stress dissipation.
Strong cohesion acts as a confining pressure [39] that likely shuts off
Reynolds dilation and the associated pore pressure effects; these suspensions
can be modeled as gels, where normal forces are negligible (Fig. 4b). Two
important concepts emerging from granular and suspension rheology are used to
build the constitutive Eq. 1. First, there are elastic, plastic and viscous
stresses that arise from the distinct material components – frictional sand,
attractive clays, and water – and these stresses are additive [39, 28, 40].
Second, rheologies can be assembled by nondimensionalizing the shear rate with
an appropriate microscopic rearrangement timescale [16]. In contrast to
(cohesionless) granular suspensions, however, the yield stress – rather than
the confining pressure – is the relevant quantity. Thus, the microscopic
timescale $t_{\mu}\approx\eta(\Delta\phi)/\tau_{y}$. Field-scale debris flows
may introduce inertial effects for both particles (collisions) and fluid
(turbulence) [2] that are not considered here. Exploring the inertial regime
for cohesive suspensions – as has been done recently for granular suspensions
[40] – is a logical next step.

Figure 4: The phase space of Soft Earth suspension behaviors. a,
Experimentally determined phase space of elastic, viscous and transitional
(plastic) regimes; compare to the proposed flow phase space in Fig. 1c. Colors
correspond to $\xi$ values in Fig. 3. b, Phase space showing suspension
mixtures that have significant normal force (plus) or negligible normal force
(dots) under shear. This normal force is associated with dilatancy and the
presence of force chains – i.e., granular-frictional effects (red box). Only
suspensions with relatively high sand content ($\phi_{sand}$) exhibit
significant normal force; as clay content ($\phi_{clay}$) increases beyond a
critical value, the normal force becomes negligible as cohesion dominates over
friction (green box). Inset boxes correspond to the associated clay-rich and
sand-rich microstructures in Fig. 1b.

The brittle-ductile transition in our suspensions appears to be due to a
handoff from frictional to cohesive control, when clay aggregates percolate
the sample. This transition is manifest as a continuous increase in
plasticity, and decrease in granular effects such as a normal force, as the
relative clay concentration for dense suspensions is increased (Fig. 4). There
are currently no modeling studies to directly compare to our results. Some of
our observations could be anticipated from simulations of idealized amorphous
solids, however, which helps to understand the origins of this transition.
Granular simulations found that frictional particles exhibit brittle yielding,
while the same particles without friction showed ductile failure [41]. A
different simulation study of an idealized glass found that increasing the
lengthscale of inter-particle interactions was sufficient to drive a
transition from brittle to ductile failure [42]. Our brittle and ductile
suspensions exhibit similar behavior to highly and poorly annealed glasses,
respectively [6]. These results suggest that adding clay may lubricate
contacts among sand grains [15], increase the lengthscale of cooperative
particle motion [20], and/or maintain “soft spots” that do no anneal under
shear [6]. Although these mechanisms are different, they all act to delocalize
failure and prevent formation of a slip plane – which is the origin of brittle
failure. Probing the brittle-ductile transition in an optically-transparent
cohesive-frictional mixture, such as clear particles suspended in a gel [25],
would allow visualization of the microscopic dynamics underlying failure; we
leave this for future work.

We conclude with considering the consequences of our findings for yielding
behavior in the Soft Earth. Sudden liquefaction of soil, due to seismic
shaking or rainfall, is typically limited to water-saturated sand-rich
materials that exhibit brittle failure due to pore-pressure induced dilation
[3, 9]. Cohesive soils tend to form slow “earthflows” that creep ductily [43].
Shale, a type of sedimentary rock, has been shown to undergo a transition from
brittle to ductile failure when the fraction of cohesive materials (clay +
organics) exceeds 0.35 [44], consistent with our data and a percolation-like
transition. Finally, deep in the Earth’s lithosphere the rock becomes soft,
and the accompanying brittle-to-ductile transition exerts a strong control on
the source, rupture dynamics and magnitude of earthquakes [45]. While these
geophysical problems involve disparate materials, scales and stresses, the
discovered connections – from idealized amorphous solids to our model Soft
Earth suspensions to natural debris-flow materials – embolden us to seek
commonality in failure across more Earth materials.

## Methods

Grain size measurements Particle size distributions were measured using a
Beckman-Coulter Particle Size Analyzer LS13-320. Grain size was determined in
114 log-spaced bins over the range 0.04 $\mu$m to 2000 $\mu$m. The particle
size distributions in both kaolin clay and silica sand contained a single
mode, and thus had distinct average particle size population. Microscopic
images of silica sand and kaolin clay were obtained using Scanning Electron
Microscope (FEI Quanta 600 Environmental Scanning Electron Microscope). The
images were acquired at an accelerating voltage of 15 kV and a water vapor
pressure of 0.75 torr.

Sample preparation Model Soft Earth suspensions were prepared at desired final
concentrations ($\phi$) by mixing the three components – sand, clay, and water
– in varying proportions. First, a pre-determined amount of kaolin clay was
mixed with de-ionized water for 30 mins in a high-shear mixer. The mixture was
left overnight to allow the clay particles to absorb water and reach an
equilibrium state. Second, the silica sand was added to the mixture and
further mixed in the high-shear mixer for $\sim$10 mins to ensure uniform
distribution. For sample preparation purposes we use density of particles
(both sand and clay) and deionized water as 2650 kg/m3 and 1000 kg/m3
respectively.

Rheological characterizations All rheological measurements are carried out
using a TA Instruments DHR-3 Model stress-controlled rheometer using a 40mm
parallel-plate setup at 250C. To reduce the sample slip effects in the
boundaries during our measurements, both the top plate and the bottom plate
were modified by attaching a 50-Grit size serrated surfaces. This translated
to a roughness length scale $\approx$ 300 $\mu$m, which is three times the
average sand particle size. We maintain a gap height of 1mm for all the
experiments ($\sim$ 10 times the largest particle size), which is shown to
reduce particle confinement effects during the shear flow [46].

Establishing reproducible rheological measurements All suspensions were pre-
sheared to ensure that the samples did not have a shear history, allowing us
to generate reproducible steady-shear and stress-growth rheological
measurements. In the first step, samples were sheared using a large-amplitude
oscillatory protocol ($\gamma$ = 500%, $\omega$ = 10 rad/s) to minimize the
directional bias in the microstructures that come from steady shear protocols
[47]. The breaking of the internal structure was monitored using the
macroscopic elastic modulus ($G^{\prime}$), which decreased and then plateaued
within 60s, for all our samples. During the second step, the sample was
allowed to recover. The recovery time was estimated using a small-amplitude
oscillatory protocol ($\gamma$ = 0.05%, $\omega$ = 1 rad/s), which resembles
near-equilibrium rheology measurements [48]. We monitored the value of the
elastic modulus, which grew as the suspension structure recovered, and
plateaued to a constant value for all our samples within 30-80s.

Steady shear and transient rheological characterizations Following the pre-
shear protocols above, the steady-shear flow curves were obtained by: applying
a constant shear rate, waiting for the stress to equilibrate to determine
shear stress $\tau$, and then repeating the same for all shear rates. The
maximum and minimum shear stress limits in our flow curves were set by the
interfacial and gravitational stresses in the particle scale, respectively.
The interfacial stress is set by $\tau_{max}\sim\gamma_{aw}a$, where
$\gamma_{aw}$ is the stress at the air-water interface and $a$ is the average
radius ($\approx$ 50 $\mu$m) for the larger particle set (silica sand). This
value sets the shear stress beyond which particles may eject out of the fluid-
air interface. The lower shear stress limit is set by the gravitational force
acting on a particle scale given by $\tau_{min}\sim\Delta\rho ga$, where
$\Delta\rho$ is the density difference between particle and the suspending
fluid ($\approx$1650 kg/m3) and $g$ is acceleration due to gravity (9.8 m/s2).
Below this stress, the sedimentation effects due to settling of silica grains
is important [2]. Transient rheological characterizations were performed using
oscillatory amplitude sweep protocol at constant oscillations ($\omega=1$
rad/s). Samples prepared close to their respective jamming points
($\Delta\phi\approx 0.03$) were loaded on the rheometer, followed by a
percentage amplitude sweep from 0.01 - 100 %, to generate the shear stress-
strain ($\tau_{osc}-\gamma\%$) curves.

## References

  *   * [1] Iverson, R. M., Reid, M. E. & LaHusen, R. G.  Debris-flow mobilization from landslides.  _Annual Review of Earth and Planetary Sciences_ 25, 85–138 (1997).
  * [2] Kostynick, R. _et al._ Rheology of debris flow materials is controlled by the distance from jamming.  _Proceedings of the National Academy of Sciences_ 119, e2209109119 (2022).
  * [3] Iverson, R. M. & George, D. L.  A depth-averaged debris-flow model that includes the effects of evolving dilatancy. i. physical basis.  _Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences_ 470, 20130819 (2014).
  * [4] Coussot, P.  Structural similarity and transition from newtonian to non-newtonian behavior for clay-water suspensions.  _Physical Review Letters_ 74, 3971 (1995).
  * [5] Ancey, C. & Jorrot, H.  Yield stress for particle suspensions within a clay dispersion.  _Journal of Rheology_ 45, 297–319 (2001).
  * [6] Ozawa, M., Berthier, L., Biroli, G., Rosso, A. & Tarjus, G.  Random critical point separates brittle and ductile yielding transitions in amorphous materials.  _Proceedings of the National Academy of Sciences_ 115, 6656–6661 (2018).
  * [7] Lin, J. & Wyart, M.  Microscopic processes controlling the herschel-bulkley exponent.  _Physical review E_ 97, 012603 (2018).
  * [8] Jerolmack, D. J. & Daniels, K. E.  Viewing earth’s surface as a soft-matter landscape.  _Nature Reviews Physics_ 1, 716–730 (2019).
  * [9] Huang, Y. & Yu, M.  _Hazard analysis of seismic soil liquefaction_ (Springer, 2017).
  * [10] Huang, Y. & Zhao, L.  The effects of small particles on soil seismic liquefaction resistance: current findings and future challenges.  _Natural hazards_ 92, 567–579 (2018).
  * [11] Kean, J. W. & Staley, D. M.  Forecasting the frequency and magnitude of postfire debris flows across southern california.  _Earth’s Future_ 9, e2020EF001735 (2021).
  * [12] Major, J. J. & Pierson, T. C.  Debris flow rheology: Experimental analysis of fine-grained slurries.  _Water Resources Research_ 28, 841–857 (1992).
  * [13] Coussot, P. & Piau, J. M.  On the behavior of fine mud suspensions.  _Rheologica acta_ 33, 175–184 (1994).
  * [14] Coussot, P. & Meunier, M.  Recognition, classification and mechanical description of debris flows.  _Earth-Science Reviews_ 40, 209–227 (1996).
  * [15] Ancey, C.  Plasticity and geophysical flows: A review.  _Journal of non-Newtonian fluid mechanics_ 142, 4–35 (2007).
  * [16] Boyer, F., Guazzelli, É. & Pouliquen, O.  Unifying suspension and granular rheology.  _Physical review letters_ 107, 188301 (2011).
  * [17] Ran, R. _et al._ Understanding the rheology of kaolinite clay suspensions using bayesian inference.  _Journal of Rheology_ 67, 241–252 (2023).
  * [18] Coussot, P., Nguyen, Q. D., Huynh, H. & Bonn, D.  Avalanche behavior in yield stress fluids.  _Physical review letters_ 88, 175501 (2002).
  * [19] Bonn, D., Coussot, P., Huynh, H., Bertrand, F. & Debrégeas, G.  Rheology of soft glassy materials.  _Europhysics Letters_ 59, 786 (2002).
  * [20] Bonn, D., Denn, M. M., Berthier, L., Divoux, T. & Manneville, S.  Yield stress materials in soft condensed matter.  _Reviews of Modern Physics_ 89, 035005 (2017).
  * [21] Guy, B., Richards, J., Hodgson, D., Blanco, E. & Poon, W.  Constraint-based approach to granular dispersion rheology.  _Physical review letters_ 121, 128001 (2018).
  * [22] Jiang, Y., Makino, S., Royer, J. R. & Poon, W. C.  Flow-switched bistability in a colloidal gel with non-brownian grains.  _Physical Review Letters_ 128, 248002 (2022).
  * [23] Li, Y., Royer, J. R., Sun, J. & Ness, C.  Impact of granular inclusions on the phase behavior of colloidal gels.  _Soft Matter_ 19, 1342–1347 (2023).
  * [24] Jiang, Y. & Seto, R.  Colloidal gelation with non-sticky particles.  _Nature Communications_ 14, 2773 (2023).
  * [25] Kammer, C., Blackwell, B., Arratia, P. E. & Ponte Castañeda, P.  A homogenization model for the rheology and local field statistics of suspensions of particles in yield stress fluids.  _Journal of Rheology_ 66, 535–549 (2022).
  * [26] Dagois-Bohy, S., Hormozi, S., Guazzelli, E. & Pouliquen, O.  Rheology of dense suspensions of non-colloidal spheres in yield-stress fluids.  _Journal of Fluid Mechanics_ 776, R2 (2015).
  * [27] Bocquet, L., Colin, A. & Ajdari, A.  Kinetic theory of plastic flow in soft glassy materials.  _Physical review letters_ 103, 036001 (2009).
  * [28] Caggioni, M., Trappe, V. & Spicer, P. T.  Variations of the herschel–bulkley exponent reflecting contributions of the viscous continuous phase to the shear rate-dependent stress of soft glassy materials.  _Journal of Rheology_ 64, 413–422 (2020).
  * [29] Lin, C.-R. & Chen, W.-J.  The links-nodes-blobs model for shear-thinning-yield-stress fluids.  _Colloid and Polymer Science_ 277, 1019–1025 (1999).
  * [30] Domenech, T. & Velankar, S. S.  On the rheology of pendular gels and morphological developments in paste-like ternary systems based on capillary attraction.  _Soft matter_ 11, 1500–1516 (2015).
  * [31] Cloitre, M., Borrega, R., Monti, F. & Leibler, L.  Glassy dynamics and flow properties of soft colloidal pastes.  _Physical Review Letters_ 90, 068303 (2003).
  * [32] Seiphoori, A., Gunn, A., Kosgodagan Acharige, S., Arratia, P. E. & Jerolmack, D. J.  Tuning sedimentation through surface charge and particle shape.  _Geophysical Research Letters_ 48, e2020GL091251 (2021).
  * [33] Gamonpilas, C., Morris, J. F. & Denn, M. M.  Shear and normal stress measurements in non-brownian monodisperse and bidisperse suspensions.  _Journal of Rheology_ 60, 289–296 (2016).
  * [34] Israelachvili, J. N.  _Surface forces_ (CRC Press, 2022).
  * [35] De Gennes, P.-G.  On a relation between percolation theory and the elasticity of gels.  _Journal de Physique Lettres_ 37, 1–2 (1976).
  * [36] Del Gado, E., de Arcangelis, L. & Coniglio, A.  Elastic properties at the sol-gel transition.  _Europhysics Letters_ 46, 288 (1999).
  * [37] Stauffer, D., Coniglio, A. & Adam, M.  _Gelation and critical phenomena_ (Springer, 2005).
  * [38] Pradeep, S., Nabizadeh, M., Jacob, A. R., Jamali, S. & Hsiao, L. C.  Jamming distance dictates colloidal shear thickening.  _Physical Review Letters_ 127, 158002 (2021).
  * [39] Vo, T. T., Nezamabadi, S., Mutabaruka, P., Delenne, J.-Y. & Radjai, F.  Additive rheology of complex granular flows.  _Nature communications_ 11, 1476 (2020).
  * [40] Tapia, F., Ichihara, M., Pouliquen, O. & Guazzelli, É.  Viscous to inertial transition in dense granular suspension.  _Physical Review Letters_ 129, 078001 (2022).
  * [41] Karimi, K., Amitrano, D. & Weiss, J.  From plastic flow to brittle fracture: Role of microscopic friction in amorphous solids.  _Physical Review E_ 100, 012908 (2019).
  * [42] Dauchot, O., Karmakar, S., Procaccia, I. & Zylberg, J.  Athermal brittle-to-ductile transition in amorphous solids.  _Physical Review E_ 84, 046105 (2011).
  * [43] Evans, S., Bovis, M. & Hutchinson, J.  Landslides of the flow type.  _Environmental & Engineering Geoscience_ 7, 221–238 (2001).
  * [44] Wang, J. _et al._ Effect of clay and organic matter content on the shear slip properties of shale.  _Journal of Geophysical Research: Solid Earth_ 124, 9505–9525 (2019).
  * [45] Kato, A. & Ben-Zion, Y.  The generation of large earthquakes.  _Nature Reviews Earth & Environment_ 2, 26–39 (2021).
  * [46] Peyla, P. & Verdier, C.  New confinement effects on the viscosity of suspensions.  _Europhysics Letters_ 94, 44001 (2011).
  * [47] Choi, J. & Rogers, S. A.  Optimal conditions for pre-shearing thixotropic or aging soft materials.  _Rheologica Acta_ 59, 921–934 (2020).
  * [48] Pradeep, S., Wessel, A. & Hsiao, L. C.  Hydrodynamic origin for the suspension viscoelasticity of rough colloids.  _Journal of Rheology_ 66, 895–906 (2022).

## Declarations

Acknowledgments We thank Jamie Ford (Nanoscale Characterization Facility,
Singh Center for Nanotechnology) and John Ruck (UPenn) for their help with
particle characterizations; and Robert Kostynick (Washington U), Alban Sauret,
Eckart Meiburg and Tom Dunne (UC Santa Barbara) for their inspiration and
feedback. Research was supported by National Science Foundation (NSF)
Materials Research Science and Engineering Center (NSF-DMR-1720530) and Army
Research Office (ARO Grant W911NF2010113) grants to D.J. and P.A., and by
grant NSF PHY-1748958 to the Kavli Institute for Theoretical Physics (KITP).

Author contributions S.P. led all aspects of this study: the conceptual
framework, experimental design and execution, and analysis of results. P.A.
and D.J. were co-equal supervisors of the work, and participated in data
analysis and interpretation. The manuscript was written collaboratively by all
authors.

Competing interests Authors declare no competing interests.

Extended Data Fig. 1: Steady shear rheological flow curves of kaolin clay
($\xi=1$) with a, 0.001 M and b, 0.1 M NaCl. Respective volume fractions
($\phi$) are indicated by different shapes (diamonds, $\phi$ = 0.10; invert
triangles with right-half black, $\phi$ = 0.15; invert triangles, $\phi$ =
0.20; triangles with right-half black, $\phi$ = 0.25; triangles, $\phi$ =
0.30). The non-dimensionalized flow curves ($\tau^{\star}-\Gamma$) are shown
in plots c,d, respectively. Corresponding model fits for the equation
$\tau^{\star}=1+\alpha\Gamma^{n}+\Gamma$, with best fit values for the
parameters $\alpha$ and $n$, are shown alongside the collapsed plots in the
figures c,d.

Extended Data Fig. 2: The stress overshoot as a function of changing sand-clay
concentration ($\xi$). The inset shows the unscaled data for Fig. 3c. The
colors indicated correspond to the $\xi$ values in Fig. 3.

Extended Data Fig. 3: Steady shear rheological flow curves of sand-clay
suspension mixtures at a, $\xi=0.4$, b, $\xi=0.6$, and c, $\xi=0.8$.
Respective volume fractions ($\phi$) are indicated by different shapes
(diamonds, $\phi$ = 0.10; invert triangles with right-half black, $\phi$ =
0.15; invert triangles, $\phi$ = 0.20; triangles with right-half black, $\phi$
= 0.25; triangles, $\phi$ = 0.30; circle with left-half black, $\phi$ = 0.35;
circles, $\phi$ = 0.40; circles with top-half black, $\phi$ = 0.55; squares,
$\phi$ = 0.50; squares with top-half black, $\phi$ = 0.55). The non-
dimensionalized flow curves ($\tau^{\star}-\Gamma$) are shown in plots d, e,
and f, respectively. Corresponding model fits for the equation
$\tau^{\star}=1+\alpha\Gamma^{n}+\Gamma$, with best fit values for the
parameters $\alpha$ and $n$, are shown alongside the collapsed plots in c,d,
with the fit values of $\alpha$ and $n$.

Extended Data Table. 1: The average fit values for the coefficient $\alpha$
and shear-thinning exponent $n$ for sand-clay suspensions at varying $\xi$
values and for clay suspensions at varying salt (NaCl) concentrations.