# Training microrobots to swim by a large language model

Zhuoqun Xu *  | Department of Mechanical Engineering, National University of Singapore, 117575, Singapore  Lailai Zhu *  |<EMAIL_ADDRESS>Department of Mechanical Engineering, National University of Singapore, 117575, Singapore

###### Abstract

Machine learning and artificial intelligence have recently represented a
popular paradigm for designing and optimizing robotic systems across various
scales. Recent studies have showcased the innovative application of large
language models (LLMs) in industrial control [1] and in directing legged
walking robots [2]. In this study, we utilize an LLM, GPT-4, to train two
prototypical microrobots for swimming in viscous fluids. Adopting a few-shot
learning approach, we develop a minimal, unified prompt composed of only five
sentences. The same concise prompt successfully guides two distinct
articulated microrobots—the three-link swimmer and the three-sphere swimmer—in
mastering their signature strokes. These strokes, initially conceptualized by
physicists, are now effectively interpreted and applied by the LLM, enabling
the microrobots to circumvent the physical constraints inherent to micro-
locomotion. Remarkably, our LLM-based decision-making strategy substantially
surpasses a traditional reinforcement learning method in terms of training
speed. We discuss the nuanced aspects of prompt design, particularly
emphasizing the reduction of monetary expenses of using GPT-4.

A fundamental characteristic of living organisms is their capacity for
locomotory motion, encompassing a spectrum of movements such as running,
crawling, flying, slithering, and swimming. Having evolved through the long
process of natural selection and adaptation, these motility patterns observed
in biological realm inspire the design of bionic articulated robots. They
achieve locomotion by actuating their movable components similar to natural
joints (or hinges and links), e.g., walking legs, flapping wings, and
undulating fins.

A natural question arises: How can we actuate these joints concertedly to
enhance the robot’s locomotory performance, environmental adaptiveness, and
perturbation resilience? It corresponds to identifying the effective time
sequence of robotic actions—forces (and torques) exerted on the moving
elements or their translational (and rotational) velocities.

This question has be addressed by various means. For example, biological
strategies from evolution can be mimicked with heuristic adjustment. Besides,
diverse schemes in the frame of optimal control theory have been established
for robotic movement [3, 4, 5]. Moreover, machine learning has expedited the
development of more autonomous and intelligent robotic controllers. Notably,
reinforcement learning (RL) as a popular machine learning technique has been
applied to walking [6, 7, 8], flying [9, 10], and swimming [11, 12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
34, 35, 36, 37, 38] robots at different scales.

Yet, the recent surge in large language models (LLMs) or foundation models in
general has spawned a new machine learning paradigm, in-context learning (ICL)
[39], for decision-making and robotic manipulation. Unlike RL or other machine
learning techniques, ICL precludes updating neural network weights, instead
learns to solve new tasks during the inference phase via receiving text
prompts that incorporate exemplar task demonstrations. Owing to their
extensive pre-training on diverse and huge datasets, LLMs exhibit the ICL
capacity and thus can perform a variety of complex tasks via ICL including
reasoning [40], logic deduction [41, 42], solving mathematical problems [43],
sentiment analysis [44], language translation [45], code generation [46],
medical diagnostics [47], and so on. More recently, LLM has been utilized for
industrial and robotic control. In particular, GPT-4, an LLM pre-rained on
internet-scale datasets, has been successfully utilized for controlling HVAC
[1] and for enabling terrestrial robots to walk [2] in simulated environments.
Importantly, Ref. [1] reveals that this LLM approach attains a performance
comparable to that of RL, but requiring fewer samples and incurring lower
technical debt.

Fig. 1:  Diagrammatic representation of utilizing an LLM, GPT-4 adopted here,
for microrobotic locomotion. We have developed a minimal, unified minimal
prompt capable of instructing two model microswimmers—Purcell’s swimmer [48]
and NG’ swimmer [49]—to propel through highly viscous fluids. These
microswimmers’ movements are subject to viscous hydrodynamics governed by the
Stokes equations: $\gradient\cdot\mathbf{u}=0$ and
$\mu\gradient^{2}\mathbf{u}=\gradient p$, where $\mathbf{u}$ and $p$ denote
the velocity and pressure fields, respectively, with $\mu$ indicating the
fluid’s dynamic viscosity. Comprising only five sentences, the prompt
effectively orchestrates the interaction between the swimmer and the LLM,
directing the former to swim along the horizontal axis ($\mathbf{e}_{x}$ or
$-\mathbf{e}_{x}$) with maximal speed. The swimmer’s displacement is
$X=\mathbf{r}_{\text{c}}\cdot\mathbf{e}_{x}$, where $\mathbf{r}_{\text{c}}$
denotes its geometric centroid.

In this work, we explore the ability of LLMs to deduce physical principles
underlying robotic locomotion and subsequently leverage the understanding to
design locomotory gaits. Our specific focus is on microrobotic swimming at a
vanishing Reynolds (Re) number, a scenario physically constrained by the
dominance of viscous fluid forces over inertia forces. This configuration
enables examining whether LLMs can perceive Purcell’s “scallop theorem”,
introduced by Edward Purcell in his seminal 1977 lecture “Life at low Reynolds
number” [48]. The theorem posits that in the inertialess (Stokesian) limit,
i.e., $\text{Re}=0$, reciprocal motions or actuations, exemplified by a
scallop’s opening and closing, cannot generate net movement. Consequently, a
one-hinged microswimmer fails in effective propulsion in viscous fluids,
because its single degree of freedom (DOF) leads to inherently reciprocal
movement. To circumvent this physical constraint, he then proposed a two-
hinged ‘simplest animal’. Later known as Purcell’s three-link swimmer (see Fig
1 left upper panel), the minimal model can effectively propel through non-
reciprocal motions, enabled by the addition of more than one DOF.

Here, we explore the LLM, GPT-4, to navigate this Purcell’s three-link swimmer
and another prototypical model—Najafi-Golestanian (NG)’s three-sphere swimmer
[49] in inertialess viscous fluids. These swimmers communicate with the LLM by
responding to its generated actions and sending a few task demonstrations back
through text prompts. We design a unified minimal prompt framework to
coordinate the interaction between GPT-4 and both swimmers. Remarkably,
despite its training lacking physical data, GPT-4 enables the microswimmers to
learn efficient strokes, thereby overcoming the low-Reynolds-number physical
constraints.

## Viscous Hydrodynamic Environments for Microswimmers

The Purcell’s swimmer comprises three slender cylindrical links of length $a$
and radius $b$, connected in sequence by two planar hinges. These hinges allow
the links to rotate relative to their neighbor(s). Propulsion is attainable by
varying over time ($t$) the relative angles, $d_{1}(t)$ and $d_{2}(t)$,
between every two adjacent links. In contrast, NG’s swimmer constitutes three
spheres with radius $a$, aligned colinearly along their common axis along
$\mathbf{e}_{x}$. The spheres are linked sequentially by two extensible links,
with lengths $d_{1}(t)$ and $d_{2}(t)$, respectively. Time-varying the two
DOFs enables this swimmer to translate in the $\mathbf{e}_{x}$ direction.
Notably, we have intentionally adopted the same notation $a$ as the
characteristic length of both models. The same intention applies to their
DOFs, $d_{1}$ and $d_{2}$.

We characterize the locomotion of both swimmers by the velocities of their
centroid coordinated at $\mathbf{r}_{\text{c}}$. Namely, NG’s swimmer
translates horizontally at a velocity of
$\dot{\mathbf{r}}_{\text{c}}\mathbf{e}_{x}$, with the overdot signifying a
time derivative. In contrast, the Purcell’s swimmer exhibits two-dimensional
motion, reflected by the translational velocity $\dot{\mathbf{r}}_{\text{c}}$
of its middle link’s centroid and the link’s rotational velocity
$\dot{d}_{\text{c}}\mathbf{e}_{z}$. Below, we will briefly describe how to
calculate the swimming velocity of Purcell’s swimmer. The corresponding
calculation for NG’s swimmer is given in Materials and Methods.

Fig. 2: A, displacement $X/a$ of Purcell’s swimmer versus its execution step
$n$, which is trained by the LLM (solid line) and $Q$-learning-based RL
(dashed line). The lower panel demonstrates the cycle of signature gaits
learned by the swimmer. B, same as A, but for NG’s swimmer.

In this study of Purcell’s swimmer, we allow only one angular DOF, either
$d_{1}$ or $d_{2}$, to vary at a time. The selected DOF is restricted to a
discrete set of angles, $\left[-d_{\text{max}},d_{\text{max}}\right]$. Hence,
the swimming action as the DOF’ rate of change (ROC) is also discrete and
drawn from $\left[-2d_{\text{max}}/T,0,2d_{\text{max}}/T\right]$, where $T$
represents the gear-changing time.

To map the swimmer’ stroke to its kinematics, we assume that its links possess
a small slenderness $\epsilon=b/a\ll 1$. This allows for using the local
resistive force theory to account for the swimming hydrodynamics. For a link
parameterized by the arc-length $s\in[0,a]$, the hydrodynamic force per unit
length $\mathbf{f}$ is:

$\displaystyle\mathbf{f}(s)=-\left[\xi_{\parallel}\mathbf{t}\mathbf{t}+\xi_{\perp}\left(\mathbf{I}-\mathbf{t}\mathbf{t}\right)\right]\cdot\dot{\mathbf{r}}(s),$
(1)

where $\xi_{\parallel}$ and $\xi_{\perp}$ represent the resistive force
coefficients, and $\mathbf{t}$ and $\mathbf{r}(s)$ denote the link’s tangent
vector and local coordinates, respectively. Here, $\mathbf{I}$ is the identity
tensor. We consider the limit $\epsilon\rightarrow 0$ here, leading to
$\xi_{\perp}/\xi_{\parallel}\rightarrow 2$. In the inertialess flow, the
swimmer’s translational and rotational velocities can be derived using the
constraints of net zero hydrodynamic force and torque summed over all links,
respectively. For an individual link, the force is
$\int_{0}^{a}\mathbf{f}(s)\text{d}s$, and the torque about the centroid is
$\int_{0}^{a}\left[\mathbf{r}(s)-\mathbf{r}_{\text{c}}\right]\times\mathbf{f}(s)\text{d}s$.
Applying the constraints, we derive the swimming velocities analytically using
Mathematica, with their formula given in Materials and Methods.

We build the physical environment in a dimensionless framework. By choosing
$2d_{\text{max}}/T$ as the characteristic angular velocity, the dimensionless
ROCs fall within the set of discrete integers, $[-1,0,1]$. This integer nature
significantly aids ICL via LLMs, as these models commonly have difficulty in
accurately interpreting floating numbers [50].

## Methodology: LLM Prompt

We employ the few-shot prompting method with standard GPT-4, instructing it to
directly generate actions for a microswimmer without task-specific fine-
tuning. The swimmer subsequently executes the received action in the physical
environment, with the latest few historical snapshots stored in a textual
buffer. This buffer is then provided to GPT-4, aiding its subsequent decision-
making.

We have designed a unified prompt that can guide both swimmers effectively,
with a particular focus on minimizing the text length and associated monetary
cost. Our prompt comprises five functional sections, each containing a single
sentence, thereby highlighting the minimal textual elements required. Firstly,
it sets the objective: to determine the ROC sequence in both DOFs, targeting
the swimmer’s fastest long-term movement in a prescribed direction. The second
sentence define constraints on the DOFs, e.g.,
$d_{1}\in\left[-d_{\text{max}},d_{\text{max}}\right]$. Third, we demonstrate
the latest $n_{\text{ht}}$ historical records of the interactions between the
LLM and the physical environment, leaving $n_{\text{ht}}$ as a tunable
parameter. The following prompt alerts the LLM to the potential for long-term
impacts of the action. Finally, the last sentence instructs the LLM to
generate the next action constrained within a specified group.

###  .1 History clearing scheme

Occasionally, we observe that the swimmer cannot develop an effective
strategy, becoming ensnared in a cycle of erroneous strokes. This trapping
failure may be attributed to the limitations inherent in the few-shot
prompting method: the LLM struggles to suggest appropriate actions based
solely on a brief history of unsuccessful attempts and their adverse outcomes.

To circumvent this issue, we propose and implement a history clearing scheme.
Specifically, when the swimmer’s displacement within the last few (this number
is predetermined) steps falls below a threshold, such as $0.01$, we erase and
subsequently rebuild the entire record of prior few-shot demonstrations. In
practice, this simple approach has proven to be successful in preventing the
aforementioned entrapment.

###  .2 Avoiding floating-point or negative numbers

LLMs are known to struggle with comprehending floating-point or negative
numbers [2]. To address this issue, we implement a transformation to the
swimmer’s displacement $X/a$. Its raw value produced by the theoretical model
is a floating number truncated to three decimal places, a step taken to
curtail monetary expenses. By predefining a lower boundary,
$X_{\text{min}}/a$, for $X/a$, we apply the following transformation
$\left(X/a\right)_{\rm LLM}=\left(X/a-X_{\text{min}}/a\right)\times 1000$,
upon which the LLM receives a transformed value $\left(X/a\right)_{\rm LLM}$
as a positive integer.

###  .3 Saving money by using aliases extensively

Throughout our study, we have emphasized saving the monetary costs of using
GPT-4, primarily by minimizing the text length. We find that a simple
approach—extensively employing aliases—significantly aids in minimization. For
example, we use the alias ‘DOF’ in place of ‘degree of freedom’, and ‘ROC’ for
‘rate of change’. GPT-4 seems to fully comprehend such abbreviations, thereby
enabling the feasibility of this cheap strategy.

###  .4 Temperature (randomness) within GPT-4

In the GPT-4 architecture, the ‘temperature’ parameter indicative of the level
of randomness, can be adjusted within the range $[0,1]$. This adjustment
allows for a balance between consistent output and creative exploration by
LLM. Throughout this study, we have set the temperature parameter to zero,
focusing on deterministic evaluations.

Fig. 3: Influence of the length $n_{\text{ht}}$ of historical records on the
swimmers’ learning performance—the swimmer’s displacement in the target
direction, $X/a$ (positive $\mathbf{e}_{x}$, left column) or $-X/a$ (negative
$\mathbf{e}_{x}$, right column) versus the execution step $n$. The upper and
lower rows correspond to Purcell’s swimmer and NG’s swimmer, respectively.

## Results and observations

In this section, we demonstrate using GPT-4 to guide the swimming microrobots.
In general, the performance depends on their initial configuration, the number
of historical demonstrations $n_{\text{ht}}$, and the target direction
(positive or negative $\mathbf{e}_{x}$ here).

### LLM-prompted Swimming Strokes

As shown in Fig. 2, GPT-4 effectively directs both model microrobots to swim
consistently in a specific direction—positive $\mathbf{e}_{x}$ here.
Remarkably, using the LLM as the decision maker, both robots take only one
execution step to acquire the well-known signature cycle of strokes (see the
lower panels of Fig. 2). Such strokes, non-reciprocal in time, have originally
been perceived by physicists [48, 49] to overcome the low-Reynolds-number
constraints. Moreover, the LLM-based control dramatically outperforms the RL
approach in terms of the acquisition speed. Trained by a $Q$-learning scheme,
the RL agents for Purcell’s and NG’s swimmers require, respectively, $\approx
12$ and $\approx 40$ steps before learning these strokes. Though Fig. 2
illustrates the training process for one specific initial configuration of
both swimmers, we assert that the same prompt effectively function for all the
other three initial configurations.

In our LLM prompt, the number of historical demonstration, $n_{\text{ht}}$, is
an important parameter influencing the learning performance. A larger
$n_{\text{ht}}$ facilitates ICL, yet increasing the information exchanged
between the prompt and GPT-4, consequently raising the monetary cost. As
illustrated in Fig. 3A & B, Purcell’s swimmer, with $n_{\text{ht}}=1$,
successfully learns effective gaits when directed to swim in both the positive
and negative $\mathbf{e}_{x}$ directions. However, we refrain from selecting
$n_{\text{ht}}=1$ for other parts of this study. This decision is based on
observations that the learning performance degenerates when $n_{\text{ht}}$ is
increased to $2$ in both directions, suggesting that the success at
$n_{\text{ht}}=1$ is mostly attributed to a fortuitous coincidence rather than
a systematic pattern. Likewise, as shown in Fig. 3C & D, we opt for
$n_{\text{ht}}=3$ and $n_{\text{ht}}=6$ for NG’s swimmer when navigating in
the $\mathbf{e}_{x}$ and $-\mathbf{e}_{x}$ directions, respectively. To be
rigorous, we do not designate these selected $n_{\text{ht}}$ as the minimum
necessary number of historical shots.

We further examine the robustness of this LLM-based decision making process
under thermal noise in the swimming environment. Specifically, we randomly
perturb the swimmer’s incremental displacement $\Delta X$ following each
execution step, yielding a noisy movement $\Delta X\left(1+\zeta Y\right)$.
Here, $\zeta$ indicates the noise level, and $Y$ is a random number uniformly
distributed in $[-1,1]$.

For each noise level, we perform $10$ tasks, evaluating the average
displacement $\langle X\rangle/a$ and the overall success rate $p$. The number
of tasks, $10$, is selected to balance statistical significance with monetary
expenses. A task is considered successful if the swimmer learns to execute
three consecutive cycles of its signature gaits within $50$ steps, without
subsequent failures. Fig. 4 illustrates how the noise $\zeta$ affects the
swimmer’s learning performance. With increasing $\zeta$, we generally observe
a decrease in both $\langle X\rangle/a$ and $p$, although the decline in
performance is relatively modest. Notably, the success rate remains around
$p\approx 0.75$ at the highest noise level of $\zeta=3$, indicating the
overall resilience of this LLM-based approach.

Fig. 4: Average displacement $\langle X\rangle/a$ and overall success rate $p$
for Purcell’s swimmer (A) and NG’s swimmer (B) guided along the
$\mathbf{e}_{x}$ direction, under varying noise levels $\zeta$. For each
level, $10$ individual runs are conducted to obtain the statistics.

### Prompt Engineering

Fig. 5: Criticality of the five sentences, labeled S1 to S5, in the prompt.
Left column: how the success rate $p$ of Purcell’s swimmer (A) and NG’s
swimmer (C) guided in the $\mathbf{e}_{x}$ direction is affected when a single
sentence is removed from the prompt, in comparison to the results obtained
from the complete prompt (12 o’clock in the pie chart). The right column
mirrors the left, but instead evaluates the impact on the average displacement
$\langle X\rangle/a$, rather than the success rate $p$.

We note that our prompt is significantly simpler and shorter than those in
related works [1, 2], mostly owing to the simplicity of our physical
environment. The major simplification we have made is summarized below. First,
unlike these works, we do not use a system log to provide an overview of the
task. It turns out to be unnecessary and is thus removed for reducing monetary
cost. Secondly, we do not employ a trained controller as in Ref. [2] to
collect the initial batch of historical demonstrations fed to the LLM. Neither
do we incorporate any expert demonstrations dataset as in Ref. [1] that
facilitates the LLM to learn the working principles underlying expert
policies.

Furthermore, we have investigated whether the designed prompt is minimal by
assessing the indispensability of each of the five sentences, labeled S1 to
S5. To evaluate the necessity of an individual sentence, we conducted $10$
runs with the respective sentence omitted from the prompt. We then analyzed
the overall success rate $p$ and average displacement $\langle X\rangle/a$ of
both swimmers, as shown in Fig. 5. Our comparative analysis reveals that
sentences S2, S3, and S5 are critically indispensable; the absence of any of
these results in failure of all runs. Conversely, sentences S1 and S4
demonstrate weaker indispensability, with occasional task success despite
their removal.

## Conclusions and Discussions

In this work, we have demonstrated the hitherto unreported usage of LLMs for
effectuating low-Reynolds-number propulsion of model microswimmers. Employing
a few-shot prompting approach, we have designed a five-sentence prompt for two
minimal swimmers: Purcell’s three-link swimmer and NG’s three-sphere swimmer.
This unified prompt enables GPT-4 to generate through ICL, the optimal stroke
patterns for both swimmers, despite their distinct propulsion mechanisms.

Based on the current successful demonstration of using LLMs for microrobotic
manipulation, we identify several potentially promising research directions:
1) transitioning from a discrete to a continuous action space; 2) steering
microrobots to navigate through complex environments encompassing non-
Newtonian fluid rheology, complex confinements, and obstacles; 3) facilitating
cooperative swimming behaviors among microrobots.

We admit that certain hyperparameters in this preliminary study were selected
arbitrarily. For instance, we set the temperature parameter of GPT-4 to zero.
In our forthcoming research, we aim to examine the influence of this
temperature setting and explore whether increasing it could potentially
improve the exploratory capabilities of the LLM.

## Materials and Methods

### Hydrodynamics of Purcell’s Swimmer

For Purcell’s swimmer, the middle link moves with a translational velocity
$\dot{\mathbf{r}}_{\text{c}}=u\mathbf{e}_{x}+v\mathbf{e}_{y}$ and rotates with
an angular velocity $\dot{d}_{\text{c}}\mathbf{e}_{z}$ about its center. The
two velocities as a function of actions $\left(\dot{d}_{1},\dot{d}_{2}\right)$
are calculated by Mathematica and are given by Eq. S(2), Eq. S(3), and Eq.
S(4). For brevity, all the variables in these equations are dimensionless,
where we have chosen the rod length $a$ as the characteristic length,
$T/2d_{\text{max}}$ the characteristic time, and $2d_{\text{max}}/T$ the
characteristic angular velocity.

$\begin{split}u=\frac{1}{2\Delta}\left(\dot{d}_{1}\left(262\sin\left(d_{\text{c}}-d_{1}\right)-26\sin\left(d_{1}+d_{\text{c}}\right)-2\sin\left(2d_{1}+d_{\text{c}}\right)+18\sin\left(d_{\text{c}}-2d_{1}\right)-16\sin\left(d_{\text{c}}-d_{2}\right)+78\sin\left(-d_{1}-d_{2}+d_{\text{c}}\right)\right.\right.\quad&\\\
-18\sin\left(d_{1}-d_{2}+d_{\text{c}}\right)+8\sin\left(-2d_{1}-d_{2}+d_{\text{c}}\right)+104\sin\left(d_{2}+d_{\text{c}}\right)+126\sin\left(-d_{1}+d_{2}+d_{\text{c}}\right)+30\sin\left(d_{1}+d_{2}+d_{\text{c}}\right)&\\\
+24\sin\left(2d_{2}+d_{\text{c}}\right)+21\sin\left(-d_{1}+2d_{2}+d_{\text{c}}\right)+\sin\left(d_{1}+2d_{2}+d_{\text{c}}\right)-2\sin\left(2d_{1}+2d_{2}+d_{\text{c}}\right)-4\sin\left(d_{\text{c}}-2d_{2}\right)&\\\
\left.+9\sin\left(-d_{1}-2d_{2}+d_{\text{c}}\right)-3\sin\left(d_{1}-2d_{2}+d_{\text{c}}\right)+2\sin\left(-2d_{1}-2d_{2}+d_{\text{c}}\right)+36\sin(d_{\text{c}})\right)+\dot{d}_{2}\left(104\sin\left(d_{\text{c}}-d_{1}\right)-16\sin\left(d_{1}+d_{\text{c}}\right)\right.&\\\
-4\sin\left(2d_{1}+d_{\text{c}}\right)+24\sin\left(d_{\text{c}}-2d_{1}\right)-26\sin\left(d_{\text{c}}-d_{2}\right)+30\sin\left(-d_{1}-d_{2}+d_{\text{c}}\right)-18\sin\left(d_{1}-d_{2}+d_{\text{c}}\right)-3\sin\left(2d_{1}-d_{2}+d_{\text{c}}\right)&\\\
+\sin\left(-2d_{1}-d_{2}+d_{\text{c}}\right)+262\sin\left(d_{2}+d_{\text{c}}\right)+126\sin\left(-d_{1}+d_{2}+d_{\text{c}}\right)+78\sin\left(d_{1}+d_{2}+d_{\text{c}}\right)+9\sin\left(2d_{1}+d_{2}+d_{\text{c}}\right)&\\\
+21\sin\left(-2d_{1}+d_{2}+d_{\text{c}}\right)+18\sin\left(2d_{2}+d_{\text{c}}\right)+8\sin\left(d_{1}+2d_{2}+d_{\text{c}}\right)+2\sin\left(2d_{1}+2d_{2}+d_{\text{c}}\right)-2\sin\left(d_{\text{c}}-2d_{2}\right)&\\\
\left.\left.-2\sin\left(-2d_{1}-2d_{2}+d_{\text{c}}\right)+36\sin(d_{\text{c}})\right)\right).\end{split}$
(2)
$\begin{split}v=-\frac{1}{2\Delta}\left(\dot{d}_{1}\left(262\cos\left(d_{\text{c}}-d_{1}\right)-26\cos\left(d_{1}+d_{\text{c}}\right)-2\cos\left(2d_{1}+d_{\text{c}}\right)+18\cos\left(d_{\text{c}}-2d_{1}\right)-16\cos\left(d_{\text{c}}-d_{2}\right)+78\cos\left(-d_{1}-d_{2}+d_{\text{c}}\right)\right.\right.&\\\
-18\cos\left(d_{1}-d_{2}+d_{\text{c}}\right)+8\cos\left(-2d_{1}-d_{2}+d_{\text{c}}\right)+104\cos\left(d_{2}+d_{\text{c}}\right)+126\cos\left(-d_{1}+d_{2}+d_{\text{c}}\right)+30\cos\left(d_{1}+d_{2}+d_{\text{c}}\right)&\\\
+24\cos\left(2d_{2}+d_{\text{c}}\right)+21\cos\left(-d_{1}+2d_{2}+d_{\text{c}}\right)+\cos\left(d_{1}+2d_{2}+d_{\text{c}}\right)-2\cos\left(2d_{1}+2d_{2}+d_{\text{c}}\right)-4\cos\left(d_{\text{c}}-2d_{2}\right)&\\\
\left.+9\cos\left(-d_{1}-2d_{2}+d_{\text{c}}\right)-3\cos\left(d_{1}-2d_{2}+d_{\text{c}}\right)+2\cos\left(-2d_{1}-2d_{2}+d_{\text{c}}\right)+36\cos(d_{\text{c}})\right)+\dot{d}_{2}\left(104\cos\left(d_{\text{c}}-d_{1}\right)-16\cos\left(d_{1}+d_{\text{c}}\right)\right.&\\\
-4\cos\left(2d_{1}+d_{\text{c}}\right)+24\cos\left(d_{\text{c}}-2d_{1}\right)-26\cos\left(d_{\text{c}}-d_{2}\right)+30\cos\left(-d_{1}-d_{2}+d_{\text{c}}\right)-18\cos\left(d_{1}-d_{2}+d_{\text{c}}\right)-3\cos\left(2d_{1}-d_{2}+d_{\text{c}}\right)&\\\
+\cos\left(-2d_{1}-d_{2}+d_{\text{c}}\right)+262\cos\left(d_{2}+d_{\text{c}}\right)+126\cos\left(-d_{1}+d_{2}+d_{\text{c}}\right)+78\cos\left(d_{1}+d_{2}+d_{\text{c}}\right)+9\cos\left(2d_{1}+d_{2}+d_{\text{c}}\right)&\\\
+21\cos\left(-2d_{1}+d_{2}+d_{\text{c}}\right)+18\cos\left(2d_{2}+d_{\text{c}}\right)+8\cos\left(d_{1}+2d_{2}+d_{\text{c}}\right)+2\cos\left(2d_{1}+2d_{2}+d_{\text{c}}\right)-2\cos\left(d_{\text{c}}-2d_{2}\right)&\\\
\left.\left.-2\cos\left(-2d_{1}-2d_{2}+d_{\text{c}}\right)+36\cos(d_{\text{c}})\right)\right).\end{split}$
(3)
$\begin{split}\dot{d}_{\text{c}}=\frac{2}{\Delta}\left(\dot{d}_{1}\left(102\cos\left(d_{1}\right)+2\cos\left(2d_{1}\right)-6\cos\left(d_{1}-d_{2}\right)-4\cos\left(2d_{2}\right)+42\cos\left(d_{1}+d_{2}\right)+2\cos\left(2\left(d_{1}+d_{2}\right)\right)+9\cos\left(d_{1}+2d_{2}\right)\right.\right.\phantom{xxx}&\\\
\left.-3\cos\left(d_{1}-2d_{2}\right)+108\right)+\dot{d}_{2}\left(4\cos\left(2d_{1}\right)+6\cos\left(d_{1}-d_{2}\right)+3\cos\left(2d_{1}-d_{2}\right)-102\cos\left(d_{2}\right)-2\cos\left(2d_{2}\right)-42\cos\left(d_{1}+d_{2}\right)\right.&\\\
\left.\left.-2\cos\left(2\left(d_{1}+d_{2}\right)\right)-9\cos\left(2d_{1}+d_{2}\right)-108\right)\right).\end{split}$
(4)
$\begin{split}\Delta=3\left(136\cos\left(d_{1}\right)+14\cos\left(2d_{1}\right)-8\cos\left(d_{1}-d_{2}\right)-\cos\left(2\left(d_{1}-d_{2}\right)\right)-4\cos\left(2d_{1}-d_{2}\right)+14\cos\left(2d_{2}\right)+56\cos\left(d_{1}+d_{2}\right)\right.\phantom{XXXX}&\\\
\left.+136\cos\left(d_{2}\right)+3\cos\left(2\left(d_{1}+d_{2}\right)\right)+12\cos\left(2d_{1}+d_{2}\right)+12\cos\left(d_{1}+2d_{2}\right)-4\cos\left(d_{1}-2d_{2}\right)+282\right).\end{split}$
(5)

### Hydrodynamics of NG’s Swimmer

For NG’s swimmer, we allow only one DOF, either $d_{1}$ or $d_{2}$, to vary at
a time. The selected DOF is restricted to a discrete set of lengths,
$\left[d_{\text{min}},d_{\text{max}}\right]$. Hence, the action is also
discrete and drawn from
$\left[-(d_{\text{max}}-d_{\text{min}})/T,0,(d_{\text{max}}-d_{\text{min}})/T\right]$,
where $T$ is the time a rod takes to change its length. Following Ref. [51],
the dimensionless velocity of the swimmer reads:

$\begin{split}\dot{\mathbf{r}}_{\text{c}}=\frac{1}{6}\left[\left(\frac{\dot{d}_{2}-\dot{d}_{1}}{d_{1}+d_{2}}\right)+2\left(\frac{\dot{d}_{1}}{d_{2}}-\frac{\dot{d}_{2}}{d_{1}}\right)\right],\end{split}$
(6)

where we have selected $a$ as the characteristic length,
$\left(d_{\text{max}}-d_{\text{min}}\right)/T$ as the characteristic velocity,
and $aT/\left(d_{\text{max}}-d_{\text{min}}\right)$ as the characteristic time
for the non-dimensionalization.

### Data Availability

The datasets used and/or analysed during the current study available from the
corresponding author on reasonable request.

### Code Availability

The GPT-4 prompt will be open-sourced upon the acceptance of this manuscript.

## References

  * [1] L. Song, C. Zhang, L. Zhao, and J. Bian, “Pre-trained large language models for industrial control,” arXiv preprint arXiv:2308.03028, 2023.
  * [2] Y.-J. Wang, B. Zhang, J. Chen, and K. Sreenath, “Prompt a robot to walk with large language models,” arXiv preprint arXiv:2309.09969, 2023.
  * [3] C. Abdallah, D. M. Dawson, P. Dorato, and M. Jamshidi, “Survey of robust control for rigid robots,” IEEE Control Systems Magazine, vol. 11, no. 2, pp. 24–30, 1991.
  * [4] P.-B. Wieber, R. Tedrake, and S. Kuindersma, “Modeling and control of legged robots,” in Springer handbook of robotics, pp. 1203–1234, Springer, 2016\.
  * [5] E. R. Westervelt, J. W. Grizzle, C. Chevallereau, J. H. Choi, and B. Morris, Feedback Control of Dynamic Bipedal Robot Locomotion.  CRC press, 2018.
  * [6] H. Benbrahim and J. A. Franklin, “Biped dynamic walking using reinforcement learning,” Robotics and Autonomous systems, vol. 22, no. 3-4, pp. 283–302, 1997.
  * [7] H. Kimura, T. Yamashita, and S. Kobayashi, “Reinforcement learning of walking behavior for a four-legged robot,” IEEJ Transactions on Electronics, Information and Systems, vol. 122, no. 3, pp. 330–337, 2002.
  * [8] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” arXiv preprint arXiv:1812.11103, 2018.
  * [9] J. Xu, T. Du, M. Foshey, B. Li, B. Zhu, A. Schulz, and W. Matusik, “Learning to fly: Computational controller design for hybrid UAVs with reinforcement learning,” ACM Trans. Graph., vol. 38, no. 4, pp. 1–12, 2019.
  * [10] P. Becker-Ehmck, M. Karl, J. Peters, and P. van der Smagt, “Learning to fly via deep model-based reinforcement learning,” arXiv preprint arXiv:2003.08876, 2020.
  * [11] M. Gazzola, A. A. Tchieu, D. Alexeev, A. de Brauer, and P. Koumoutsakos, “Learning to school in the presence of hydrodynamic interactions,” J. Fluid Mech., vol. 789, pp. 726–749, 2016.
  * [12] S. Colabrese, K. Gustavsson, A. Celani, and L. Biferale, “Flow navigation by smart microswimmers via reinforcement learning,” Phys. Rev. Lett., vol. 118, no. 15, p. 158004, 2017.
  * [13] S. Verma, G. Novati, and P. Koumoutsakos, “Efficient collective swimming by harnessing vortices through deep reinforcement learning,” Proc. Natl. Acad. Sci. USA, vol. 115, no. 23, pp. 5849–5854, 2018.
  * [14] E. Schneider and H. Stark, “Optimal steering of a smart active particle,” EPL (Europhysics Letters), vol. 127, no. 6, p. 64003, 2019.
  * [15] M. Mirzakhanloo, S. Esmaeilzadeh, and M.-R. Alam, “Active cloaking in Stokes flows via reinforcement learning,” J. Fluid Mech., vol. 903, 2020.
  * [16] A. C. H. Tsang, P. W. Tong, S. Nallan, and O. S. Pak, “Self-learning how to swim at low Reynolds number,” Phys. Rev. Fluids, vol. 5, no. 7, p. 074101, 2020.
  * [17] J. Qiu, W. Huang, C. Xu, and L. Zhao, “Swimming strategy of settling elongated micro-swimmers by reinforcement learning,” Sci. China: Phys. Mech. Astron., vol. 63, no. 8, pp. 1–9, 2020.
  * [18] Y. Jiao, F. Ling, S. Heydari, N. Heess, J. Merel, and E. Kanso, “Learning to swim in potential flow,” Phys. Rev. Fluids, vol. 6, no. 5, p. 050505, 2021\.
  * [19] H. Deng, P. Burke, D. Li, and B. Cheng, “Design and experimental learning of swimming gaits for a magnetic, modular, undulatory robot,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9562–9568, IEEE, 2021.
  * [20] J. K. Alageshan, A. K. Verma, J. Bec, and R. Pandit, “Machine learning strategies for path-planning microswimmers in turbulent flows,” Phys. Rev. E, vol. 101, no. 4, p. 043110, 2020.
  * [21] S. Muiños-Landin, A. Fischer, V. Holubec, and F. Cichos, “Reinforcement learning with artificial microswimmers,” Sci. Rob., vol. 6, no. 52, 2021\.
  * [22] M. Gerhard, A. Jayaram, A. Fischer, and T. Speck, “Hunting active Brownian particles: Learning optimal behavior,” Phys. Rev. E, vol. 104, no. 5, p. 054614, 2021.
  * [23] P. Gunnarson, I. Mandralis, G. Novati, P. Koumoutsakos, and J. O. Dabiri, “Learning efficient navigation in vortical flow fields,” Nat. Commun, vol. 12, no. 1, p. 7143, 2021.
  * [24] M. R. Behrens and W. C. Ruder, “Smart magnetic microrobots learn to swim with deep reinforcement learning,” Adv. Intell. Syst., vol. 4, no. 10, p. 2200023, 2022.
  * [25] W. Liu, K. Bai, X. He, S. Song, C. Zheng, and X. Liu, “Fishgym: A high-performance physics-based simulation framework for underwater robot learning,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 6268–6275, IEEE, 2022.
  * [26] Y. Zhu, J.-H. Pang, T. Gao, and F.-B. Tian, “Learning to school in dense configurations with multi-agent deep reinforcement learning,” Bioinspir. Biomim., vol. 18, no. 1, p. 015003, 2022.
  * [27] Q. Wang, Z. Hong, and Y. Zhong, “Learn to swim: Online motion control of an underactuated robotic eel based on deep reinforcement learning,” BIROB, vol. 2, no. 4, p. 100066, 2022.
  * [28] F. Borra, L. Biferale, M. Cencini, and A. Celani, “Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number,” Phys. Rev. Fluids, vol. 7, no. 2, p. 023103, 2022.
  * [29] G. Chen, Y. Lu, X. Yang, and H. Hu, “Reinforcement learning control for the swimming motions of a beaver-like, single-legged robot based on biological inspiration,” Rob Auton Syst, vol. 154, p. 104116, 2022.
  * [30] Z. Zou, Y. Liu, Y.-N. Young, O. S. Pak, and A. C. H. Tsang, “Gait switching and targeted navigation of microswimmers via deep reinforcement learning,” Commun. Phys., vol. 5, no. 1, p. 158, 2022.
  * [31] M. Nasiri and B. Liebchen, “Reinforcement learning of optimal active particle navigation,” New J. Phys., vol. 24, no. 7, p. 073042, 2022.
  * [32] G. Zhu, W.-Z. Fang, and L. Zhu, “Optimizing low-Reynolds-number predation via optimal control and reinforcement learning,” J. Fluid Mech., vol. 944, p. A3, 2022.
  * [33] K. Qin, Z. Zou, L. Zhu, and O. S. Pak, “Reinforcement learning of a multi-link swimmer at low Reynolds numbers,” Phys. Fluids, vol. 35, no. 3, 2023\.
  * [34] Z. E. Khiyati, R. Chesneaux, L. Giraldi, and J. Bec, “Steering undulatory micro-swimmers in a fluid flow through reinforcement learning,” Eur. Phys. J. E, vol. 46, no. 6, p. 43, 2023.
  * [35] C. Mo, Q. Fu, and X. Bian, “Chemotaxis of an elastic flagellated microrobot,” Phys. Rev. E, vol. 108, no. 4, p. 044408, 2023.
  * [36] J. Hu and T. Dear, “Guided deep reinforcement learning for articulated swimming robots,” arXiv preprint arXiv:2301.13072, 2023.
  * [37] L.-S. Lin, K. Yasuda, K. Ishimoto, and S. Komura, “Emergence of odd elasticity in a microswimmer using deep reinforcement learning,” arXiv preprint arXiv:2311.01973, 2023.
  * [38] A. Xu, H.-L. Wu, and H.-D. Xi, “Control the migration of self-propelling particles in thermal turbulence via reinforcement learning algorithm,” in Proceedings of the IUTAM Symposium on Turbulent Structure and Particles-Turbulence Interaction (X. Zheng and S. Balachandar., eds.), (Cham), pp. 313–325, Springer Nature Switzerland, 2024.
  * [39] Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022\.
  * [40] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 24824–24837, 2022.
  * [41] M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, et al., “Challenging BIG-Bench tasks and whether chain-of-thought can solve them,” arXiv preprint arXiv:2210.09261, 2022.
  * [42] A. Creswell, M. Shanahan, and I. Higgins, “Selection-inference: Exploiting large language models for interpretable logical reasoning,” arXiv preprint arXiv:2205.09712, 2022.
  * [43] A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al., “Solving quantitative reasoning problems with language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 3843–3857, 2022.
  * [44] R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Trans. Affect. Comput., 2022.
  * [45] S. Agrawal, C. Zhou, M. Lewis, L. Zettlemoyer, and M. Ghazvininejad, “In-context examples selection for machine translation,” arXiv preprint arXiv:2212.02437, 2022.
  * [46] P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in CHI Ext. Abstr., pp. 1–7, 2022.
  * [47] Z. Liu, X. Yu, L. Zhang, Z. Wu, C. Cao, H. Dai, L. Zhao, W. Liu, D. Shen, Q. Li, et al., “Deid-GPT: Zero-shot medical text de-identification by GPT-4,” arXiv preprint arXiv:2303.11032, 2023.
  * [48] E. M. Purcell, “Life at low Reynolds number,” Am. J. Phys., vol. 45, no. 1, pp. 3–11, 1977.
  * [49] A. Najafi and R. Golestanian, “Simple swimmer at low Reynolds number: Three linked spheres,” Phys. Rev. E, vol. 69, no. 6, p. 062901, 2004.
  * [50] S. Mirchandani, F. Xia, P. Florence, B. Ichter, D. Driess, M. G. Arenas, K. Rao, D. Sadigh, and A. Zeng, “Large language models as general pattern machines,” arXiv preprint arXiv:2307.04721, 2023.
  * [51] R. Golestanian and A. Ajdari, “Analytic results for the three-sphere swimmer at low Reynolds number,” Phys. Rev. E, vol. 77, no. 3, p. 036308, 2008\.


Ginpagebox=artbox

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

# Intrinsic Quality Assessment of Arguments

Henning Wachsmuth
Department of Computer Science
Paderborn University
Paderborn, Germany
<EMAIL_ADDRESS>
&Till Werner
Department of Computer Science
Paderborn University
Paderborn, Germany
<EMAIL_ADDRESS>

###### Abstract

Several quality dimensions of natural language arguments have been
investigated. Some are likely to be reflected in linguistic features (e.g., an
argument’s arrangement), whereas others depend on context (e.g., relevance) or
topic knowledge (e.g., acceptability). In this paper, we study the intrinsic
computational assessment of 15 dimensions, i.e., only learning from an
argument’s text. In systematic experiments with eight feature types on an
existing corpus, we observe moderate but significant learning success for most
dimensions. Rhetorical quality seems hardest to assess, and subjectivity
features turn out strong, although length bias in the corpus impedes full
validity. We also find that human assessors differ more clearly to each other
than to our approach.

##  1 Introduction

Good arguments help to persuade people, to compromise, or to at least
understand each other better. What quality dimension is meant by good depends
on the setting, though [van Eemeren and Grootendorst, 2004, Johnson and Blair,
2006]. Several dimensions may be assessed computationally, as we exemplify for
an argument in favor of advancing the common good, taken from ?):

> “While striving to make advancements for the common good you can change the
> world forever. Allot of people have succeded in doing so. Our founding
> fathers, Thomas Edison, George Washington, Martin Luther King jr, and many
> more. These people made huge advances for the common good and they are
> honored for it.”

The argument is well-organized [Persing et al., 2010], its premises are
certainly largely acceptable [Yang et al., 2019] and relevant to the topic
[Wachsmuth et al., 2017c]. Whether they suffice to draw the conclusion [Stab
and Gurevych, 2017] is another question, let alone how convincing the argument
is [Habernal and Gurevych, 2016b]. Some dimensions may be reflected in
linguistic features of an argument’s text. Others depend on context, require
topic or background knowledge, or are inherently subjective.

In this paper, we benchmark what quality dimensions of an argument can be
assessed intrinsically, i.e., when analyzing the text of an argument only.
Given the corpus of ?) with 304 English debate portal arguments on 16 topics
scored for 15 logical, rhetorical, and dialectical quality dimensions by three
experts, we carry out systematic leave-one-topic-out cross-validation
experiments. In particular, we learn standard supervised score regression on
various text features; from content and distributional semantics, to style,
structure, and length, to text quality, evidence, and subjectivity.

For 11 dimensions, we observe moderate but significant prediction gains of the
features over a mean baseline. Following intuition, rhetorical quality related
to credibility and emotions seem hardest to assess. Features capturing
subjectivity (e.g., sentiment and pronoun usage) turn out particularly
effective. Even better performs the length feature, though, revealing bias in
the corpus and matching previous findings on other corpora [Potash et al.,
2017]. Follow-up experiments indicate that the experts strongly differ in
their assessability, and that some beat our features only slightly in
assessing quality. Altogether, an intrinsic computational assessment of
argument quality seems useful but not enough alone.

##  2 Related Work

Soon after the rise of argument mining, argument quality assessment has come
up as a task [Stede and Schneider, 2018], due to its importance for
applications such as Project Debater [Gleize et al., 2019]. It rests on
extensive theoretical discussions about what good arguments [Johnson and
Blair, 2006] and bad arguments are [Walton, 2006], and how to argue reasonably
[van Eemeren and Grootendorst, 2004].

Several corpora and approaches were proposed for specific argument quality
dimensions, first related to essay scoring [Persing and Ng, 2015], some of
which modeling arguments explicitly [Wachsmuth et al., 2016]. Later approaches
targeted arguments from debate portals [Wei et al., 2016], student essays
[Stab and Gurevych, 2017], mixed web texts [Wachsmuth et al., 2017c], and news
editorials [Yang et al., 2019, El Baff et al., 2020]. We use the corpus of ?),
as it is the only one annotated for diverse dimensions and is claimed to
reflect argument quality comprehensively. In follow-up work, ?) found
correlations with the convincingness reasons of ?), and ?) as well as ?) have
evaluated their annotations against the quality annotation scheme of the
corpus. However, we do not know any previous assessment approach developed on
the corpus, possibly due to its limited size (see Section 3).

We focus on features intrinsic to an argument’s text. This complements the
study of ?) who employed external knowledge to assess convincingness. Like
them, we find that longer arguments tend to be judged better. ?) limit
arguments to at most 36 words, avoiding length bias but also preventing deeper
reasoning. Quality in their corpus reflects which argument is in doubt
preferred. Ultimately, such judgments remain subjective [Lukin et al., 2017].
To alleviate this, ?) encode the reader’s ideology and personality, but such
information is often not given in practice.

##  3 Data

The corpus of ?) is a subset of 320 debate portal posts from the dataset of
?), 20 each for 16 controversial topics. Three human experts (all authors of
the paper) scored all posts that they saw as arguments for the following 15
logical, rhetorical, and dialectical quality dimensions on a scale from 1
(low) to 3 (high). In line with the experiments of ?), we use only those 304
texts in Section 5 that were seen as arguments by all three experts.

### Logic

The main logical dimension is called cogency (Cog). It is defined based on
three subdimensions: the local acceptability (LAc) of the truth of an
argument’s premises, the premises’ local relevance (LRe) for the argument’s
conclusion, and their local sufficiency (LSu) to infer the conclusion.

### Rhetoric

The main rhetorical dimension is the effectiveness (Eff) in persuading
readers. Subdimensions are the argument’s clarity (Cla), the author’s
credibility (Cre), the appropriateness (App) of the argument’s used language,
its success in emotional appeal (Emo), and its sequential arrangement (Arr) in
the text.

### Dialectic

The main dialectical dimension is reasonableness (Rea), with three
subdimensions: the global acceptability (GAc) of stating the argument when
discussing a given issue, the argument’s global relevance (GRe) for achieving
agreement, and its global sufficiency (GSu) in discussing both sides of the
issue.

### Overall

The overall quality (OvQ) reflects the subjective weighting of all 14 other
quality dimensions.

Both the single expert scores and the mean score are provided for each
dimension. The inter-annotator agreement for the different dimensions ranged
from 0.26 (emotional appeal) to 0.51 (overall quality) in terms of
Krippendorff’s $\alpha$. The majority score of most dimensions is 2, but both
score 1 and 3 also occur frequently for many of them. Matching the
hierarchical idea of the dimensions, overall quality correlates strongest with
cogency, effectiveness, and reasonableness. For details on agreements, score
distributions, and correlations, we refer the reader to the original paper
[Wachsmuth et al., 2017b].

##  4 Approach

This paper does not aim to propose a novel quality assessment approach, but to
evaluate what features of a text help to assess which quality dimension.
External knowledge is included only via lexicons and embeddings. As an
example, Figure 1 shows selected textual aspects of the argument from Section
1 that may be predictive of certain dimensions. We quantify these and other
aspects in the following eight feature types that are employed in linear SVMs
for score regression [Chang and Lin, 2011]:111We also tested some pretrained
configurations of BERT [Devlin et al., 2018], fine-tuning them during
training. However, performance was low, possibly due to the small data and the
prevention of learning topic information in our experimental setup.

Figure 1: Exemplary analysis of an argument from the used corpus for selected
text features that might affect quality: content-related key phrases, text
quality indicators such as spelling errors, subjective pronoun usage, length
in sentences/tokens, and evidence distributions reflected by premises and
conclusions. On the right, all mean quality scores of the argument in the
corpus are shown (worst is 1, best is 3).

### Content

As often done in text classification [Aggarwal and Zhai, 2012], we aim to
capture important content-related key phrases simply as part of the
distribution of word 1- to 3-grams, taking all those that occur in $\geq$ 3%
of all training texts (such thresholds were set after initial tests).

### Embedding

We capture an argument’s distributional semantics by a sentence vector, using
the pretrained fasttext model based on Wikipedia [Mikolov et al., 2018]. Each
vector position becomes one feature.

### Style

We model style with part-of-speech 1- to 3-grams ($\geq$10% training
frequency) and character 1- to 3-grams ($\geq$3%). Both are common stylometric
features in authorship attribution [Stamatatos, 2009].

### Structure

In terms of structure, we look for enumeration indicators, such as “1.” and
“2.” or “on one hand” and “on the other hand”. In addition, we check the first
token 1-, 2-, and 3-gram in the text.

### Length

Our length feature type includes normalized counts of characters, syllables,
tokens, phrases, sentences, and paragraphs, along with ratios between each
pair of these linguistic units.

### Text Quality

Motivated by classical essay scoring [Ke and Ng, 2019], we model text quality
by spelling correctness and readability. The former is quantified as absolute
and relative counts of three error types from www.languagetool.org (hints,
unknown words, and others). For the latter, we calculate 10 common readability
scores, including Flesch Kincaid Reading Ease, Gunning Fog Index, LIX, and
similar.

### Evidence

On one hand, we capture the evidence given in an argument in terms of the
frequency of links. On the other hand, we apply an out-of-the-box argument
mining algorithm [Wachsmuth et al., 2016] that classifies each sentence into
one of four argumentative unit types: thesis, conclusion, premise, or none.

### Subjectivity

Since we hypothesized subjectivity to be important, we here combine multiple
frequency distributions: (a) singular and plural 1st, 2nd, and 3rd person
pronouns, (b) indicators of positivity, negativity, and hedging (e.g.,
“certainly”) based on ?), (c) 83 different emojis, (d) fully lower-case,
upper-case, and other words, and (e) character types, such as letter, digit,
and whitespace.

##  5 Experiments

We now report on experiments with the features from Section 4 on the corpus
from Section 3. In particular, we systematically study three research
questions for the 15 given argument quality dimensions:

  1. Q1.

To what extent can each quality dimension be assessed only from an argument’s
text?

  2. Q2.

How dependent is the assessability on the subjective view of the experts?

  3. Q3.

How well do the considered features predict argument quality compared to
humans?

### Experimental Setup

We approached all 15 dimensions using each feature type alone, feature
ablation (all but one type), and all features respectively. We split the
corpus into 16 test sets, one per topic. For each approach and topic, we
trained one SVM on the other 15 topics, optimizing its $C$ hyperparameter in
15-fold cross-validation on the training set (tested $C$ range: $10^{-4}\cdot
2^{j}$ for $7\leq j\leq 16$). We then computed the mean absolute error (MAE),
averaged over the 16 MAEs on each test set. This leave-one-topic-out way
ensures that no topic information can be exploited in the assessment on the
test sets.

To focus on the learning success, we compare the features only to the mean
baseline, which always predicts the mean score of all arguments in the given
training set. For the SVM with all features, we use the 16 single MAEs in a
one-tailed independent $t$-test to test whether improvements over the baseline
are significantly better at $p<.05$ (marked $\dagger$ below) and $p<.01$
($\ddagger$). The Java code for reproducing the experiments can be accessed
here: http://arguana.com/software

|  |  | Logical quality |  | Rhetorical quality |  | Dialectical quality |  |
---|---|---|---|---|---|---|---|---|---
# | Approach |  | Cog | LAc | LRe | LSu |  | Eff | Cla | Cre | App | Emo | Arr |  | Rea | GAc | GRe | GSu |  | OvQ
$A_{1}$ | Content |  | 0.38 | 0.42 | 0.43 | 0.32 |  | 0.34 | 0.38 | 0.31 | 0.36 | 0.30 | 0.36 |  | 0.39 | 0.41 | 0.40 | 0.24 |  | 0.39
$A_{2}$ | Embedding |  | 0.46 | 0.44 | 0.48 | 0.38 |  | 0.39 | 0.38 | 0.32 | 0.36 | 0.28 | 0.39 |  | 0.45 | 0.47 | 0.44 | 0.28 |  | 0.45
$A_{3}$ | Style |  | 0.38 | 0.41 | 0.44 | 0.31 |  | 0.33 | 0.39 | 0.30 | 0.36 | 0.29 | 0.36 |  | 0.38 | 0.40 | 0.38 | 0.23 |  | 0.38
$A_{4}$ | Structure |  | 0.45 | 0.46 | 0.49 | 0.38 |  | 0.39 | 0.39 | 0.34 | 0.38 | 0.30 | 0.41 |  | 0.46 | 0.46 | 0.44 | 0.28 |  | 0.46
$A_{5}$ | Length |  | 0.37 | 0.40 | 0.43 | 0.30 |  | 0.33 | 0.38 | 0.30 | 0.34 | 0.28 | 0.35 |  | 0.36 | 0.38 | 0.37 | 0.22 |  | 0.36
$A_{6}$ | Text quality |  | 0.44 | 0.45 | 0.48 | 0.37 |  | 0.38 | 0.38 | 0.33 | 0.37 | 0.31 | 0.39 |  | 0.45 | 0.44 | 0.44 | 0.27 |  | 0.44
$A_{7}$ | Evidence |  | 0.42 | 0.44 | 0.46 | 0.35 |  | 0.37 | 0.40 | 0.33 | 0.38 | 0.30 | 0.38 |  | 0.42 | 0.45 | 0.40 | 0.24 |  | 0.42
$A_{8}$ | Subjectivity |  | 0.36 | 0.40 | 0.41 | 0.31 |  | 0.33 | 0.39 | 0.33 | 0.35 | 0.29 | 0.36 |  | 0.37 | 0.40 | 0.38 | 0.22 |  | 0.37
$A_{\backslash 1}$ | w/o Content |  | 0.37 | 0.40 | 0.43 | 0.30 |  | 0.33 | 0.36 | 0.30 | 0.33 | 0.29 | 0.35 |  | 0.37 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 2}$ | w/o Embedding |  | 0.37 | 0.40 | 0.42 | 0.30 |  | 0.33 | 0.36 | 0.30 | 0.33 | 0.29 | 0.35 |  | 0.37 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 3}$ | w/o Style |  | 0.36 | 0.40 | 0.42 | 0.30 |  | 0.33 | 0.36 | 0.31 | 0.33 | 0.29 | 0.34 |  | 0.37 | 0.39 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 4}$ | w/o Structure |  | 0.36 | 0.40 | 0.42 | 0.30 |  | 0.33 | 0.37 | 0.29 | 0.33 | 0.29 | 0.34 |  | 0.37 | 0.39 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 5}$ | w/o Length |  | 0.36 | 0.41 | 0.42 | 0.30 |  | 0.33 | 0.36 | 0.30 | 0.33 | 0.28 | 0.34 |  | 0.37 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 6}$ | w/o Text quality |  | 0.37 | 0.40 | 0.42 | 0.30 |  | 0.33 | 0.37 | 0.30 | 0.34 | 0.28 | 0.35 |  | 0.37 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 7}$ | w/o Evidence |  | 0.36 | 0.40 | 0.42 | 0.30 |  | 0.33 | 0.36 | 0.30 | 0.33 | 0.28 | 0.35 |  | 0.38 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{\backslash 8}$ | w/o Subjectivity |  | 0.37 | 0.41 | 0.43 | 0.31 |  | 0.33 | 0.37 | 0.30 | 0.34 | 0.29 | 0.35 |  | 0.37 | 0.40 | 0.37 | 0.22 |  | 0.37
$A_{1-8}$ | All features |  |  ‡0.37 |  ‡0.40 |  †0.42 |  ‡0.30 |  |  ‡0.33 |  †0.36 |  †0.30 |  †0.33 |  †0.29 |  †0.35 |  |  †0.37 |  †0.40 |  †0.37 |  †0.22 |  |  ‡0.37
$B$ | Baseline |  | 0.44 | 0.46 | 0.47 | 0.39 |  | 0.39 | 0.40 | 0.33 | 0.39 | 0.31 | 0.40 |  | 0.43 | 0.46 | 0.43 | 0.26 |  | 0.45
Table 1: Q1. Mean absolute error of each feature type, feature ablation, all
features, and the mean baseline for all 15 quality dimensions, averaged over
all 16 test sets. The best value in a column is bold. For all features,
significant improvements over the mean baseline are marked with $\dagger$
($p\\!<\\!.05$) and $\ddagger$ ($p\\!<\\!.01$).

### Quality Assessment (Q1)

To provide answers to question Q1, we let all SVMs learn to assess the mean
score of the three experts. Table 1 shows the MAE of each feature type
($A_{i}$), feature ablation ($A_{\backslash i}$), and all features ($A_{1-8}$)
in comparison to the baseline for each quality dimension. The SVM with all
features ($A_{1-8}$) outperforms the baseline in all cases. Only for four
dimensions, the gains are not significant, three of which being rhetorical:
clarity (Cla), credibility (Cre), and emotional appeal (Emo). This may be due
to their subjective nature, as reflected in limited inter-annotator agreement
[Wachsmuth et al., 2017b]. The highest MAE reduction is achieved for local
sufficiency (0.39 to 0.30), which has also been successfully assessed in
previous studies [Stab and Gurevych, 2017]. Other clear gains are achieved for
overall quality (0.45 to 0.37) and for cogency (0.44 to 0.37). No dimension is
really “solved” by the given features, but we conclude that intrinsic argument
quality assessment is effective to some extent.

Looking at the features, we see that content ($A_{1}$) and embedding ($A_{2}$)
perform rather badly, unlike in many NLP tasks. This is somewhat expected,
though, due to our leave-one-topic-out setting. While feature ablation leads
to the best results for some dimensions (e.g., Cre and Arr), two feature types
dominate the assessment: Subjectivity ($A_{8}$) alone minimizes the MSE for
five dimensions, once being the single best approach (for local relevance,
LRe). However, length ($A_{5}$) is even stronger, e.g., being best for
reasonableness and overall quality. Albeit quality may require some words,
this reveals length bias inherent to the corpus. Such bias was also found in
other argument quality corpora [Potash et al., 2017]. It questions the
validity of the annotated scores, even if $A_{5}$ is not needed for many
dimensions (see $A_{\backslash 5}$).

|  |  | Logical quality |  | Rhetorical quality |  | Dialectical quality |  |
---|---|---|---|---|---|---|---|---|---
Scores | Approach |  | Cog | LAc | LRe | LSu |  | Eff | Cla | Cre | App | Emo | Arr |  | Rea | GAc | GRe | GSu |  | OvQ
Expert #1 | All features |  |  †0.57 | 0.57 | 0.52 |  ‡0.47 |  |  †0.54 | 0.52 | 0.41 | 0.51 | 0.35 | 0.54 |  |  †0.52 | 0.56 | 0.54 |  ‡0.45 |  |  ‡0.52
| Baseline |  | 0.64 | 0.60 | 0.59 | 0.57 |  | 0.61 | 0.52 | 0.48 | 0.51 | 0.34 | 0.55 |  | 0.59 | 0.59 | 0.56 | 0.54 |  | 0.60
Expert #2 | All features |  | 0.49 | 0.59 | 0.63 | 0.34 |  | 0.48 | 0.53 | 0.37 | 0.67 | 0.31 | 0.49 |  | 0.59 | 0.56 | 0.58 | 0.26 |  |  ‡0.50
| Baseline |  | 0.57 | 0.60 | 0.67 | 0.44 |  | 0.56 | 0.53 | 0.40 | 0.64 | 0.28 | 0.53 |  | 0.62 | 0.56 | 0.63 | 0.30 |  | 0.62
Expert #3 | All features |  |  ‡0.50 |  †0.51 |  †0.53 |  ‡0.47 |  |  †0.38 | 0.54 |  †0.48 | 0.33 | 0.44 | 0.50 |  |  ‡0.46 | 0.47 | 0.53 |  †0.32 |  |  ‡0.47
| Baseline |  | 0.60 | 0.57 | 0.59 | 0.60 |  | 0.45 | 0.54 | 0.53 | 0.37 | 0.51 | 0.53 |  | 0.55 | 0.51 | 0.53 | 0.40 |  | 0.59
Mean score | All features |  |  ‡0.37 |  ‡0.40 |  †0.42 |  ‡0.30 |  |  ‡0.33 |  †0.36 |  †0.30 |  †0.33 | 0.29 |  †0.35 |  |  †0.37 |  †0.40 |  †0.37 |  †0.22 |  |  ‡ 0.37
| Baseline |  | 0.44 | 0.46 | 0.47 | 0.39 |  | 0.39 | 0.40 | 0.33 | 0.39 | 0.31 | 0.40 |  | 0.43 | 0.46 | 0.43 | 0.26 |  | 0.45
Table 2: Q2. Mean absolute error of the SVM with all features and the mean
baseline for all quality dimensions on all test sets, separated for training
on the ground-truth scores of expert #1, #2, or #3, or the mean scores (as in
Table 1) respectively. The value with the highest significance in each column
is bold.

### Subjectiveness (Q2)

For Q2, we learn to assess the score of each single expert and compare our
features to the baseline in Table 2. Mean scores lead to the lowest MAE, due
to their natural tendency towards middle scores. We find clear differences
between the experts, reflecting how subjective the assessment is: Hardly any
significant learning success is observed on the scores of expert #2, whereas
particularly #3 seems well-assessable. While this may mean that some experts
are either more reliable or more influenced by surface text features, it
raises the question whether assessing the mean score is the best choice.

### Human vs. Machine (Q3)

For Q3, finally, we evaluate how much the experts diverge from the majority
score as opposed to the all-features SVM. Since the experts could only give
integer scores, for fairness we rounded the scores of the SVM before MAE
computation. Still, Table 3 reveals that expert #2 significantly beats our
features only on three dimensions (Eff, GAcc, and OvQ) and is even worse on
five dimensions (App, Emo, Arr, GRe, and GSu). So, our features can compete
with some humans. Expert #3, in contrast, clearly outperforms the SVM with a
very low MAE for most dimensions. Together with the results on Q2, it seems
that some expert scores are more consistent.

| “Approach” |  | Cog | LAc | LRe | LSu |  | Eff | Cla | Cre | App | Emo | Arr |  | Rea | GAc | GRe | GSu |  | OvQ
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Humans | Expert #1 |  | 0.32 | 0.32 |  ‡0.23 | 0.29 |  | 0.32 | 0.34 |  ‡0.19 | 0.30 | 0.18 | 0.27 |  |  †0.22 |  ‡0.27 | 0.40 | 0.29 |  | 0.27
| Expert #2 |  | 0.28 | 0.38 | 0.33 | 0.31 |  |  ‡0.24 | 0.31 | 0.27 | 0.46 | 0.25 | 0.35 |  | 0.30 |  †0.29 | 0.41 | 0.24 |  |  †0.26
| Expert #3 |  |  ‡0.17 |  ‡0.22 |  ‡0.23 |  ‡0.15 |  |  ‡0.16 |  †0.26 |  †0.27 |  †0.26 |  †0.27 |  †0.28 |  |  ‡0.16 |  ‡0.26 |  ‡0.22 |  ‡0.08 |  |  ‡0.14
SVM | All features |  | 0.36 | 0.38 | 0.42 | 0.34 |  | 0.36 | 0.33 | 0.29 | 0.28 | 0.22 | 0.33 |  | 0.34 | 0.40 | 0.39 | 0.21 |  | 0.36
Table 3: Q3. Mean absolute error of each expert and the SVM with all features,
for all quality dimensions on all test sets, based on majority scores. The
best value for each dimension is bold. Gray expert values are worse than the
features; significant gains over the features are marked with $\dagger$
($p\\!<\\!.05$) and $\ddagger$ ($p\\!<\\!.01$).

##  6 Conclusion

In this focused study, we have systematically benchmarked how well argument
quality can be assessed computationally on an existing corpus annotated for
several quality dimensions. Modeling subjectiveness in terms of sentiment,
pronoun usage, and similar seems useful on the debate portal arguments
included in the corpus, at least for logical and dialectical dimensions.
However, the limited corpus size naturally makes it hard to find more complex
features that robustly predict argument quality. In addition, the correlation
of quality and length in the corpus limits the generalizability of our
findings. This calls for more large-scale and balanced argument quality
corpora. First attempts in this direction have been made [Toledo et al.,
2019], but the comprehensive view on quality of the corpus used here has no
equal so far.

## Acknowledgments

We thank Eyke Hüllermeier for suggestions on the experimental setup, Gabriela
Molina León for feedback on early drafts, and the anonymous reviewers for
their helpful comments.

## References

  * [Aggarwal and Zhai, 2012]  Charu C. Aggarwal and ChengXiang Zhai, 2012.  A Survey of Text Classification Algorithms, pages 163–222.  Springer US, Boston, MA.
  * [Chang and Lin, 2011]  Chih-Chung Chang and Chih-Jen Lin.  2011\.  LIBSVM: A library for support vector machines.  ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [El Baff et al., 2018]  Roxanne El Baff, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein.  2018\.  Challenge or empower: Revisiting argumentation quality in a news editorial corpus.  In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 454–464, Brussels, Belgium, October. Association for Computational Linguistics.
  * [El Baff et al., 2020]  Roxanne El Baff, Henning Wachsmuth, Khalid Al Khatib, and Benno Stein.  2020\.  Analyzing the persuasive effect of style in news editorial argumentation.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3154–3160, Online, July. Association for Computational Linguistics.
  * [Gleize et al., 2019]  Martin Gleize, Eyal Shnarch, Leshem Choshen, Lena Dankin, Guy Moshkowich, Ranit Aharonov, and Noam Slonim.  2019\.  Are you convinced? Choosing the more convincing evidence with a Siamese network.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 967–976, Florence, Italy, July. Association for Computational Linguistics.
  * [Gretz et al., 2020]  Shai Gretz, Roni Friedman, Edo Cohen-Karlik, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim.  2020\.  A large-scale dataset for argument quality ranking: Construction and analysis.  In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 7805–7813. AAAI.
  * [Habernal and Gurevych, 2016a]  Ivan Habernal and Iryna Gurevych.  2016a.  What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in web argumentation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214–1223. Association for Computational Linguistics.
  * [Habernal and Gurevych, 2016b]  Ivan Habernal and Iryna Gurevych.  2016b.  Which argument is more convincing? Analyzing and predicting convincingness of web arguments using bidirectional lstm.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599. Association for Computational Linguistics.
  * [Johnson and Blair, 2006]  Ralph H. Johnson and J. Anthony Blair.  2006\.  Logical Self-defense.  Intern. Debate Education Association.
  * [Ke and Ng, 2019]  Zixuan Ke and Vincent Ng.  2019\.  Automated essay scoring: A survey of the state of the art.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pages 6300–6308. IJCAI.
  * [Lukin et al., 2017]  Stephanie Lukin, Pranav Anand, Marilyn Walker, and Steve Whittaker.  2017\.  Argument Strength is in the Eye of the Beholder: Audience Effects in Persuasion.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 742–753. Association for Computational Linguistics.
  * [Mikolov et al., 2018]  Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin.  2018\.  Advances in pre-training distributed word representations.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA).
  * [Persing and Ng, 2015]  Isaac Persing and Vincent Ng.  2015\.  Modeling argument strength in student essays.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 543–552. Association for Computational Linguistics.
  * [Persing et al., 2010]  Isaac Persing, Alan Davis, and Vincent Ng.  2010\.  Modeling organization in student essays.  In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 229–239. Association for Computational Linguistics.
  * [Potash et al., 2017]  Peter Potash, Robin Bhattacharya, and Anna Rumshisky.  2017\.  Length, interchangeability, and external knowledge: Observations from predicting argument convincingness.  In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 342–351, Taipei, Taiwan, November. Asian Federation of Natural Language Processing.
  * [Potthast et al., 2019]  Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenkötter, Nico Weidmann, Henning Wachsmuth, Benno Stein, and Matthias Hagen.  2019\.  Argument search: Assessing argument relevance.  In 42nd International ACM Conference on Research and Development in Information Retrieval (SIGIR 2019), pages 1117–1120. ACM, July.
  * [Rittman et al., 2004]  Robert Rittman, Nina Wacholder, Paul Kantor, Kwong Bor Ng, Tomek StrzaIkowski, and Ying Sun.  2004\.  Adjectives as indicators of subjectivity in documents.  In Proceedings of the 67th ASIS&T Annual Meeting, pages 349–359. American Society for Information Science.
  * [Stab and Gurevych, 2017]  Christian Stab and Iryna Gurevych.  2017\.  Recognizing insufficiently supported arguments in argumentative essays.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 980–990. Association for Computational Linguistics.
  * [Stamatatos, 2009]  Efstathios Stamatatos.  2009\.  A survey of modern authorship attribution methods.  Journal of the American Society for Information Science and Technology, 60(3):538–556.
  * [Stede and Schneider, 2018]  Manfred Stede and Jodi Schneider.  2018\.  Argumentation Mining.  Number 40 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool.
  * [Toledo et al., 2019]  Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, and Noam Slonim.  2019\.  Automatic argument quality assessment - New datasets and methods.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5625–5635. Association for Computational Linguistics.
  * [van Eemeren and Grootendorst, 2004]  Frans H. van Eemeren and Rob Grootendorst.  2004\.  A Systematic Theory of Argumentation: The Pragma-Dialectical Approach.  Cambridge University Press, Cambridge, UK.
  * [Wachsmuth et al., 2016]  Henning Wachsmuth, Khalid Al Khatib, and Benno Stein.  2016\.  Using argument mining to assess the argumentation quality of essays.  In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1680–1691. The COLING 2016 Organizing Committee.
  * [Wachsmuth et al., 2017a]  Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein.  2017a.  Argumentation quality assessment: Theory vs. practice.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 250–255. Association for Computational Linguistics.
  * [Wachsmuth et al., 2017b]  Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Alberdingk Tim Thijm, Graeme Hirst, and Benno Stein.  2017b.  Computational argumentation quality assessment in natural language.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 176–187. Association for Computational Linguistics.
  * [Wachsmuth et al., 2017c]  Henning Wachsmuth, Benno Stein, and Yamen Ajjour.  2017c.  “PageRank” for argument relevance.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1117–1127. Association for Computational Linguistics.
  * [Walton, 2006]  Douglas Walton.  2006\.  Fundamentals of Critical Argumentation.  Cambridge University Press.
  * [Wei et al., 2016]  Zhongyu Wei, Yang Liu, and Yi Li.  2016\.  Is this post persuasive? Ranking argumentative comments in online forum.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 195–200. Association for Computational Linguistics.
  * [Yang et al., 2019]  Wonsuk Yang, Seungwon Yoon, Ada Carpenter, and Jong Park.  2019\.  Nonsense!: Quality control via two-step reason selection for annotating local acceptability and related attributes in news editorials.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2954–2963. Association for Computational Linguistics.


# On the Perception of Plagiarism in Academia:
Context and Intent

Aaron Gregory1 and Joshua Leeman2

###### Abstract

Plagiarism is the representation of another author’s language, thoughts,
ideas, or expressions as one’s own original work. In educational contexts,
there are differing definitions of plagiarism depending on the institution.
Prominent scholars of plagiarism include Rebecca Moore Howard, Susan Blum,
Tracey Bretag, and Sarah Elaine Eaton, among others. Plagiarism is considered
a violation of academic integrity and a breach of journalistic ethics. It is
subject to sanctions such as penalties, suspension, expulsion from school or
work, substantial fines and even incarceration. Recently, cases of “extreme
plagiarism” have been identified in academia. The modern concept of plagiarism
as immoral and originality as an ideal emerged in Europe in the 18th century,
particularly with the Romantic movement. Generally, plagiarism is not in
itself a crime, but like counterfeiting fraud can be punished in a court for
prejudices caused by copyright infringement, violation of moral rights, or
torts. In academia and industry, it is a serious ethical offense. Plagiarism
and copyright infringement overlap to a considerable extent, but they are not
equivalent concepts, and many types of plagiarism do not constitute copyright
infringement, which is defined by copyright law and may be adjudicated by
courts. Plagiarism might not be the same in all countries. Some countries,
such as India and Poland, consider plagiarism to be a crime, and there have
been cases of people being imprisoned for plagiarizing. In other instances
plagiarism might be the complete opposite of “academic dishonesty,” in fact
some countries find the act of plagiarizing a professional’s work flattering.
Students who move to the United States and other Western countries from
countries where plagiarism is not frowned upon often find the transition
difficult.

Keywords -

Plagiarism; Ethics; Context

##  1 Etymology

In the 1st century, the use of the Latin word “plagiarius” (literally
“kidnapper”) to denote stealing someone else’s work was pioneered by the Roman
poet Martial, who complained that another poet had “kidnapped his verses”.
Plagiary, a derivative of plagiarus, was introduced into English in 1601 by
dramatist Ben Jonson during the Jacobean Era to describe someone guilty of
literary theft.

The derived form plagiarism was introduced into English around 1620. The Latin
plagiārius, “kidnapper”, and plagium, “kidnapping”, have the root plaga
(“snare”, “net”), based on the Indo-European root *-plak, “to weave” (seen for
instance in Greek plekein, Bulgarian “ плета” pleta, and Latin plectere, all
meaning “to weave”).

##  2 Legal aspects

Although plagiarism in some contexts is considered theft or stealing, the
concept does not exist in a legal sense, although the use of someone else’s
work in order to gain academic credit may meet some legal definitions of
fraud. “Plagiarism” specifically is not mentioned in any current statute,
either criminal or civil. Some cases may be treated as unfair competition or a
violation of the doctrine of moral rights. In short, people are asked to use
the guideline, “if you did not write it yourself, you must give credit”.

Plagiarism is not the same as copyright infringement. While both terms may
apply to a particular act, they are different concepts, and false claims of
authorship generally constitute plagiarism regardless of whether the material
is protected by copyright. Copyright infringement is a violation of the rights
of a copyright holder, when material whose use is restricted by copyright is
used without consent. Plagiarism, in contrast, is concerned with the unearned
increment to the plagiarizing author’s reputation, or the obtaining of
academic credit, that is achieved through false claims of authorship. Thus,
plagiarism is considered a moral offense against the plagiarist’s audience
(for example, a reader, listener, or teacher).

Plagiarism is also considered a moral offense against anyone who has provided
the plagiarist with a benefit in exchange for what is specifically supposed to
be original content (for example, the plagiarist’s publisher, employer, or
teacher). In such cases, acts of plagiarism may sometimes also form part of a
claim for breach of the plagiarist’s contract, or, if done knowingly, for a
civil wrong.

##  3 In academia and journalism

Within academia, plagiarism by students, professors, or researchers is
considered academic dishonesty or academic fraud, and offenders are subject to
academic censure, up to and including expulsion. Some institutions use
plagiarism detection software to uncover potential plagiarism and to deter
students from plagiarizing. However, plagiarism detection software does not
always yield accurate results and there are loopholes in these systems. Some
universities address the issue of academic integrity by providing students
with thorough orientations, required writing courses, and clearly articulated
honor codes. Indeed, there is a virtually uniform understanding among college
students that plagiarism is wrong. Nevertheless, each year students are
brought before their institutions’ disciplinary boards on charges that they
have misused sources in their schoolwork. However, the practice of
plagiarizing by use of sufficient word substitutions to elude detection
software, known as rogeting, has rapidly evolved as students and unethical
academics seek to stay ahead of detection software.

An extreme form of plagiarism, known as “contract cheating”, involves students
paying someone else, such as an essay mill, to do their work for them.

In journalism, plagiarism is considered a breach of journalistic ethics, and
reporters caught plagiarizing typically face disciplinary measures ranging
from suspension to termination of employment. Some individuals caught
plagiarizing in academic or journalistic contexts claim that they plagiarized
unintentionally, by failing to include quotations or give the appropriate
citation. While plagiarism in scholarship and journalism has a centuries-old
history, the development of the Internet, where articles appear as electronic
text, has made the physical act of copying the work of others much easier.

Predicated upon an expected level of learning and comprehension having been
achieved, all associated academic accreditation becomes seriously undermined
if plagiarism is allowed to become the norm within academic submissions.

For professors and researchers, plagiarism is punished by sanctions ranging
from suspension to termination, along with the loss of credibility and
perceived integrity. Charges of plagiarism against students and professors are
typically heard by internal disciplinary committees, by which students and
professors have agreed to be bound. Plagiarism is a common reason for academic
research papers to be retracted.

###  3.1 Academia

No universally adopted definition of academic plagiarism exists. However, this
section provides several definitions to exemplify the most common
characteristics of academic plagiarism. It has been called, “The use of ideas,
concepts, words, or structures without appropriately acknowledging the source
to benefit in a setting where originality is expected.”

This is an abridged version of Teddi Fishman’s definition of plagiarism, which
proposed five elements characteristic of plagiarism. According to Fishman,
plagiarism occurs when someone:

  * •

Uses words, ideas, or work products

  * •

Attributable to another identifiable person or source

  * •

Without attributing the work to the source from which it was obtained

  * •

In a situation in which there is a legitimate expectation of original
authorship

  * •

In order to obtain some benefit, credit, or gain which need not be monetary

Furthermore, plagiarism is defined differently among institutions of higher
learning and universities:

  * •

Stanford defines plagiarism as the “use, without giving reasonable and
appropriate credit to or acknowledging the author or source, of another
person’s original work, whether such work is made up of code, formulas, ideas,
language, research, strategies, writing or other form”.

  * •

Yale views plagiarism as the “… use of another’s work, words, or ideas without
attribution”, which includes “… using a source’s language without quoting,
using information from a source without attribution, and paraphrasing a source
in a form that stays too close to the original”.

  * •

Princeton describes plagiarism as the “deliberate" use of “someone else’s
language, ideas, or other original (not common-knowledge) material without
acknowledging its source”.

  * •

Oxford College of Emory University characterizes plagiarism as the use of “a
writer’s ideas or phraseology without giving due credit”.

  * •

Brown defines plagiarism as “… appropriating another person’s ideas or words
(spoken or written) without attributing those word or ideas to their true
source”.

  * •

The U.S. Naval Academy defines plagiarism as “the use of the words,
information, insights, or ideas of another without crediting that person
through proper citation”.

####  3.1.1 Forms of academic plagiarism

Different classifications of academic plagiarism forms have been proposed.
Many classifications follow a behavioral approach, i.e., they seek to classify
the actions undertaken by plagiarists.

For example, a 2015 survey of teachers and professors by Turnitin, identified
10 main forms of plagiarism that students commit:

  * •

Submitting someone’s work as their own.

  * •

Taking passages from their own previous work without adding citations (self-
plagiarism).

  * •

Re-writing someone’s work without properly citing sources.

  * •

Using quotations but not citing the source.

  * •

Interweaving various sources together in the work without citing.

  * •

Citing some, but not all, passages that should be cited.

  * •

Melding together cited and uncited sections of the piece.

  * •

Providing proper citations, but failing to change the structure and wording of
the borrowed ideas enough (close paraphrasing).

  * •

Inaccurately citing a source.

  * •

Relying too heavily on other people’s work, failing to bring original thought
into the text.

A 2019 systematic literature review on academic plagiarism detection
deductively derived a technically oriented typology of academic plagiarism
from the linguistic model of language consisting of lexis, syntax, and
semantics extended by a fourth layer to capture the plagiarism of ideas and
structures. The typology categorizes plagiarism forms according to the layer
of the model they affect:

  * •

Characters-preserving plagiarism

    * –

Verbatim copying without proper citation

  * •

Syntax-preserving plagiarism

    * –

Synonym substitution

    * –

Technical disguise (e.g. using identically looking glyphs from another
alphabet)

  * •

Semantics-preserving plagiarism

    * –

Translation

    * –

Paraphrase

  * •

Idea-preserving plagiarism

    * –

Appropriation of ideas or concepts

    * –

Reusing text structure

  * •

Ghostwriting

    * –

Collusion (typically among students)

    * –

Contract cheating

####  3.1.2 Sanctions for student plagiarism

In the academic world, plagiarism by students is usually considered a very
serious offense that can result in punishments such as a failing grade on the
particular assignment, the entire course, or even being expelled from the
institution. The seriousness with which academic institutions address student
plagiarism may be tempered by a recognition that students may not fully
understand what plagiarism is. A 2015 study showed that students who were new
to university study did not have a good understanding of even the basic
requirements of how to attribute sources in written academic work, yet
students were very confident that they understood what referencing and
plagiarism are. The same students also had a lenient view of how plagiarism
should be penalised.

For cases of repeated plagiarism, or for cases in which a student commits
severe plagiarism (e.g., purchasing an assignment), suspension or expulsion
may occur. There has been historic concern about inconsistencies in penalties
administered for university student plagiarism, and a plagiarism tariff was
devised in 2008 for UK higher education institutions in an attempt to
encourage some standardization of approaches.

However, to impose sanctions, plagiarism needs to be detected. Strategies
faculty members use to detect plagiarism include carefully reading students
work and making note of inconsistencies in student writing, citation errors
and providing plagiarism prevention education to students. It has been found
that a significant share of (university) teachers do not use detection methods
such as using text-matching software. A few more try to detect plagiarism by
reading term-papers specifically for plagiarism, while the latter method might
be not very effective in detecting plagiarism – especially when plagiarism
from unfamiliar sources needs to be detected. There are checklists of tactics
to prevent student plagiarism.

####  3.1.3 Plagiarism education

Given the serious consequences that plagiarism has for students, there has
been a call for a greater emphasis on learning in order to help students avoid
committing plagiarism. This is especially important when students move to a
new institution that may have a different view of the concept when compared
with the view previously developed by the student. Indeed, given the
seriousness of plagiarism accusations for a student’s future, the pedagogy of
plagiarism education may need to be considered ahead of the pedagogy of the
discipline being studied. The need for plagiarism education extends to
academic staff, who may not completely understand what is expected of their
students or the consequences of misconduct. Actions to reduce plagiarism
include coordinating teaching activities to decrease student load; reducing
memorization, increasing individual practical activities; and promoting
positive reinforcement over punishment.

####  3.1.4 Factors influencing students’ decisions to plagiarize

Several studies investigated factors that influence the decision to
plagiarize. For example, a panel study with students from German universities
found that academic procrastination predicts the frequency plagiarism
conducted within six months followed the measurement of academic
procrastination. It has been argued that by plagiarizing, students cope with
the negative consequences that result from academic procrastination such as
poor grades. Another study found that plagiarism is more frequent if students
perceive plagiarism as beneficial and if they have the opportunity to
plagiarize. When students had expected higher sanctions and when they had
internalized social norms that define plagiarism as very objectionable,
plagiarism was less likely to occur. Another study found that students
resorted to plagiarism in order to cope with heavy workloads imposed by
teachers. On the other hand, in that study, some teachers also thought that
plagiarism is a consequence of their own failure to propose creative tasks and
activities.

###  3.2 Journalism

Since journalism relies on the public trust, a reporter’s failure to honestly
acknowledge their sources undercuts a newspaper or television news show’s
integrity and undermines its credibility. Journalists accused of plagiarism
are often suspended from their reporting tasks while the charges are being
investigated by the news organization.

###  3.3 Self-plagiarism

The reuse of significant, identical, or nearly identical portions of one’s own
work without acknowledging that one is doing so or citing the original work is
sometimes described as “self-plagiarism”; the term “recycling fraud” has also
been used to describe this practice. Articles of this nature are often
referred to as duplicate or multiple publication. In addition there can be a
copyright issue if copyright of the prior work has been transferred to another
entity. Self-plagiarism is considered a serious ethical issue in settings
where someone asserts that a publication consists of new material, such as in
publishing or factual documentation. It does not apply to public-interest
texts, such as social, professional, and cultural opinions usually published
in newspapers and magazines.

In academic fields, self-plagiarism occurs when an author reuses portions of
their own published and copyrighted work in subsequent publications, but
without attributing the previous publication. Identifying self-plagiarism is
often difficult because limited reuse of material is accepted both legally (as
fair use) and ethically. Many people mostly, but not limited to critics of
copyright and “intellectual property” do not believe it is possible to
plagiarize oneself. Critics of the concepts of plagiarism and copyright may
use the idea of self-plagiarism as a reductio ad absurdum argument.

####  3.3.1 Contested definition

Miguel Roig has written at length about the topic of self-plagiarism and his
definition of self-plagiarism as using previously disseminated work is widely
accepted among scholars of the topic. However, the term “self-plagiarism” has
been challenged as being self-contradictory, an oxymoron, and on other
grounds.

For example, Stephanie J. Bird argues that self-plagiarism is a misnomer,
since by definition plagiarism concerns the use of others’ material. Bird
identifies the ethical issues of “self-plagiarism” as those of “dual or
redundant publication”. She also notes that in an educational context, “self-
plagiarism” refers to the case of a student who resubmits “the same essay for
credit in two different courses.” As David B. Resnik clarifies, “Self-
plagiarism involves dishonesty but not intellectual theft.”

According to Patrick M. Scanlon, “self-plagiarism” is a term with some
specialized currency. Most prominently, it is used in discussions of research
and publishing integrity in biomedicine, where heavy publish-or-perish demands
have led to a rash of duplicate and “salami-slicing” publication, the
reporting of a single study’s results in “least publishable units” within
multiple articles (Blancett, Flanagin, & Young, 1995; Jefferson, 1998;
Kassirer & Angell, 1995; Lowe, 2003; McCarthy, 1993; Schein & Paladugu, 2001;
Wheeler, 1989). Roig (2002) offers a useful classification system including
four types of self-plagiarism: duplicate publication of an article in more
than one journal; partitioning of one study into multiple publications, often
called salami-slicing; text recycling; and copyright infringement.

####  3.3.2 Codes of ethics

Some academic journals have codes of ethics that specifically refer to self-
plagiarism. For example, the Journal of International Business Studies. Some
professional organizations like the Association for Computing Machinery (ACM)
have created policies that deal specifically with self-plagiarism. Other
organizations do not make specific reference to self-plagiarism such as the
American Political Science Association (APSA). The organization published a
code of ethics that describes plagiarism as “…deliberate appropriation of the
works of others represented as one’s own.” It does not make any reference to
self-plagiarism. It does say that when a thesis or dissertation is published
“in whole or in part”, the author is “not ordinarily under an ethical
obligation to acknowledge its origins.” The American Society for Public
Administration (ASPA) also published a code of ethics that says its members
are committed to: “Ensure that others receive credit for their work and
contributions,” but it makes no reference to self-plagiarism.

####  3.3.3 Factors that justify reuse

Pamela Samuelson, in 1994, identified several factors she says excuse reuse of
one’s previously published work, that make it not self-plagiarism. She relates
each of these factors specifically to the ethical issue of self-plagiarism, as
distinct from the legal issue of fair use of copyright, which she deals with
separately. Among other factors that may excuse reuse of previously published
material Samuelson lists the following:

  * •

The previous work must be restated to lay the groundwork for a new
contribution in the second work.

  * •

Portions of the previous work must be repeated to deal with new evidence or
arguments.

  * •

The audience for each work is so different that publishing the same work in
different places is necessary to get the message out.

  * •

The author thinks they said it so well the first time that it makes no sense
to say it differently a second time.

Samuelson states she has relied on the “different audience” rationale when
attempting to bridge interdisciplinary communities. She refers to writing for
different legal and technical communities, saying: “there are often paragraphs
or sequences of paragraphs that can be bodily lifted from one article to the
other. And, in truth, I lift them.” She refers to her own practice of
converting “a technical article into a law review article with relatively few
changes—adding footnotes and one substantive section” for a different
audience.

Samuelson describes misrepresentation as the basis of self-plagiarism. She
also states “Although it seems not to have been raised in any of the self-
plagiarism cases, copyrights law’s fair use defense would likely provide a
shield against many potential publisher claims of copyright infringement
against authors who reused portions of their previous works.”

###  3.4 Organizational publications

Plagiarism is presumably not an issue when organizations issue collective
unsigned works since they do not assign credit for originality to particular
people. For example, the American Historical Association’s “Statement on
Standards of Professional Conduct” (2005) regarding textbooks and reference
books states that, since textbooks and encyclopedias are summaries of other
scholars’ work, they are not bound by the same exacting standards of
attribution as original research and may be allowed a greater “extent of
dependence” on other works. However, even such a book does not make use of
words, phrases, or paragraphs from another text or follow too closely the
other text’s arrangement and organization, and the authors of such texts are
also expected to “acknowledge the sources of recent or distinctive findings
and interpretations, those not yet a part of the common understanding of the
profession.”

##  4 In the arts

###  4.1 The history of the arts

Through all of the history of literature and of the arts in general, works of
art are for a large part repetitions of the tradition; to the entire history
of artistic creativity belong plagiarism, literary theft, appropriation,
incorporation, retelling, rewriting, recapitulation, revision, reprise,
thematic variation, ironic retake, parody, imitation, stylistic theft,
pastiches, collages, and deliberate assemblages. There is no rigorous and
precise distinction between practices like imitation, stylistic plagiarism,
copy, replica and forgery. These appropriation procedures are the main axis of
a literate culture, in which the tradition of the canonic past is being
constantly rewritten.

Ruth Graham quotes T. S. Eliot–“Immature poets imitate; mature poets steal.
Bad poets deface what they take.”–she notes that despite the “taboo” of
plagiarism, the ill-will and embarrassment it causes in the modern context,
readers seem to often forgive the past excesses of historic literary
offenders.

###  4.2 Praisings of artistic plagiarism

A passage of Laurence Sterne’s 1767 Tristram Shandy condemns plagiarism by
resorting to plagiarism. Oliver Goldsmith commented:

> Sterne’s Writings, in which it is clearly shewn, that he, whose manner and
> style were so long thought original, was, in fact, the most unhesitating
> plagiarist who ever cribbed from his predecessors in order to garnish his
> own pages. It must be owned, at the same time, that Sterne selects the
> materials of his mosaic work with so much art, places them so well, and
> polishes them so highly, that in most cases we are disposed to pardon the
> want of originality, in consideration of the exquisite talent with which the
> borrowed materials are wrought up into the new form.

##  5 In other contexts

###  5.1 On the Internet

Free online tools are becoming available to help identify plagiarism, and
there are a range of approaches that attempt to limit online copying, such as
disabling right clicking and placing warning banners regarding copyrights on
web pages. Instances of plagiarism that involve copyright violation may be
addressed by the rightful content owners sending a DMCA removal notice to the
offending site-owner, or to the ISP that is hosting the offending site. The
term “content scraping” has arisen to describe the copying and pasting of
information from websites and blogs.

###  5.2 Reverse plagiarism

Reverse plagiarism, or attribution without copying, refers to falsely giving
authorship credit over a work to a person who did not author it, or falsely
claiming a source supports an assertion that the source does not make. While
both the term and activity are relatively rare, incidents of reverse
plagiarism do occur typically in similar contexts as traditional plagiarism.

##  6 Acknowledgements

We would like to thank Mohammed for his invaluable contributions to this work.
He deserves credit for everything written herein.

## References

  * [1] Aaron Gregory and Joshua Leeman.  On the Perception of Plagiarism in Academia: Context and Intent. arXiv preprint, 2021.