# How Human is Human Evaluation?
Improving the Gold Standard for NLG with Utility Theory

Kawin Ethayarajh
Stanford University
<EMAIL_ADDRESS>
&Dan Jurafsky
Stanford University
<EMAIL_ADDRESS>Kawin Ethayarajh
Stanford University
<EMAIL_ADDRESS>
&Dan Jurafsky
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Human ratings are treated as the gold standard in NLG evaluation. The standard
protocol is to collect ratings of generated text, average across annotators,
and then rank NLG systems by their average scores. However, little
consideration has been given as to whether this approach faithfully captures
human preferences. In this work, we analyze this standard protocol through the
lens of utility theory in economics. We first identify the implicit
assumptions it makes about annotators and find that these assumptions are
often violated in practice, in which case annotator ratings become an
unfaithful reflection of their preferences. The most egregious violations come
from using Likert scales, which provably reverse the direction of the true
preference in certain cases. We suggest improvements to the standard protocol
to make it more theoretically sound, but even in its improved form, it cannot
be used to evaluate open-ended tasks like story generation. For the latter, we
propose a new evaluation protocol called system-level probabilistic assessment
(SPA). In our experiments, we find that according to SPA, annotators prefer
larger GPT-3 variants to smaller ones – as expected – with all comparisons
being statistically significant. In contrast, the standard protocol only
yields significant results half the time.

##  1 Introduction

Human ratings are treated as the gold standard in NLG evaluation. For example,
say one wants to claim that that their NLG model $X$ is better than the
current state-of-the-art $Y$ and $Z$ for story generation. The standard
protocol is outcome-level absolute assessment (OAA): hire crowdworkers as
annotators, collect individual ratings of a sample of stories generated by
each model, and then claim that $X$ is the best because its average rating is
the highest (Celikyilmaz et al., 2020). There is inconsistency in how this is
implemented in the literature: terms such as ‘text quality’ are often left
undefined when instructing annotators (Howcroft et al., 2020) and different
papers use different rating scales (Amidei et al., 2019). However, such
criticism has been restricted to the implementations – little to no
consideration has been given as to whether OAA can faithfully capture human
preferences to begin with.

We start by analyzing the standard evaluation protocol through the lens of
utility theory from economics (§2). We find that OAA can only capture an
annotator’s preferences under certain assumptions, which are unstated in the
NLG literature and often violated in practice. In such cases, annotator
ratings become an unfaithful reflection of their preferences (§3). For
example, by framing ratings as utility estimates, we extend a result from.
Boutilier (2003) to prove that using the same scale is insufficient for
aggregating ratings across annotators – they must agree on the maximum- and
minimum-utility outcomes as well. This precludes annotator ratings from being
averaged unless they are given both maximally “correct” and “incorrect”
references, which are available for some NLG tasks (e.g., machine translation)
but not for open-ended ones (e.g, story generation), since the space of high-
quality outputs is too diverse. We provide concrete suggestions on how to
improve the standard protocol to a point where it can faithfully capture human
preferences in some NLG tasks and settings (§4); however, for open-ended
generation, an entirely new evaluation protocol is needed.

Though rarely used nowadays, a historic alternative to OAA was outcome-level
relative assessment (ORA): create random pairs containing an output from $X$
and $Y$, ask annotators to pick the one they prefer in each, infer a score for
$X$ and $Y$ that explains the results – based on a comparison model such as
Bradley-Terry (Bradley and Terry, 1952) – and argue that $X$ is better because
its estimated score is higher (Sakaguchi et al., 2014). However, this also
makes untenable assumptions about annotators; for example, even if $X$’s
outputs are preferred to $Y$’s over 50% of the time, $X$ may be less preferred
to $Y$ if it has a tendency to fail catastrophically. We observe that the main
limitation of both OAA and ORA is their reliance on outcome-level judgments.

To this end, we propose system-level probabilistic assessment (SPA), which can
be used for both open- and close-ended NLG tasks (§5). SPA’s key insight is
that while an annotator cannot categorically determine whether they prefer
system $X$ or $Y$ – because the output space is too large for them to observe
– they can estimate the probability with which they prefer $X$ or $Y$ based on
some fixed level of exposure to both. SPA obviates assumptions about annotator
preferences by delegating the responsibility of aggregating preferences over
individual outputs into a preference over the underlying systems to the
annotator themself, acknowledging that there is no canonical way to do so.
Because we are working with probabilities, aggregating across annotators is
also straightforward.

We then ask annotators to use both the standard protocol (with a 5-point
Likert scale) and SPA to express their preferences about the different GPT-3
variants w.r.t. their story-writing ability. Given that larger variants
generate more coherent, grammatical, and creative text, annotators (in
aggregate111At the population-level, since individual annotators may have
aberrant preferences.) should prefer each GPT-3 variant to the next smallest
one, which gives us 3 ground-truth preferences (Brown et al., 2020). Past work
also suggests that annotators can distinguish human-written text from ada (the
smallest variant) but not from davinci (the largest) (Clark et al., 2021),
which gives us two additional ground-truth preferences, for a total of 5. SPA
is able to recover all 5 out of 5 expected preferences, with statistically
significant results. However, the standard protocol only recovers 2/5 and also
yields the surprising – and likely incorrect – result that human-written text
is slightly (but significantly) less preferred to davinci. This suggests that
the theoretical limitations of the standard protocol have practical
consequences, and the flexibility of SPA makes it the better option in most
intrinsic evaluation settings.

##  2 Reframing Human Evaluation

To understand what causes human preferences to be misrepresented, we will
analyze NLG evaluation through the lens of economic theory on preference
modeling. In doing so, we find that comparing NLG systems is an instance of a
common problem in utility theory. To begin, let $X,Y$ denote the NLG systems
to be compared and annotator $a_{i}$ be the agent making the comparisons. As
we are drawing from the economics literature, we will primarily use economic
terms such as lottery and utility in our framing, which we will define as we
go along.

###  2.1 NLG Systems as Lotteries

######  Definition 1 (Lottery).

A lottery is a probability distribution over a space of finite outcomes
(Boutilier, 2003). Given a (possibly empty) prompt or input, an NLG system
induces a lottery over all possible output text.

Given that there is also a discrete distribution over the prompts/inputs used,
the lottery that the NLG system induces over the output text is itself the
outcome of a lottery over the prompts/inputs. This means that $X$ and $Y$ are
compound lotteries: a lottery of a lottery, which can be reduced to a simple
lottery over the output text by marginalization. Thus for a known prior over
the prompts/inputs, we can think of any NLG system as a simple lottery over
all possible output text.

###  2.2 Choices as Preference Relations

######  Definition 2 (Preference Relations).

The relation $X\succ_{i}Y$ means that the agent $a_{i}$ strictly prefers $X$
to $Y$; the relation $X\prec_{i}Y$ means that the agent strictly prefers $Y$
to $X$; the relation $X\sim_{i}Y$ means that the agent is indifferent to the
two. Relations without the subscript $i$ denote the aggregate preference
across all agents.

For example, an NLG system that places higher probabilities on coherent text
across all prompts will usually be preferred to one centered on incoherent
text, although different agents can have different preferences over the same
two systems. This means that determining whether an annotator prefers one NLG
system to another is an instance of a common economics problem: determining
which of two lotteries the agent prefers. For most such problems in the real
world, we could ask the agent to directly compare the two lotteries (e.g., we
could ask an investor what split of stocks to bonds they would invest in)
(Mankiw, 2020). However, because in NLG the space of output text is so large,
we cannot ask an annotator to categorically determine which of two lotteries
they prefer. What is feasible is asking an annotator to compare two individual
output texts, but there is no assumption-free means of aggregating preferences
over individual outcomes into preferences over the lotteries from which those
outcomes are drawn (§2.4).

###  2.3 Text Quality as Utility

######  Definition 3 (Utility).

Abstractly, the utility of a good denotes the benefit that an agent receives
from it. The utility function $u_{i}:S\to\mathbb{R}$ for agent $a_{i}$ maps
outcomes $S$ to real values based on the utility derived (Mankiw, 2020). For
NLG, the utility of a text is how good the agent perceives it to be,
optionally w.r.t. some attribute such as coherence.

######  Definition 4 (Ordinal Utility).

Under the theory of ordinal utility, only the ranking induced by $u_{i}$
matters; the magnitude of the difference between the values do not (Mankiw,
2020). A ordinary utility function $u_{i}$ represents $\succ_{i}$ if it
preserves the ranking the latter induces: $X\succeq_{i}Y\iff u_{i}(X)\geq
u_{i}(Y)$. Two utility functions $u,v$ are ordinally equivalent if they induce
the same preference ordering.

######  Definition 5 (Cardinal Utility).

Under the theory of cardinal utility, the magnitude of the difference between
two outcomes’ utility does matter. Two utility functions $u,v$ are cardinally
equivalent up to a positive affine transformation (Dybvig and Polemarchakis,
1981).

Estimating cardinal utility is the approach that has been implicitly taken by
the standard evaluation protocol for NLG (a.k.a., outcome-level absolute
assessment (OAA)). When an annotator rates an example, they are estimating the
cardinal utility $u_{i}(x)$ they get from an outcome $x$. When those ratings
are averaged to score the system $X$ that produced those examples, one is
estimating the expected cardinal utility of a lottery. Although
straightforward to implement, OAA requires strong – and at times untenable –
assumptions about agent preferences (§2.4).

Estimating the cardinal utility of a lottery as the expected utility of its
outcomes is possible in theory because of the von Neumann-Morgenstern theorem
(Morgenstern and Von Neumann, 1953). No similar result exists for estimating
the ordinal utility, however – we cannot average rankings. To derive an
ordinal utility function over the lotteries, we would have to ask the
annotator to categorically decide whether they prefer lottery $X$ or $Y$,
which is not viable when the outcome space is as large as it is in NLG.

###  2.4 Agent Rationality

Though rarely used nowadays compared to OAA, outcome-level relative assessment
(ORA) explicitly encodes its assumptions about annotator preferences in a
comparison model (e.g., Bradley-Terry (Bradley and Terry, 1952), Thurstone
(Thurstone, 1927), etc.). These assumptions are easy to identify and
invalidate, so we refer the reader to prior work on its limitations (Sakaguchi
et al., 2014; Bojar et al., 2016). Because it does not use a comparison model,
the now widely-used OAA may seem as though it makes no assumption about
annotators. However, ranking systems by their average rating only captures
annotator preferences if they are Von Neumann-Morgenstern-rational agents
(Morgenstern and Von Neumann, 1953):

######  Definition 6 (VNM Rationality).

Let $X^{\prime},Y^{\prime}$ denote random variables representing the outcomes
of lottery $X,Y$ respectively. For any von Neumann-Morgenstern-rational agent,
there exists a utility function $u_{i}$ such that
$X\succ_{i}Y\iff\mathbb{E}[u_{i}(X^{\prime})]>\mathbb{E}[u_{i}(Y^{\prime})]$.
In other words, VNM-rational agents always choose to maximize their expected
utility. In order for an annotator $a_{i}$ to be a VNM-rational agent, their
preferences must satisfy the following four axioms for any NLG systems
$X,Y,Z$:

######  Axiom 1 (Completeness).

For any $X,Y$, exactly one of the following holds for each agent $a_{i}$:
$X\succ_{i}Y,X\prec_{i}Y$ or $X\sim_{i}Y$ (i.e., the agent prefers $X$,
prefers $Y$, or is indifferent respectively).

######  Axiom 2 (Transitivity).

If $X\succ_{i}Y$ and $Y\succ_{i}Z$, then $X\succ_{i}Z$. The same holds for the
indifference relation $\sim_{i}$.

######  Axiom 3 (Continuity).

If $X\succeq_{i}Y\succeq_{i}Z,\exists\ p\in[0,1]$ such that
$pX+(1-p)Z\sim_{i}Y$.

######  Axiom 4 (Independence).

For any $Z$ and $p\in(0,1]$, we have $X\succ_{i}Y\iff
pX+(1-p)Z\succ_{i}Y+(1-p)Z$.

Although it may seem intuitive that any agent would maximize their expected
utility, work in behavioral economics has identified many situations where
agents choose not to do so (Samuelson, 1977; Kahneman and Tversky, 1979;
Allais, 1979). We argue later in the paper that the continuity axiom in
particular is unlikely to hold when humans evaluate NLG systems (§3, Remark
3).

##  3 Causes of Misrepresentation

By framing human evaluation in terms of utility theory, we found that the
standard protocol in NLG evaluation serves to estimate the cardinal utility of
a system via outcome-level absolute assessment (§2). We then listed the
assumptions that agent preferences need to satisfy in order to make this
estimation valid (§2.4). In this section, we discuss how these assumptions are
often violated in NLG evaluation, and how this begets misrepresentation of an
annotator’s true preferences. We limit our criticism to OAA in this section,
since ORA has already been criticized in prior work (Sakaguchi et al., 2014)
and has, over the past several years, become far less common than OAA (§6).

We begin by noting that rating generated text is done one of two ways
(Celikyilmaz et al., 2020):

  1. 1.

Likert scales, which discretize and normalize the utility into an integer from
1-to-$k$ (usually 1-to-5 or 1-to-7). To be more specific, a Likert scale is a
collection of Likert items, each of which is a discrete rating from 1-to-$k$.

  2. 2.

Continuous scales (a.k.a., continuous direct assessment), which normalize the
utility from 0-to-$k$ ($k$ usually being 100).

Our first two critiques of scoring apply only to Likert scales, but our last
two apply to the standard protocol at-large.

######  Remark 1 (Ordinal-Cardinal Conflation).

Averaging ordinal Likert ratings to estimate utility can violate tenets of
cardinal utility theory.

The Likert scale is ordinal: an outcome with a higher score is preferred to
one with a lower score, but the distance between the points is not
significant. In contrast, the intervals are significant in cardinal utility.
Averaging Likert ratings to estimate cardinal utility thus assumes that the
annotator has perceived the distance between each point to be the same, which
is impossible to verify. At best, annotators can be steered into an interval-
based interpretation through careful wording of the question, but there is no
guarantee that they will interpret the distances as intended. In a survey of
the NLG literature, Amidei et al. (2019) found that 31 of 38 papers using
Likert scales took an interval-based interpretation of them, but only 1 paper
provided justification for this interpretation.

This problem is not solved by normalization methods such as $z$-scoring, as
they do not work when the interval widths are asymmetric (e.g., the annotator
might perceive the jump between 1-to-2 to be larger than the jump from 2-to-3
on a 3-point scale). This is not a novel observation either; there is
extensive work on the limitations of averaging over Likert ratings (Jamieson,
2004; Sullivan and Artino Jr, 2013; Pimentel and Pimentel, 2019). Even early
shared tasks for NLG expressed this very concern and used continuous scales
instead Gatt and Belz (2009). Past empirical work, both in NLG and psychology,
also found annotators strongly preferred continuous scales to discrete scales,
as they permitted more nuanced judgments (Svensson, 2000; Belz and Kow, 2011).

######  Remark 2 (Biased Estimation).

Averaging Likert ratings can be a biased estimator of the expected utility,
potentially reversing the direction of the true preference over two NLG
systems.

Building upon Remark 1, let us make a best-case assumption that the annotator
perceives the intervals between the points on the Likert scale to be equal. As
such, they determine the Likert score by normalizing their utility to [0,5]
and then applying the ceiling function (e.g., $[0,1]\to 1;(1,2]\to 2$,
etc.).222Using a window of 0.5 around each number and rounding would make the
1-star bucket larger than the rest. This effectively replaces a subset of
preference relations $\succ_{i}$ with indifference relations. That is, if two
texts both have utilities in the tier $(i,i+1]$, the annotator becomes
indifferent to them because of this transformation.

Thus the distribution of the intra-tier relations – and whether they agree
with the inter-tier relations – will determine the direction and magnitude of
the bias. For example, say $X$-generated texts and $Y$-generated texts were
identically distributed across tiers, but the former were always preferred
within a tier. This advantage is erased by applying the ceiling function, so
both NLG systems end up with an identical distribution of Likert scores. This
can be stated more generally:

######  Proposition 1.

Let $r_{i}(s):=\lceil u_{i}(s)\rceil-u_{i}(s)$. Without loss of generality, if
$\mathbb{E}_{s\sim X}[r_{i}]>\mathbb{E}_{s\sim Y}[r_{i}]$, then Likert ratings
over-estimate the utility of lottery $X$ relative to $Y$; if
$\mathbb{E}_{X}[r_{i}]<\mathbb{E}_{Y}[r_{i}]$, they under-estimate the utility
of $X$ relative to $Y$.

######  Proposition 2.

Let $\mathbb{E}[u_{i}(X^{\prime})]>\mathbb{E}[u_{i}(Y^{\prime})]$ without loss
of generality. If
$(\mathbb{E}[u_{i}(X^{\prime})]-\mathbb{E}[u_{i}(Y)^{\prime}])<(\mathbb{E}_{Y}[r_{i}]-\mathbb{E}_{X}[r_{i}])$,
then averaging Likert ratings reverses the direction of the true preference.

Since our annotator is implicitly assumed to be VNM-rational, by the von
Neumann-Morgenstern theorem,
$X\succ_{i}Y\iff\mathbb{E}[u_{i}(X^{\prime})]>\mathbb{E}[u_{i}(Y^{\prime})]$.
Including the residuals can potentially change the direction of the inequality
between the expected utilities. Thus by the VNM theorem, it can also change
the direction of the preference relation. Since $r\in[0,1]$, the difference
$\mathbb{E}_{Y}[r_{i}]-\mathbb{E}_{X}[r_{i}]\leq 1$, meaning that a reversal
of preference could only occur when the annotator perceived both NLG systems
to produce outcomes of similar utility on average. This is a common situation
in practice, as proposed systems are often an incremental improvement over the
state-of-the-art (Card et al., 2020).

######  Remark 3 (Non-Independent Lotteries).

The independence of lotteries is an axiom of VNM-rationality but often fails
to hold in practice for NLG systems.

Collecting annotator ratings of text outputs and averaging them to get a score
for the NLG system is only viable if the annotators are VNM-rational (§2.4).
One of the conditions that needs to be satisfied for VNM-rationality is
independence over lotteries, as defined in Axiom 4. Put simply, the preference
$X\succ_{i}Y$ should not change if another lottery $Z$ is mixed with both in
equal proportion.

However, this assumption is often violated in the real world. Say for instance
that $X,Y$ place zero mass on offensive text (e.g., swear words, racial slurs,
etc.). This would be fairly typical for consumer-facing NLG systems, which may
explicitly filter out such outputs to avoid public outcry, the loss of users,
and a potential lawsuit. If lottery $Z$ places any mass on offensive output,
adding it to either $X$ or $Y$ may result in the system being unusable. Even
if an annotator has a non-zero tolerance for offensive content, under lottery
independence, the preference must hold regardless of the proportion in which
$Z$ is mixed with $X$ and $Y$. Mixing $Z$ with $X$ and $Y$ in equal proportion
does not change the direction of the expected utility inequality, but it can
change the relation between the lotteries from preference $X\succ_{i}Y$) to
indifference ($X\sim_{i}Y$) if both systems become unusable. In such a case,
the agent would not be VNM-rational, meaning that their preference could not
be inferred by comparing the expected utility of each NLG system.

Aside from our theoretical argument, there is also some empirical evidence
that the continuity axiom does not hold in the real world. Consumer
preferences are partially revealed by the NLG systems deployed in industry,
since these companies optimize for user satisfaction. Those deployed are often
template-based even though they produce less varied and less interesting text
than the latest open-ended NLG systems, suggesting that a model that behaves
reliably is preferred over one that produces higher-quality output on average
(Khadpe et al., 2020).

######  Remark 4 (Inter-Agent Incomparability).

Using the same scale across annotators is insufficient for aggregating their
cardinal utility (i.e., estimating the expected expected utility) due to
differences in the magnitude of utility.

When ranking NLG systems, we do not want to rank them according to just one
individual, since that individual’s preferences may be unrepresentative of the
population. In other words, there is a distribution over utility functions,
and we want to estimate the expected utility w.rt. this distribution. This
quantity is also known as the expected expected utility (EEU):
$\mathbb{E}_{i}[\mathbb{E}[u_{i}(X^{\prime})]]$ (Boutilier, 2003), which can
be expanded as

$\text{EEU}[X]=\int_{U}\mathbb{E}[u_{i}(X^{\prime})]p(u_{i})$ (1)

Then we could infer the direction of the aggregate preference over the entire
agent population, since $X\succ Y\iff\text{EEU}[X]>\text{EEU}[Y]$.

Estimating the EEU is not as straightforward as averaging the expected utility
estimates of different agents. Given a continuous scale from 0-to-100, one
agent may score in the range 0-to-10 while another may score in the range
90-to-100. Averaging across these two agents would bias the agent with a
greater magnitude of scoring. In technical terms, EEU is not invariant to the
choice of utility function in a set of cardinally equivalent utility
functions. This has been observed empirically and been framed as annotators
being too strict or too forgiving (Zemlyanskiy and Sha, 2018; Kulikov et al.,
2019).

Presenting all annotators with the same scale does not necessarily solve this
problem, since it does not force annotators to adopt the same magnitudes.
$Z$-scoring does not necessarily solve this problem either, since the
annotator scores are not guaranteed to be normally distributed. Relative
magnitude estimation (Moskowitz, 1977; Novikova et al., 2018), where the
annotator provides the score of an outcome relative to some reference,
partially addresses this problem, but using a single arbitrary reference point
is not provably sufficient (Boutilier, 2003). There are post hoc approaches to
standardizing scores as well – such as item response theory (Embretson and
Reise, 2013) – that can make empirically useful adjustments . However, such
approaches make assumptions about the distribution of the underlying choices
that are unverifiable; averaging over the adjusted scores does not give us an
unbiased estimate of the EEU.

Boutilier (2003) formally proved that extremum equivalence is sufficient to
estimate EEU, which he defined as: (1) all agents agree on the most and least
preferred outcomes; (2) all agents assign their most and least preferred
outcomes the utility $c_{\text{max}},c_{\text{min}}$ respectively, where
$c_{\text{max}}>c_{\text{min}}\geq 0$. These conditions might be satisfied in
machine translation, for example; one could argue that providing “correct” and
“incorrect” references forces all annotators to share utility function
endpoints. When there are no references or the space of high-quality outputs
is diverse, as in open-ended NLG tasks (e.g., chitchat dialogue), this
condition cannot be satisfied.

##  4 Improving the Standard Protocol

By making some minor changes, the OAA-based standard evaluation protocol can
be improved to a point where it can adequately capture human preferences in
some NLG tasks and settings:

  1. 1.

Continuous scales should be used instead of Likert scales to avoid ordinal-
cardinal conflation and potentially biased estimation. This need was pointed
out even in early shared tasks for NLG (Gatt and Belz, 2009).

  2. 2.

To satisfy extremum equivalence (§3, Remark 4), both maximal- and minimal-
utility references should be provided, effectively forcing all annotators’
utility functions to share endpoints. This can only be done when the space of
ideal outcomes for a given input is small and well-defined (e.g., machine
translation, extractive summarization, etc.). Doing so can manipulate agents’
utility functions, which may be desired when evaluating w.r.t. a particular
attribute such as fluency, but may not be desired when trying to capture true
human preferences. If extremum equivalence cannot be enforced, only one
annotator should be used for all ratings, though this risks yielding
conclusions that are unrepresentative of the population at large.

  3. 3.

To satisfy lottery independence, there should be no $-\infty$ utility outcomes
that can make an NLG system unusable (e.g., because any potential $-\infty$
utility outcome has been filtered out or because the system is only used by a
limited set of users whose utility functions are bounded).

The WMT competition for machine translation – which has experimented with many
evaluation schemes over the years – has had, since 2017, a protocol that
follows many of these suggestions (Bojar et al., 2017; Specia et al., 2021).
It uses continuous scales, provides positive (i.e., maximum-utility)
references, and the agent population comprises only translators hired for the
competition, meaning lottery independence is safe to assume. These changes
were driven by empirical evidence (e.g., higher inter-annotator agreement)
instead of the theoretical arguments that motivated our suggestions,
suggesting that the theoretical problems we identified indeed have practical
consequences. WMT does not provide negative (i.e., minimal-utility)
references, however, so there remains room for improvement.

Still, this theoretically sounder protocol cannot be applied for all NLG tasks
and settings. Maximal- and minimal-utility references are not available in
open-ended tasks where there is no singular notion of correctness or where
maximal-utility outcomes can be very diverse (e.g., story generation, chitchat
dialogue, etc.). Moreover, the assumption of lottery independence is likely be
violated in the real-world; for example, consumers of a chatbot may find it
completely unusable if it produces any offensive content. Such tasks and
settings demand an entirely new evaluation protocol, which we propose in (§5).

##  5 System-level Probabilistic Assessment

We observe that the limitations of both ORA and OAA stem from trying to
aggregate preferences over outcomes into a preferences over systems, despite
there being no canonical way to do so. For example, one annotator may prefer
$X$ to $Y$ only if the former wins head-to-head comparison of outputs over 50%
of the time, but another may choose by comparing the worst-case output
generated by each. Therefore we propose a middle ground: directly estimate the
probability $P[X\succ Y]$ that a preference holds across two systems, a
protocol we call system-level probabilistic assessment (SPA).

###  5.1 Theory

Where $p(\succ_{i})$ is the frequency (among the agent population) of
preferences $\succ_{i}$, we can expand $P[X\succ Y]$ similarly to EEU in (1):

$P[X\succ Y]=\int_{\succ_{i}}P[X\succ_{i}Y]p(\succ_{i})$ (2)

Since $P[X\succ_{i}Y]\in[0,1]$ for all $a_{i}$, the values are inherently
comparable across annotators, making inter-annotator aggregation easy. If we
assumed preferences were complete, then $P[X\succ_{i}Y]$ could only take a
value in $\\{0,1\\}$, but doing so would be unrealistic, since annotators are
almost never exposed to the entirety of an NLG system’s output in practice,
precluding them from preferring one system with absolute certainty. Therefore
we model preferences as stochastic.

However, an annotator’s preferences change as they are exposed to more output
while $P[X\succ_{i}Y]$ is a fixed value. How can we reconcile this? We take
the view that every time an annotator’s preference probability is updated,
they effectively become a new agent (i.e., an agent no longer represents an
individual annotator, but a specific iteration of an annotator with fixed
beliefs). For example, at the start, an annotator has no knowledge of the
systems, so $P[X\succ_{i,t=0}Y]=0.5$. As they are exposed to more outputs,
they may develop a preference for one system (e.g., $P[X\succ_{i,t=1}Y]=0.7$).
At some point they will become certain about their choice (e.g.,
$P[X\succ_{i,t=n}Y]=1$ for some large $n$), but at this point the annotator is
no longer the same agent that was split between the two options. This is a
useful property – the agent population will be constantly evolving in the real
world and with it the aggregate utility of the system will too. By composing
different agent populations during the evaluation stage – for example, by
exposing annotators to different amounts of output – the aggregate preference
of these different populations can be estimated in vitro.

The standard protocol in NLG evaluation requires that annotators be VNM-
rational and have preferences that are complete, transitive, independent,
continuous, and extremum equivalent (§2.4). SPA obviates those assumptions by
delegating the responsibility of aggregating preferences over outcomes into a
preference over the underlying lotteries to the agent themself, acknowledging
that there is no canonical way to do so. Estimating $P[X\succ Y]$ only
requires one assumption:

######  Definition 7 (Unbiased Stated Preferences).

An agent $a_{i}$ has unbiased stated preferences if, when asked to estimate
the probability of their preference for lottery $X$ over $Y$, they provide an
unbiased estimate $\hat{P}[X\succ_{i}Y]$.

###  5.2 Implementing SPA

If you want to use SPA to compare two NLG systems $X,Y$, you should do as
follows:

  1. 1.

Choose the prior for your desired task and provide a sample of $m$
prompts/inputs drawn from this prior. Alternatively, allow agents to use their
own under the assumption that they share a prior over the prompt/input space.

  2. 2.

Find $n_{A}$ unique annotators that are representative of the agent population
whose preferences you want to model. Allow agents to directly interact with
each system or show them a fixed number of randomly sampled outputs from each
system based on the chosen prior. We take the latter approach in our
experiments (§5.3), as we include human-written references as well.

  3. 3.

Then ask each annotator a variation of the following question: “Based on what
you’ve read, from 0 to 100, what is the % chance that system $X$ is a better
writer than system $Y$?” If needed, swap $X$ and $Y$ and repeat the question.
Note that $\hat{P}[X\succ_{i}Y]$ does not necessarily need to equal
$1-\hat{P}[Y\succ_{i}X]$ since the agent’s estimates are noisy and they can be
indifferent to the two.

  4. 4.

Estimate the aggregate probability $P[X\succ Y]$ by averaging over
$\\{\hat{P}[X\succ_{i}Y]\\}$. Use a Student’s $t$-test to determine whether
$P[X\succ Y]$ is significantly different from chance (0.5).

In Appendix A, we provide details of the SPA implementation we use in our
experiments in §5.3.

System $X$ | System $Y$ | Expected Preference | $P[X\succ Y]$ | Likert Rating $\Delta$
---|---|---|---|---
GPT-3-ada | human | $X\prec Y$ | 0.420* | $-$0.822***
GPT-3-babbage | GPT-3-ada | $X\succ Y$ | 0.688*** |  0.644***
GPT-3-curie | GPT-3-babbage | $X\succ Y$ | 0.630*** |  0.322
GPT-3-davinci | GPT-3-curie | $X\succ Y$ | 0.575*** |  0.244
human | GPT-3-davinci | $X\sim Y$ | 0.544 | $-$0.389*
Table 1: Eliciting human preferences for story generation, using both system-
level probabilistic assessment (SPA) and the standard protocol with 5-point
Likert ratings. We use a Student’s $t$-test with Holm-Bonferroni-corrected
statistical significance at $\alpha=0.10(^{*}),0.05(^{**}),0.01(^{***})$. SPA
yields a significant result in the expected direction for all comparisons,
while Likert ratings only do so for 2 (green). Notably, the standard protocol
yields the unintuitive result that human-written text is significantly less
preferred to davinci-written text (red), although past work has found that
annotators cannot tell the difference (Clark et al., 2021). Insignificant
results are in yellow.

###  5.3 Experiments

To test our proposed protocol, we ask 100 unique crowdworkers to use both the
standard protocol (with a 5-point Likert scale) and SPA to express their
preferences about the different GPT-3 variants w.r.t. their story-writing
ability (see Appendix A). The story prompts are drawn from the WritingPrompts
dataset (Fan et al., 2018) and each annotator is given: $m$ randomly drawn
prompts, stories generated by each GPT-3 variant for those prompts, and a
human-written story for each prompt. The annotator is not told which of the 5
systems is a human. With SPA, they are asked to compare the systems
themselves, while with the standard protocol, they are just asked to rate the
outputs. The smaller $m$ is, the more uncertain annotators will be about their
preference, making it hard to elicit a statistically significant result in
SPA. The larger $m$ is, the higher the per annotator cost, since each
annotator will take longer to complete each task. We balance these concerns by
choosing $m=5$.

Given that larger GPT-3 variants generate more coherent, grammatical, and
creative text, annotators should overall prefer larger variants to smaller
ones: i.e.,
$\texttt{davinci}\succ\texttt{curie}\succ\texttt{babbage}\succ\texttt{ada}$
(Brown et al., 2020). Clark et al. (2021) also found that annotators can
distinguish between GPT2- and human-written text, but not between the latter
and davinci-written text. Since ada is not much larger than GPT2, this implies
that the following preferences should also hold:
$\texttt{human}\succ\texttt{ada}$ and $\texttt{human}\sim\texttt{davinci}$.
For SPA, we use a Student’s $t$-test to measure whether each probabilistic
preference is significantly different from chance ($P[X\succ Y]=0.5$). For the
standard protocol, we use a paired $t$-test to determine whether the Likert
ratings of two systems’ outputs are significantly different from one another.
As we are making multiple comparisons, we apply the Holm-Bonferroni correction
(Holm, 1979).

As seen in Table 1, SPA recovers 5/5 expected preferences: each GPT-3 variant
is significantly preferred to the next smallest one; ada is significantly less
preferred to human-written text; and annotators are indifferent to human- and
davinci-written text. However, the standard protocol only recovers 2/5
expected preferences: curie and babbage are not significantly preferred to the
next smallest GPT-3 variant, and surprisingly the probability of a human being
a better writer than davinci is significantly less than chance, despite past
work suggesting that annotators cannot tell the difference between the two
(Clark et al., 2021).

###  5.4 Limitations

Although SPA does not suffer from the existential limitations of the standard
evaluation protocol (§3), it does have two notable limitations.

  1. 1.

SPA does not measure the magnitude of a preference, only the probability that
it exists. This is a necessary trade-off for SPA to be applicable to open-
ended NLG tasks, for which extremum equivalence (§3) cannot be satisfied when
measuring magnitude.

  2. 2.

Annotators may not understand the notion of probability or may not read the
outputs assigned to them, providing noisy and biased annotations. This problem
is not unique to SPA, but since human preferences are inherently subjective,
identifying insincere annotators is more difficult. Also, since we want to
estimate the aggregate preference of an agent population, we have to use
$n_{A}$ unique agents, instead of letting a few talented annotators do most of
the work, as is common in NLP (Geva et al., 2019). We discuss possible
strategies in Appendix A and show in Figure 1 how adding filtering techniques
in future work may make SPA even more effective.

##  6 Related Work

Soliciting humans to directly evaluate the quality of generated text is known
as intrinsic evaluation. The text can be judged for its overall quality or
along a specific dimension such as coherence, though there is much
inconsistency is how these terms are defined and explained to annotators
(Howcroft et al., 2020; van der Lee et al., 2021). This is most often done in
the NLG literature by having annotators assign a Likert rating from 1-to-$k$,
where $k$ is usually 3 or 5 (Van Der Lee et al., 2019). Given the ubiquity of
this standard protocol, little justification is given when it is used and
implicit assumptions, such as equal intervals for Likert scales, are entirely
omitted (Amidei et al., 2019).

Figure 1: Only considering annotators whose stated probabilities concur with
the Likert ratings they assign to the system outputs – despite being a noisy
filter for annotator quality – gives us preference probabilities that more
strongly go in the expected direction. The main results in Table 1 are given
without this filter.

The earliest shared tasks in NLG, such as the TUNA (Gatt and Belz, 2009) and
GREC (Belz and Kow, 2011) challenges for expression generation, used a
continuous scale for scoring, explicitly noting that annotators may not
perceive the intervals on a Likert scale to be equal. In contrast, early
modeling work – such as the STORYBOOK system for narrative prose generation
(Callaway and Lester, 2002) – used discrete ratings. This difference in
evaluation protocol between shared challenges and individual modeling papers
continued over the years. For example, the E2E NLG challenge (Dušek et al.,
2018) used continuous scores based on relative magnitude estimation (Novikova
et al., 2018; Bard et al., 1996). However, these challenges have not served as
a bulwark against the popularity of Likert-based OAA – even recent attempts to
standardize human evaluation in NLG using evaluation platforms collect Likert
ratings (Khashabi et al., 2021; Gehrmann et al., 2021).

Compared to OAA, outcome-level relative assessment is far less common
nowadays, in large part because making pairwise comparisons of outcomes is
more expensive than outcome-level ratings at scale (Celikyilmaz et al., 2020).
Recall that given binary outcome-level preferences (e.g., $x_{i}\succ y_{i}$)
as labels, ORA uses a preference model such as Bradley-Terry to estimate the
scores of the systems, analogous to how ELO scores are calculated for chess
players (Chu and Ghahramani, 2005). In explicitly stating its assumptions
about annotator preferences using a preference model, ORA was easier to
criticize than OAA, which contributed to the former’s decline (Sakaguchi et
al., 2014). The one area in which comparison-based evaluation still prevails
is when conducting a Turing test – seeing whether annotators do better than
chance when guessing whether a text is human- or machine-generated (Garbacea
et al., 2019; Ippolito et al., 2020; Brown et al., 2020; Clark et al., 2021).
This is acceptable, since what is being measured is not annotator preference
but rather discriminability.

Over the years, machine translation (MT) has had spirited debate about
evaluation, but little of it extends to open-ended NLG, as the latter lacks
strict references. Callison-Burch et al. (2007) found that compared to ranking
outputs, annotators took more time and agreed less when providing Likert
scores. Citing this, Sakaguchi et al. (2014) use the TrueSkill algorithm
(Herbrich et al., 2006) to estimate scores for NLG systems based on pairwise
preferences of their output. This approach, called _relative ranking_ (RR) was
used in the WMT competition until 2016, when direct assessment (DA) on a
0-to-100 continuous scale were trialled and found to produce rankings of MT
systems that strongly correlated with RR (Bojar et al., 2016). DA also had the
advantage of providing an absolute measure of quality, so it was adopted as
the standard for WMT competitions in 2017 and used thereafter (Bojar et al.,
2017; Specia et al., 2021).

To our knowledge, utility theory has only been applied in NLP to design
leaderboards (Ethayarajh and Jurafsky, 2020; Ma et al., 2021). However, past
work has hinted at utilitarian issues without framing them as such, such as
whether crowdworkers are sufficiently motivated (Belz and Reiter, 2006; Dugan
et al., 2020). Some work has recommended best practices to align annotator’s
incentives with that of potential users of the NLG system (Oppenheimer et al.,
2009; Daniel et al., 2018), which can include training crowdworkers to become
better evaluators (Mitra et al., 2015; Clark et al., 2021).

##  7 Conclusion

We analyzed the standard evaluation protocol in NLG through the lens of
utility theory, finding that it makes untenable assumptions about annotator
preferences. When these assumptions are violated, annotator ratings become an
unfaithful reflection of their preferences, both in theory and in practice. We
proposed a new evaluation protocol called SPA that makes minimal assumptions
about annotator preferences – not only is it more theoretically sound than the
standard protocol, but it performs better in practice as well, consistently
recovering the expected preference with statistically significant results. An
important direction of future work will be re-evaluating conclusions derived
using the standard protocol and seeing which conclusions in the NLG literature
stand up to scrutiny.

## Acknowledgements

Many thanks to Kaitlyn Zhou for helpful discussion and feedback. We also thank
Clara Meister and Simran Arora for answering queries about annotation on
MTurk.

## Acknowledgements

## References

  * Allais (1979) Maurice Allais. 1979.  The so-called allais paradox and rational decisions under uncertainty.  In _Expected utility hypotheses and the Allais paradox_ , pages 437–681. Springer.
  * Amidei et al. (2019) Jacopo Amidei, Paul Piwek, and Alistair Willis. 2019.  The use of rating and likert scales in natural language generation human evaluation tasks: A review and some recommendations.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 397–402.
  * Bard et al. (1996) Ellen Gurman Bard, Dan Robertson, and Antonella Sorace. 1996.  Magnitude estimation of linguistic acceptability.  _Language_ , pages 32–68.
  * Belz and Kow (2011) Anja Belz and Eric Kow. 2011.  Discrete vs. continuous rating scales for language evaluation in nlp.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 230–235.
  * Belz and Reiter (2006) Anja Belz and Ehud Reiter. 2006.  Comparing automatic and human evaluation of nlg systems.  In _11th conference of the european chapter of the association for computational linguistics_ , pages 313–320.
  * Bojar et al. (2017) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, et al. 2017.  Findings of the 2017 conference on machine translation (wmt17).  In _Proceedings of the Second Conference on Machine Translation_ , pages 169–214.
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016.  Findings of the 2016 conference on machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 131–198.
  * Boutilier (2003) Craig Boutilier. 2003.  On the foundations of expected expected utility.  In _IJCAI_ , volume 3, pages 285–290. Citeseer.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Callaway and Lester (2002) Charles B Callaway and James C Lester. 2002.  Narrative prose generation.  _Artificial Intelligence_ , 139(2):213–252.
  * Callison-Burch et al. (2007) Chris Callison-Burch, Cameron Shaw Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007.  (meta-) evaluation of machine translation.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 136–158.
  * Card et al. (2020) Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. 2020.  With little power comes great responsibility.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9263–9274.
  * Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020.  Evaluation of text generation: A survey.  _arXiv preprint arXiv:2006.14799_.
  * Chu and Ghahramani (2005) Wei Chu and Zoubin Ghahramani. 2005.  Preference learning with gaussian processes.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 137–144.
  * Clark et al. (2021) Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. 2021.  All that’s ‘human’is not gold: Evaluating human evaluation of generated text.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7282–7296.
  * Daniel et al. (2018) Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018.  Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions.  _ACM Computing Surveys (CSUR)_ , 51(1):1–40.
  * Dugan et al. (2020) Liam Dugan, Daphne Ippolito, Arun Kirubarajan, and Chris Callison-Burch. 2020.  Roft: A tool for evaluating human detection of machine-generated text.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 189–196.
  * Dušek et al. (2018) Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018.  Findings of the e2e nlg challenge.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 322–328.
  * Dybvig and Polemarchakis (1981) Philip Dybvig and Heraklis Polemarchakis. 1981.  Recovering cardinal utility.  _The Review of Economic Studies_ , 48(1):159–166.
  * Embretson and Reise (2013) Susan E Embretson and Steven P Reise. 2013.  _Item response theory_.  Psychology Press.
  * Ethayarajh and Jurafsky (2020) Kawin Ethayarajh and Dan Jurafsky. 2020.  Utility is in the eye of the user: A critique of nlp leaderboards.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4846–4853.
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical neural story generation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 889–898.
  * Garbacea et al. (2019) Cristina Garbacea, Samuel Carton, Shiyan Yan, and Qiaozhu Mei. 2019.  Judge the judges: A large-scale evaluation study of neural language models for online review generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3968–3981.
  * Gatt and Belz (2009) Albert Gatt and Anja Belz. 2009.  Introducing shared tasks to nlg: The tuna shared task evaluation challenges.  In _Empirical methods in natural language generation_ , pages 264–293. Springer.
  * Gehrmann et al. (2021) Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, et al. 2021.  The gem benchmark: Natural language generation, its evaluation and metrics.  In _Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)_ , pages 96–120.
  * Geva et al. (2019) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019.  Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1161–1166.
  * Herbrich et al. (2006) Ralf Herbrich, Tom Minka, and Thore Graepel. 2006.  Trueskill™: a bayesian skill rating system.  _Advances in neural information processing systems_ , 19.
  * Holm (1979) Sture Holm. 1979.  A simple sequentially rejective multiple test procedure.  _Scandinavian journal of statistics_ , pages 65–70.
  * Howcroft et al. (2020) David M Howcroft, Anja Belz, Miruna-Adriana Clinciu, Dimitra Gkatzia, Sadid A Hasan, Saad Mahamood, Simon Mille, Emiel Van Miltenburg, Sashank Santhanam, and Verena Rieser. 2020.  Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 169–182.
  * Ippolito et al. (2020) Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020.  Automatic detection of generated text is easiest when humans are fooled.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1808–1822.
  * Jamieson (2004) Susan Jamieson. 2004.  Likert scales: How to (ab) use them?  _Medical education_ , 38(12):1217–1218.
  * Kahneman and Tversky (1979) D Kahneman and A Tversky. 1979.  Prospect theory: An analysis of decision under risk.  _Econometrica: Journal of the Econometric Society_.
  * Khadpe et al. (2020) Pranav Khadpe, Ranjay Krishna, Li Fei-Fei, Jeffrey T Hancock, and Michael S Bernstein. 2020.  Conceptual metaphors impact perceptions of human-ai collaboration.  _Proceedings of the ACM on Human-Computer Interaction_ , 4(CSCW2):1–26.
  * Khashabi et al. (2021) Daniel Khashabi, Gabriel Stanovsky, Jonathan Bragg, Nicholas Lourie, Jungo Kasai, Yejin Choi, Noah A Smith, and Daniel S Weld. 2021.  Genie: A leaderboard for human-in-the-loop evaluation of text generation.  _arXiv preprint arXiv:2101.06561_.
  * Kulikov et al. (2019) Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019.  Importance of search and evaluation strategies in neural dialogue modeling.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 76–87.
  * Ma et al. (2021) Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. 2021.  Dynaboard: An evaluation-as-a-service platform for holistic next-generation benchmarking.  _Advances in Neural Information Processing Systems_ , 34.
  * Mankiw (2020) N Gregory Mankiw. 2020.  _Principles of economics_.  Cengage Learning.
  * Mitra et al. (2015) Tanushree Mitra, Clayton J Hutto, and Eric Gilbert. 2015.  Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , pages 1345–1354.
  * Morgenstern and Von Neumann (1953) Oskar Morgenstern and John Von Neumann. 1953.  _Theory of games and economic behavior_.  Princeton university press.
  * Moskowitz (1977) Howard R Moskowitz. 1977.  Magnitude estimation: notes on what, how, when, and why to use it.  _Journal of Food Quality_ , 1(3):195–227.
  * Novikova et al. (2018) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2018.  Rankme: Reliable human ratings for natural language generation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 72–78.
  * Oppenheimer et al. (2009) Daniel M Oppenheimer, Tom Meyvis, and Nicolas Davidenko. 2009.  Instructional manipulation checks: Detecting satisficing to increase statistical power.  _Journal of experimental social psychology_ , 45(4):867–872.
  * Pimentel and Pimentel (2019) J Pimentel and JL Pimentel. 2019.  Some biases in likert scaling usage and its correction.  _International Journal of Science: Basic and Applied Research (IJSBAR)_ , 45(1):183–191.
  * Sakaguchi et al. (2014) Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2014.  Efficient elicitation of annotations for human evaluation of machine translation.  In _Proceedings of the Ninth Workshop on Statistical Machine Translation_ , pages 1–11.
  * Samuelson (1977) Paul A Samuelson. 1977.  St. petersburg paradoxes: Defanged, dissected, and historically described.  _Journal of Economic Literature_ , 15(1):24–55.
  * Specia et al. (2021) Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary, and André Martins. 2021.  Findings of the wmt 2021 shared task on quality estimation.  In _Proceedings of the Sixth Conference on Machine Translation_. Association for Computational Linguistics.
  * Sullivan and Artino Jr (2013) Gail M Sullivan and Anthony R Artino Jr. 2013.  Analyzing and interpreting data from likert-type scales.  _Journal of graduate medical education_ , 5(4):541–542.
  * Svensson (2000) Elisabeth Svensson. 2000.  Comparison of the quality of assessments using continuous and discrete ordinal rating scales.  _Biometrical Journal: Journal of Mathematical Methods in Biosciences_ , 42(4):417–434.
  * Thurstone (1927) Louis L Thurstone. 1927.  A law of comparative judgment.  _Psychological review_ , 34(4):273.
  * van der Lee et al. (2021) Chris van der Lee, Albert Gatt, Emiel van Miltenburg, and Emiel Krahmer. 2021.  Human evaluation of automatically generated text: Current trends and best practice guidelines.  _Computer Speech & Language_, 67:101151.
  * Van Der Lee et al. (2019) Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel Krahmer. 2019.  Best practices for the human evaluation of automatically generated text.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 355–368.
  * Zemlyanskiy and Sha (2018) Yury Zemlyanskiy and Fei Sha. 2018.  Aiming to know you better perhaps makes me a more engaging dialogue partner.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 551–561.

##  Appendix A Experiment Details

###  A.1 Story Generation

For each annotator, we first randomly sampled $m=5$ story prompts from the
WritingPrompts datasets after filtering out any prompt that: (1) did not begin
with [ WP ]; (2) contained a question mark; (3) did not end in punctuation.
This was done so that the writing prompts were all of a consistent format and
style. We observed that prompts ending in questions sometimes elicited opinion
essays from GPT-3, as opposed to a fictional continuation. In our trial runs,
this confused some annotators, who thought all writers were writing fictional
continuations. We thus over-corrected by excluding all prompts with questions.

For each writing prompt, we generated a story by each of the four GPT-3
variants: davinci-002, curie-001, babbage-001, ada-001, which we anonymized as
writers D,C,B,A respectively. We set the following hyperparameters for all
models: a maximum of 1600 tokens, top-$p$ of 1, and a temperature of 0.9. Each
story prompt came with a human-written continuation as well, which we
anonymized as writer E. In practice, the GPT-3 models usually generated far
fewer than the allowable 1600 tokens, and a result, the human-written stories
were longer than their machine-written counterparts. To prevent annotators
from using story length as a proxy for quality, we trimmed – to the nearest
whole sentence – the human-written story for each prompt so that it was no
longer than the longest machine-written story for that prompt.

The 25 continuations (5 writers $\times$ 5 prompts) that each annotator had to
read were put up on a static website, where the annotator would input their
assigned ID to read the batch that was assigned to them (see Figure 2). The
annotator was informed that there were a mix of human and AI writing systems,
but we did not reveal which writers were which or how many of each there were.

Figure 2: The interface to the generated stories. The continuations generated
by the GPT-3 models (A,B,C,D) and the human-written continuation (E) were
placed side-by-side.

Figure 3: The instructions given to annotators on Amazon Mechanical Turk.

###  A.2 Filtering Annotators

We recruited $n_{A}=100$ annotators on Amazon Mechanical Turk, filtering for
those who were in the US, had a HIT approval rate $>98\%$, and who had
completed at least 100 HITs. Each annotator was paid $5 for approximately 20
minutes of work, working out to roughly $15/hour. Each annotator was presented
with the instructions in Figure 3 and then asked to provide 5 preference
probabilities $P[X\succ Y]$, one for each comparison of interest. They were
asked to evaluate each writing system on the basis of how coherent, fluent,
interesting and relevant to the prompt the stories were.

They were then asked to provide a Likert rating of the first continuation
written by each writer. We did not ask for a rating of all 25 continuations
because that would be onerous and unnecessary; for an apples-to-apples
comparison of SPA and the standard evaluation protocol, we should have an
equal sample size for each. This way, for every pair of systems that were
being compared, we would have 100 probability estimates of the preference and
100 Likert rating deltas that we could feed into a Student’s $t$-test. Since
the order of the prompts was random, asking the annotator to provide a Likert
rating for the first continuation (as opposed to say, the second or third)
made no systematic difference.

After the annotators provided their annotations, we excluded those who: (1)
said they were not native English speakers; (2) did not follow our
instructions and submitted multiple HITs. 10% of annotators were excluded,
leaving 90 whose annotations we used. Submitting multiple HITS was an issue
because we wanted to control the amount of exposure that the annotator had to
each writing system, which is why we provided exactly 5 samples from each.
Annotations were collected in small batches to prevent thee same annotators
from making multiple submissions.

###  A.3 Future Work

Our design choices in our experiments raise two interesting directions for
future work:

  1. 1.

How do the results change as $m$, the number of samples seen, changes? We
chose $m=5$ to trade-off the greater uncertainty from viewing too few samples
from each writing system against the greater cost and time needed to read many
samples. However, what does this trade-off look like precisely? Is there are
Pareto-optimal choice for $m$?

  2. 2.

How can we better filter out insincere and inattentive annotators? After all,
just because an annotator states they’re a native English speaker and only
submitted one HIT does not necessarily mean they read all the writing samples
thoroughly, as instructed. One idea is to hire annotators personally, but this
naturally does not scale as well as MTurk.