# Crowdsourcing with Difficulty:
A Bayesian Rating Model for Heterogeneous Items

Seong Woo Han University of Pennsylvania  Ozan Adıgüzel Own Company  Bob
Carpenter Center for Computational Mathematics, Flatiron Institute

###### Abstract

In applied statistics and machine learning, the “gold standards” used for
training are often biased and almost always noisy. Dawid and Skene’s
justifiably popular crowdsourcing model [10] adjusts for rater (coder,
annotator) sensitivity and specificity, but fails to capture distributional
properties of rating data gathered for training, which in turn biases
training. In this study, we introduce a general purpose measurement-error
model with which we can infer consensus categories by adding item-level
effects for difficulty, discriminativeness, and guessability. We further show
how to constrain the bimodal posterior of these models to avoid (or if
necessary, allow) adversarial raters. We validate our model’s goodness of fit
with posterior predictive checks, the Bayesian analogue of $\chi^{2}$ tests.
Dawid and Skene’s model is rejected by goodness of fit tests, whereas our new
model, which adjusts for item heterogeneity, is not rejected. We illustrate
our new model with two well-studied data sets, binary rating data for caries
in dental X-rays and implication in natural language.

##  1 Introduction

Figure 1: Graphical sketch of the model. Sizes: $J$ number of annotators,
$K=2$ number of categories, $I$ number of items, $N$ number of categories.
Observed data: $rating,rater,item$ labels. Parameters: $\theta$ annotator
accuracies/biases, $\pi$ category prevalence, $z$ true item category, $\beta$
item difficulty, $\delta$ item discrimination, $\lambda$ item guessing.

Crowdsourcing is the process of soliciting ratings (labels, annotations,
codes) from a group of respondents for a set of items and combining them in a
useful way for some task, such as training a neural network classifier or a
large language model.

The use of the term “crowdsourcing” is not meant to imply that the raters are
untrained—the crowd may be one of oncologists, lawyers, or professors. This
note focuses on binary ratings, which arise in two-way classification
problems, such as whether a patient has a specific condition, whether an image
contains a traffic light, whether one consumer product is better than another,
whether an action is ethical, or whether one output of a large language model
is more truthful than another.

We evaluate with two representative data sets, one for image recognition and
one for natural language processing. The first data set involves dental X-rays
being rated as positive or negative for caries, a kind of pre-cavity [12].
There are thousands of X-rays, each of which is rated positive or negative for
caries (a pre-cavity) by the same five dentists. The dentists showed
surprisingly little agreement and consensus, especially in cases where at
least one dentist rated the X-ray positive for caries. The dentists also
varied dramatically in the number of cases each rated as having caries,
indicating rater-specific bias toward positive or negative ratings. The second
data set consists of pairs of sentences, which are rated positive if the first
entails the second. The second data set was collected through Amazon’s
Mechanical Turk and involves thousands of sentence pairs and dozens of raters,
each of whom rated only a subset of the items [37]. Classification among
dentists and natural language semantics are both challenging classification
tasks with subjectively defined boundaries, and thus it is not surprising that
the rater agreement level is low. The same kind of “bias” arises in rating
data for human values as might be used to align a large language model [4].

There are several tasks to which crowdsourcing models are applied, and for
every one of them a rating model improves performance over heuristic
baselines. For example, rating models outperform majority voting for category
training and outperform indirect measurements like inter-annotator agreement
statistics to measure task difficulty (see, e.g., [3], [34], [22]).

Inferring a gold-standard data set. The first and foremost application of
crowdsourcing is to generate a “gold standard” data set, where a single
category (or label) is assigned to each item. In terms of generating
representative data, it is best to sample data according to its probability
(i.e., follow the generative model) rather than to choose the “best” rating
for each item according to some metric such as highest probability, which is
similar to the commonly used heuristic of majority vote. If the goal of
creating a gold standard data set is to train a machine learning classifier,
the second section of this paper shows why it is better for downstream
accuracy to train with a probabilistic corpus that retains information about
rating uncertainty. Barring that, it is far better to sample labels according
to their posterior probability in the rating model than to choose the “best”
label. In particular, we will demonstrate that majority voting schemes among
raters are sub-optimal compared to sampling, which is in turn dominated by
training with the probabilities (a kind of Rao-Blackwellization). If a
classifier for the data is available, an even better approach is to jointly
train a classifier and rating model [30].

Inferring population prevalence. The second most common application of
crowdsourcing is to understand the probability of positivity among items in
the population represented by the crowdsourcing data. This is particularly
common in epidemiology, where the probability of positive outcomes is the
prevalence of the disease in the (sub)population [2]. It can also be used to
analyze the prevalence of hate speech on a social media site or bias in
televised news, the prevalence of volcanoes on Venus, or the prevalence of
positive reviews for a restaurant.

Understanding and improving the coding standard. The third most common
application of crowdsourcing is to understand the coding standard, which is
the rubric under which rating is carried out. Typically this is measured
through inter-annotator agreement, but rating models provide finer-grained
analysis of sensitivity and specificity of raters [24].

Understanding and improving raters. What is the mean sensitivity and
specificity and how does it vary among raters? Are sensitivity and specificity
anticorrelated or correlated in the population? This understanding can be fed
back to the raters themselves for ongoing training. For example, American
baseball umpires have extensive feedback on how they call balls and strikes as
measured against a very accurate machine’s call, which has led to much higher
accuracy and consistency among umpires [13]. Understanding rater populations,
such as those available through Mechanical Turk or Upwork, is important when
managing raters for multiple crowdsourcing tasks. For example, it is
straightforward with a rating model to infer the proportion of spammers and
the proportion of high quality raters. The number of raters required for high
quality joint ratings may also be assessed [24].

Understanding the items. A fifth task that is rarely considered in the
crowdsourcing literature is to understand the structure of the population of
items. For example, which items are difficult to rate and why? Which items
simply have too little signal to be consistently rated? Which items live on
the boundary of the rating decision boundary and which live far away? Which
items have high discrimination and why? A discriminative item is one which
cleanly separates high ability from low ability raters in their ability to
rate it correctly. Understanding the items is the primary focus of educational
test design, where the items are test questions and the raters are students.
Test questions are selected for standardized tests like the ACT (American
College Testing) based on having high discrimination and a useful range of
difficulties [1].

###  1.1 Posterior predictive checks

We use posterior predictive checks (PPC) for goodness-of-fit testing [16].
They are the Bayesian analogue of the $\chi^{2}$ goodness-of-fit tests widely
employed in epidemiology [2]. PPCs work by probabilistically generating
replicated data sets based on the fitted model parameters, then comparing
statistics obtained from the replicated data set to the same statistic applied
to the original data set. The posterior predictive distribution for
replications $y^{\textrm{rep}}$ of the original data set $y$ given model
parameters $\theta$ is defined by $p(y^{\textrm{rep}}\mid y)=\int
p(y^{\textrm{rep}}\mid\theta)\cdot p(\theta\mid y)\ \textrm{d}\theta.$

If a model accurately represents the data, data statistics should exhibit
similar values in both the original and replicated data sets. This can be
evaluated using a $p$-value-like statistic, which is the probability that the
test statistic $s(\cdot)$ in a replicated data set surpasses that in the
original data, $\Pr[s(y^{\textrm{rep}})\geq s(y)\mid
y]=\int\text{I}(s(y^{\textrm{rep}})\geq s(y))\cdot p(y^{\textrm{rep}}\mid y)\
\textrm{d}y^{\textrm{rep}}$.

Any choice of statistic guided by the quantities of interest in the model
itself can be used for the data. We will consider recovering marginal vote
counts for items and raters.

##  2 Previous work

Rating models show up in multiple fields, including educational testing, from
which the model variants introduced here are derived [19, 20, 29]. Rating
models are widely used in epidemiology, both for multiple diagnostic testing
[2] and extracting health status information from patient records [10]. In
sociology, rating models were independently developed for cultural consensus
theory [4, 32]. More recently, they have become popular for providing human
feedback for classification of images [30, 36]; human ratings are the basis of
massive data sets of millions of images and thousands of classes like ImageNet
[11]. Rating models have long been popular for natural language tasks [24,
37]. It is possible to introduce a crude model of difficulty by assuming items
are a mixture of “easy” items on which all annotators will agree and “hard”
items on which they will struggle [5]. More recently, crowdsourced ratings of
language model output are used as a fine-tuning step that adjusts a
foundational large language model like GPT-4, which is trained to complete
text, into a chatbot like ChatGPT that is (imperfectly) trained to be helpful,
truthful, and harmless [23, 28].

##  3 Contributions

The primary contribution of this work is the expansion of rating models to
account for item variation in the form of difficulty, discriminability, and
guessability. A space of models along five dimensions is considered and
thoroughly evaluated using two medium-sized rating data sets. Only models with
item-level effects for difficulty and rater effects distinguishing sensitivity
and specificity pass the posterior predictive checks [16, 33]. Posterior
predictive checks are the Bayesian equivalent of $\chi^{2}$ goodness-of-fit
tests in regression, which test a “null” of the fitted model against the data
to evaluate whether the data could have reasonably been generated by the model
[14].

A secondary contribution is a parameterization of sensitivity and specificity
on the log odds scale that restricts parameters to a cooperative range by
jointly constraining sensitivity and specificity to lie above the spam line.
Without this constraint, the rating model likelihoods have two modes, one in
which raters are cooperative and one in which prevalence is inverted and the
raters are adversarial. An adversarial rater is one who consistently provides
the wrong rating (i.e., they know the answer and provide the wrong answer).
The spam line is determined by random guessing—spammy annotators have a
sensitivity equal to one minus their specificity [24].

A final contribution is replicable open-source implementations of these models
in Stan [8], which makes them easy to use in R, Python, or Julia.

##  4 A general crowdsourcing model

This section presents the most general crowdsourcing model, then considers how
it may be simplified by removing features (i.e., an ablation study). Removing
features here amounts to pinning or tying parameters, such as assuming all
raters have the same accuracy or all items have the same difficulty.

###  4.1 The rating data

Consider a crowdsourcing problem for which there are $I\in\mathbb{N}$ items to
rate and $J\in\mathbb{N}$ raters. Long-form data accomodates the varying
number of raters per item and the varying number of items per rater. Let
$N\in\mathbb{N}$ be the number of ratings, with
$\textrm{rating}_{n}\in\\{0,1\\}$, each of which has a corresponding
$\textrm{rater}_{n}\in\\{1,\ldots,J\\}$ and item being rated
$\textrm{item}_{n}\in\\{1,\ldots,I\\}$. As shown in Table 1, each row
represents an annotation, with columns indicating indices for the rater and
item as well as the rating.

For each item $i$, we may also have covariates (features, predictors)
$x_{i}\in\mathbb{R}^{K}$. The full data matrix is $x\in\mathbb{R}^{I\times L}$
with $x_{i}$ considered a row vector. In the simplest case, $L=1$, and $x$ is
a single column of 1 values representing an intercept. In the more general
case, other features may represent information relevant for classifying the
item, such as the pixels of an image, demographic information, or medical test
results.

###  4.2 The generative model

rater | item | rating
---|---|---
1 | 1 | 0
1 | 1 | 1
15 | 52 | 1
⋮ | ⋮ | ⋮
Table 1: Rating data table in which each row represents a rating by a rater of
an item.

The probabilistic models we provide are generative in the sense that they
generate parameters $\theta$ describing the process from the prior $p(\theta)$
and generate the observed labels $y$ based on the parameters using the
sampling distribution $p(y\mid\theta)$ (equivalently, the likelihood when
considered a function of $\theta$). Bayesian inference enables us to condition
on observed data $y$ to draw inferences about the parameters $\theta$ through
the posterior, $p(\theta\mid y)$. The generative model combines Dawid and
Skene’s epidemiology model of rater sensitivity and specificity with the item-
response theory educational testing model of item difficulty, discrimination,
and guessing. The categories for the items are generated independently given
the covariates using a logistic regression (such a model is sometimes called
“discriminative” because the covariates are not modeled). The ratings for an
item are also generated conditionally based on the item’s category and the
rater’s abilities and biases; they could also be conditioned on item-level
predictors. In frequentist terms, this section presents a complete data
likelihood for the categories and rating data in which the latent categories
are treated as “missing data” to finesse the philosophical prohibition against
random parameters.

#### Generating categories.

For each item, let $z_{i}\in\\{0,1\\}$ be its (unobserved/latent) category,
with a 1 conventionally denoting “success” or a “positive” result. The
complete data likelihood is complete in the sense that it includes the latent
category. Marginalizing out this category, the technical details and numerical
stability of which are deferred until later, leads to the ordinary likelihood
function used in the model to avoid challenging inference over discrete
parameters. Let $\pi\in(0,1)$ be the parameter representing the prevalence of
positive outcomes. Categories are generated independently given the
prevalence, $z_{i}\sim\textrm{bernoulli}(\pi).$

#### Generating ratings.

The rating from rater $j$ for item $i$ is generated conditionally given the
category $z_{i}$ of an item. For positive items ($z_{i}=1$), sensitivity
(i.e., accuracy on positive items) is used, whereas for negative items
($z_{i}=0$), specificity (i.e., accuracy on negative items) is used. Thus
every rater $j$ will have a sensitivity and specificity
$\alpha^{\textrm{sens}}_{j},\alpha^{\textrm{spec}}_{j}\in\mathbb{R}$ on the
log odds (e.g., $\textrm{logit}^{-1}(\alpha^{\textrm{sens}}_{j})$ is the
sensitivity). If the sensitivity is higher than the specificity there will be
a bias toward 1 ratings, whereas if the specificity is higher than the
sensitivity, there is a bias toward 0 ratings. If the model only has
sensitivity and specificity parameters that vary by rater, it reduces to Dawid
and Skene’s diagnostic testing model [10]. Fixing
$\alpha^{\textrm{sens}}=\alpha^{\textrm{spec}}$ introduces an unbiasedness
assumption whereby a rater has equal sensitivities and specificities.

The items are parameterized with a difficulty $\beta_{i}\in\mathbb{R}$ on the
log odds scale. This difficulty is subtracted from the sensitivity (if
$z_{i}=1$) or specificity (if $z_{i}=0$) as appropriate to give the raw log
odds of a correct rating (i.e., a rating matching the true category $z_{i}$).
Fixing $\beta_{i}=0$ introduces the (typically erroneous) assumption that
every item is equally difficult.

Each item is further parameterized with a positive-constrained discrimination
parameter $\delta_{i}\in(0,\infty)$. This is multiplied by the raw log odds to
give a discrimination-adjusted log odds to give a probability of correctly
rating the item. With high discrimination, it is more likely a rater with
ability greater than the difficulty will get the correct answer and less
likely that a rater with ability less than difficulty will get the correct
answer. For educational testing, high discrimination test questions are
preferable, but for rating wild type data, low discrimination items are common
because of natural variations in the signal (e.g., natural language text or an
image). Fixing $\delta_{i}=1$ introduces the assumption that the items are
equally discriminative.

The final parameter associated with an item is a guessability parameter
$\lambda_{i}\in(0,1)$, giving the probability that a rater can just “guess”
the right answer. The probability that a rater assigns the correct rating will
thus be the combination of the probability of guessing correctly and otherwise
getting the correct answer in the usual way. Fixing $\lambda_{i}=0$ introduces
the assumption that the raters never guess an answer. Without a guessing
parameter, as difficulty goes to infinity, the probability a rater provides
the correct label for an item goes to zero. With guessing, the probability of
a correct label is always at least the probability of guessing.

The full model follows the item-response theory three-parameter logistic
(IRT-3PL) model generalized with sensitivity and specificity, where the
probability that rater $j$ assigns the correct rating to item $i$ is given by

$c_{n}\sim\textrm{bernoulli}\\!\left(\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha^{k}_{j}-\beta_{i}))\right),$
(1)

where $k=\textrm{sens}$ if $z_{i}=1$ and $k=\textrm{spec}$ if $z_{i}=0.$

In order to convert to a distribution over rating, the probability of a 1
outcome must be flipped when $z_{i}=0$ so that a 90% accurate rating results
in a 90% chance of a 0 rating. Thus the rating is given by

$y_{n}\sim\begin{cases}\textrm{bernoulli}\\!\left(\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}\\!\left(\delta_{i}\cdot\left(\alpha^{\textrm{sens}}_{j}-\beta_{i}\right)\right)\right)&\textrm{if
}z_{i}=1,\textrm{ and}\\\\[4.0pt]
\textrm{bernoulli}\\!\left(1-\left(\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}\\!\left(\delta_{i}\cdot\left(\alpha^{\textrm{spec}}_{j}-\beta_{i}\right)\right)\right)\right)&\textrm{if
}z_{i}=0.\end{cases}$ (2)

The second case ($z_{i}=0$) reduces to
$\textrm{bernoulli}\\!\left(\left(1-\lambda_{i}\right)\cdot\left(1-\textrm{logit}^{-1}\\!\left(\delta_{i}\cdot\left(\alpha^{\textrm{sens}}_{j}-\beta_{i}\right)\right)\right)\right).$

#### Parameter priors

We further complete the IRT-3PL model by placing appropriate priors on the
parameters as follows. We use weakly informative priors which constrain the
scale of the result,

$\pi\sim\text{Beta}(2,2),\quad\alpha^{\textrm{spec}}_{j}\sim\text{Normal}(2,2),\quad\alpha^{\textrm{spec}}_{j}\sim\text{Normal}(1,2),$
$\beta_{i}\sim\text{Normal}(0,1),\quad\delta_{i}\sim\text{LogNormal}(0,0.25),\quad\lambda_{i}\sim\text{Beta}(2,2).$

When we fix $\alpha^{\text{sens}}=\alpha^{\text{spec}}$, we assume
unbiasedness where the rater has equal sensitivities and specificities,
leading us to model the accuracy parameter $\alpha^{\text{acc}}$. Under this
assumption, $\alpha^{\text{acc}}\sim\text{Normal}(1,2)$ acts as a rater
accuracy.

#### Spammy and adversarial raters.

A spammy rater is one for whom the rating does not depend on the item being
rated, which arises when $\alpha^{\textrm{sens}}=-\alpha^{\textrm{spec}}$, so
that
$\textrm{logit}^{-1}(\alpha^{\textrm{sens}})=1-\textrm{logit}^{-1}(\alpha^{\textrm{spec}})$
[24]. For example, a rater with 40% sensitivity and 60% specificity has a 40%
chance of returning a 1 rating no matter what the true category is.

An adversarial rater is one for which
$\alpha^{\textrm{sens}}<-\alpha^{\textrm{spec}}$, and we have restricted all
models to enforce the constraint that
$\alpha^{\textrm{sens}}>-\alpha^{\textrm{spec}}$. In contrast, a rater with
60% sensitivity and 60% specificity provides informative ratings, whereas one
with 40% sensitivity and 40% specificity provides adversarial ratings which
are more likely to be wrong than correct.

#### Including covariate information

If item-level or rater-level covariates are available, they may be used to
inform the parameters through a regression in the form of a generalized linear
model. For example, suppose there are item-level covariates
$x_{i}\in\mathbb{R}^{K}$. With a parameter vector $\gamma\in\mathbb{R}^{K}$,
the generative model for the category of an item may be extended to a logistic
regression,
$z_{i}\sim\textrm{bernoulli}\\!\left(x_{i}^{\top}\cdot\gamma\right).$ An
intercept for the regression may be added by including a column of 1 values in
$x$. The resulting coefficients will act as a classifier for new items by
generating a covariate-specific prevalence. This motivated Raykar et al.’s
combination of Dawid and Skene’s rating model and logistic regression of this
form [30].

Item-level covariates can be used to generate the item-level parameters of
difficulty, discrimination, and guessability. For example, a covariate that
indicates the number of options in a multiple choice test would inform the
guessability parameter. If the grade level of the textbook from which a
problem was culled was included as a predictor, that could be used to inform
the difficulty parameter. If there are rater level covariates
$u_{i}\in\mathbb{R}^{L}$, these may be used in the same way to generate the
ability parameters for the raters. There might also be an indicator of the
rater was an undergraduate, graduate student, faculty member, or crowdsourced
external agent, which could be used to inform ability.

###  4.3 Marginal likelihood

The model generates latent discrete categories $z_{i}\in\\{0,1\\}$ for each
item. For both optimization and sampling, it is convenient to marginalize the
complete likelihood $p(y,z\mid\pi,\alpha,\beta,\delta,\lambda)$ to the rating
likelihood $p(y\mid\pi,\alpha,\beta,\delta,\lambda)$. The marginalization
calculation is efficient because it is factored by data item. Letting
$\theta=\pi,\alpha,\beta,\delta,\lambda$ be the full set of continuous
parameters, the trick is to rearrange the long-form data by item, then
marginalizing out, resulting in the likelihood (see Appendix A).

$\textstyle
p(y\mid\theta)=\prod_{i=i}^{I}\sum_{z_{i}=0}^{1}\,p(z_{i}\mid\theta)\cdot\prod_{n:\textrm{item}_{n}=i}p(y_{n}\mid
z_{i},\theta).$ (3)

###  4.4 Model reductions

Reduction | Description | Tag
---|---|---
$\lambda_{i}=0$ | no guessing items | A
$\delta_{i}=1$ | equal discrimination items | B
$\beta_{i}=0$ | equal difficulty items | C
$\alpha^{\text{spec}}=\alpha^{\text{sens}}$ | equal error raters | D
$\alpha_{i}=\alpha^{\prime}_{j}$ | identical raters | E
Table 2: Possible model reductions.

By tying or fixing parameters, the full model may be reduced to define a wide
range of natural submodels. Six of these models correspond to item-response
theory models of the one-, two-, and three-parameter logistic variety, either
with or without a sensitivity/specificity distinction. The model with varying
rater sensitivity and specificity and no item effects reduces to Dawid and
Skene’s model. Other models, such as the model with a single item effect and
no rater effects have been studied in the epidemiology literature.

Table 2 summarizes the possible model reductions and gives them identifying
tags. First, we consider models which do not distinguish sensitivity and
specificity. Such models should be used when the categories are not
intrinsically ordered (e.g., rating two consumer brands for preference).

$\begin{array}[]{ccl}\textrm{Reductions}&\textrm{Probability
Correct}&\textrm{Note}\\\ \hline\cr
D&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha_{j}-\beta_{i}))&\textrm{\small
IRT 3PL}\\\
CD&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot\alpha_{j})\\\
BD&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha_{j}-\beta_{i})&\textrm{\small
IRT 2PL}\\\ \hline\cr
BCD&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha_{j})\\\
AD&\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha_{j}-\beta_{i}))\\\
ACD&\textrm{logit}^{-1}(\delta_{i}\cdot\alpha_{j})\\\ \hline\cr
ABD&\textrm{logit}^{-1}(\alpha_{j}-\beta_{i})&\textrm{\small IRT 1PL}\\\
ABCD&\textrm{logit}^{-1}(\alpha_{j})\\\
ABCDE&\textrm{logit}^{-1}(\alpha)\end{array}$ (4)

All of these models other than the last (ABCDE) assumes raters have varying
accuracy. The following models introduce separate parameters for sensitivity
and specificity rather than assuming they are the same.

$\begin{array}[]{ccl}\textrm{Reductions}&\textrm{Probability
Correct}&\textrm{Note}\\\
\hline\cr&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha^{k}_{j}-\beta_{i}))&\textrm{\small
IRT 3PL + sens/spec}\\\
C&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot\alpha^{k}_{j})\\\
BC&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha^{k}_{j})\\\
\hline\cr
A&\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha^{k}_{j}-\beta_{i}))&\textrm{\small
IRT 2PL + sens/spec}\\\
AC&\textrm{logit}^{-1}(\delta_{i}\cdot\alpha^{k}_{j})\\\
AB&\textrm{logit}^{-1}(\alpha^{k}_{j}-\beta_{i})&\textrm{\small IRT 1PL +
sens/spec}\\\ \hline\cr ABC&\textrm{logit}^{-1}(\alpha^{k}_{j})&\textrm{\small
Dawid and Skene}\\\ ABCE&\textrm{logit}^{-1}(\alpha^{k})\end{array}$ (5)

As with the first list, only the last model (ABCE) does not distinguish rater
abilities. There is one last model, which includes item effects without any
rater effects.

$\begin{array}[]{cc}\textrm{Reductions}&\textrm{Probability Correct}\\\
\hline\cr ABDE&\textrm{logit}^{-1}(-\beta_{i})\\\ \hline\cr\end{array}$ (6)

The remaining thirteen models are redundant in the sense that fixing their
non-identifiability issues reduces to a model with a single item effect; see
Appendix B for the full list of reductions.

##  5 Training on probabilistic data

Figure 2: L2 norm of training error. The L2 norm of parameter estimation
error, $||\widehat{\theta}-\theta||_{2}$, for different approaches to training
with probabilities: (log odds) training a linear regression on the log odds,
(max prob) assigning the highest probability category, (noisy odds) add
standard normal noise to the log odds approach, (random) assign a random
category according to the probability, (weighted) train a weighted logistic
regression. The errors are consistent whether using Bayesian posterior means
with a normal prior (left) or ridge-penalized maximum likelihood estimates
(right). Regression is 32-dimensional, with correlated inputs and 1024
training data points. Results show standard bar-and-whisker plots over 32
trials with paired random $x,\beta$.

As part of their likelihood calculation, the rating models compute the
probability that each item in the rating set is either category 0 or category
1. It is far better to use this probabilistic information directly when
training a classifier than to collapse the uncertainty to either a 0 or 1
“gold standard” category. Using probabilities acts as a form of
regularization—if an item has probability 0.6329, training will try to avoid
attenuated results for this item with probabilities near 1 or 0.

Consider a classification data set where for each data item $n,$ there is a
covariate (feature) vector $x_{n}\in\mathbb{R}^{L}$ and a binary outcome
$y_{n}\in\\{0,1\\}.$ The standard logistic regression likelihood takes a
parameter (coefficient, weight) vector $\beta\in\mathbb{R}^{L}$ with
$y_{n}\sim\textrm{bernoulli}\\!\left(\textrm{logit}^{-1}(x_{n}\cdot\beta)\right).$

Now suppose that we have fit a rating model and do not actually know $y_{n}$
but only have an estimate of $\Pr[y_{n}=1]$. The conventional strategy in
machine-learning data curation is to take the “best” category, often the
result of a majority vote. More generally, majority voting is equivalent to
taking the highest probability category, $y_{i}=1$ if
$\Pr[y_{i}=1]>\frac{1}{2}$, in a rating model with equal rater accuracies and
no item effects (i.e., the model tagged ABCDE).

For the experiment, we generate a synthetic data set of covariates
$x_{i}\sim\textrm{normal}(0,\Sigma)$, where the positive definite covariance
matrix is defined by $\Sigma_{m,n}=\rho^{|m-n|}$, with $\rho=0.9$. The result
is highly correlated predictors; uncorrelated predictors show the same trend.
We evaluate five approaches to estimation: (max prob) take $y_{n}$ to be 1 if
$\textrm{logit}^{-1}(x_{n}\cdot\beta)>\frac{1}{2},$ (log odds) train a linear
regression with outcome $\log x_{n}\cdot\beta,$ (noisy odds) log odds with
standard normal noise, $\textrm{logit}^{-1}(x_{n}\cdot\beta+\epsilon_{n})$,
with $\epsilon_{n}\sim\textrm{normal}(0,1),$ (random) randomly generate
$y_{n}\sim\textrm{bernoulli}\\!\left(\textrm{logit}^{-1}(x_{n}\cdot\beta)\right)$
according to its probability distribution, and (weighted) train a weighted
logistic regression with outcome 1 and weight
$\textrm{logit}^{-1}(x_{n}\cdot\beta)$ and outcome 0 and weight
$1-\textrm{logit}^{-1}(x_{n}\cdot\beta).$

We provide both Bayesian (posterior mean) and frequentist (penalized maximum
likelihood) estimates. For the Bayesian setting, we use a standard normal
prior $\beta_{k}\sim\textrm{normal}(0,1)$ and in the frequentist setting we
use ridge regression, with penalty function
$\frac{1}{2}\cdot\beta^{\top}\cdot\beta$ to match the Bayesian prior. The
resulting error norms in estimating $\beta$ are shown in Figure 2.

It is clear from the plot that training with the probabilities is much better
than taking the category with the highest probability. The best approach is
training a linear regression based on the log odds, with the noisy version of
the same approach, which matches what we might get from a rating model, not
far behind. Weighted training with logistic regression is almost as good.
Randomly selecting a category according to the generative model is not as good
as using the weights directly, but it still dominates taking the “best”
category.

##  6 Empirical evaluations

The posteriors of all 18 distinct models introduced above were sampled using
Markov chain Monte Carlo (MCMC) for two data sets. The first data set consists
of 5 dentists rating each of roughly 4000 dental X-rays for caries (a kind of
pre-cavity) [12]. The second is nearly 200 Mechanical Turkers, each rating a
subset of roughly 3000 pairs of sentences for entailment [37].

###  6.1 Dentistry data

Model | Rater p-value | Ratings p-value
---|---|---
ABCDE | < 0.001 | < 0.001
ABCD | < 0.001 | < 0.001
ABCE | < 0.001 | 0.019
ABDE | < 0.001 | < 0.001
ABC | 0.462 | < 0.001
ABD | 0.074 | < 0.001
ACD | < 0.001 | < 0.001
BCD | < 0.001 | < 0.001
AB | 0.468 | 0.218
AC | 0.405 | < 0.001
AD | 0.325 | < 0.001
BC | 0.102 | 0.001
BD | < 0.001 | < 0.001
CD | < 0.001 | < 0.001
A | 0.046 | 0.289
C | 0.014 | 0.001
D | < 0.001 | < 0.001
full | 0.120 | 0.01
Table 3: Posterior predictive p-values for 18 different models.

The models were coded in Stan (version 2.33) and fit with default sampling
settings using CmdStanPy (version 1.20). The default sampler is the
multinomial no-U-turn sampler, an adaptive form of Hamiltonian Monte Carlo [6,
17] that adapts a diagonal mass matrix. The default number of chains is four,
and the default runs 1000 warmup iterations (for burn-in and adaptation) and
1000 sampling iterations. All sampling runs ended with split-$\widehat{R}$
values less than 1.01 for all parameters (prevalence, rater parameters, and
item parameters), indicating consistency with convergence to approximate
stationarity [15].

Table 3 shows the posterior predictive $p$-values for all of the models on the
dentistry data using statistics for marginal positive votes per rater and per
item. The posterior predictive $p$-values assess the consistency between the
observed data and the model-simulated ratings and votes, providing a measure
of the model’s fit. Only three models pass the posterior predictive checks,
all of which are IRT models with difficulty (AB) and optionally discrimination
(A) and guessing (full). It is important here to note that the widely used
Dawid and Skene model (ABC) fails the posterior predictive checks.

###  6.2 Mechanical Turker data

Figure 3: Distribution of positive votes per item comparing the baseline Dawid
& Skene model (ABC) with IRT models with difficulty (AB), discrimination (A),
and guessing (full) alongside actual data, demonstrating the varying levels of
dispersion captured by each model.

To demonstrate how posterior predictions underestimate dispersion, we compare
Dawid and Skene’s baseline (ABC) with three item-level models: difficulty
(AB), plus discrimination (A), and plus guessing (full). The goodness-of-fit
tests are similar, so in Figure 3, we plot the expected number of marginal
positive votes for models A, AB, ABC, and full along with the actual data. The
Dawid and Skene model (ABC) has an inflated number of middling positive votes
compared to the data; in contrast the full model underestimates them. In
contrast, posterior predictive inference for models A (ability plus difficulty
plus discrimination) and AB (ability plus difficulty) more closely matches the
actual data. This alignment enhances the model’s ability to reflect true voter
consensus, thereby producing a distribution of votes that more accurately
mirrors observed voting patterns.

##  7 Extensions

A natural extension is to $K$-way categorical ratings, such as classifying a
dog image by species, classifying an article in a newspaper by topic, rating a
movie on a one to five scale, classifying a doctor’s visit with an ICD-10
code, and so on. Most of the work on ratings has been in this more general
categorical setting. With more than two categories, sensitivity and
specificity are replaced with categorical responses based on the latent true
category. Discrimination and guessing act the same way, but difficulty must be
replaced with a more general notion of a categorical item level effect, which
may represent either focused alternatives (e.g., a border collie is confusable
with an Irish shepherd) or diffuse (e.g., can’t tell what’s in the image).

With enough raters, these models may also be extended hierarchically to make
population-level inferences about the distribution of rater abilities or item
difficulties [26]. Several of the crowdsourcing tasks may be combined to
select raters and items to rate online with active learning, which is a form
of reinforcement learning. With a hierarchical model, inference may be
expanded to new raters [26].

It is also straightforward to extend a rating model to ordered responses such
as Likert scales [18, 31, 35], rank ordering [9, 28], counts (the “textbook”
case of crowdsourcing is estimating the number of jelly beans in a container
[38]), proportions/probabilities, distances, or pairs of real numbers such as
planetary locations [36]. All that needs to change is the response model and
the representation of the latent truth—the idea of getting noisy ratings and
inferring a ground truth remains. As an example, Smyth had raters mark images
of Venus for volcano locations [36]. The true location is represented as a
latitude and longitude and rater responses can be multivariate normal centered
around the true, but unknown, location. For ordinal ratings, an ordinal logit
model of the truth may be used [31]. For comparisons, the Bradley-Terry model
can be used [7], and for ranking, the Plackett-Luce generalization [27, 21],
as used in direct preference optimization for fine-tuning large language
models [28].

##  8 Conclusion

Dawid and Skene’s model has two modes, an adversarial and cooperative one.
Adversarial solutions arise when the majority of raters have below-chance
accuracy. We showed how to parameterize the models to ensure a cooperative
solution. In crowdsourcing, it is more common to find random (aka “spammy”)
annotators than adversarial ones; constraining the models to a cooperative
solution identifies the model, but should not be used if the data contains a
mix of adversarial and cooperative raters [24]. Dealing with majority
adversarial raters is an open problem.

Until this work, models used for crowdsourcing categorical responses, such as
Dawid and Skene’s [10], produced biased results. We demonstrated this bias
through failed goodness-of-fit tests, where Dawid and Skene’s model predicts
too little inter-annotator agreement relative to the actual ratings observed
in the wild. The IRT-based models, by introducing item-level effects, are able
to correct for item difficulty, discrimination and guessability, resulting in
fits that pass our goodness-of-fit tests for both items and ratings.

Models of annotation introduce significant improvements compared to the
conventional approach of evaluating inter-annotator agreement statistics such
as Cohen’s $\kappa$ (see Paun et al. for an overview [25]). They may also be
used to diagnose rater biases, infer adjusted population prevalence, or select
highly discriminative items, and so on. We also showed how using uncertainty
of the form generated by crowdsourcing models, either as weights or to
directly train or to sample, improves classifier training [30]. As a result,
most training efforts in machine learning are not making the most of their
crowdsourced data.

## References

  * [1] ACT, Inc.  ACT [American College Testing] technical manual.  Technical report, ACT, Inc., 2024.
  * [2] Paul S Albert and Lori E Dodd.  A cautionary note on the robustness of latent class models for estimating diagnostic error without a gold standard.  Biometrics, 60(2):427–435, 2004.
  * [3] Ron Artstein and Massimo Poesio.  Inter-coder agreement for computational linguistics.  Computational Linguistics, 34(4):555–596, 2008.
  * [4] William H Batchelder and A Kimball Romney.  Test theory without an answer key.  Psychometrika, 53(1):71–92, 1988.
  * [5] Beata Beigman Klebanov and Eyal Beigman.  From annotator agreement to noise models.  Computational Linguistics, 35(4):495–503, 2009.
  * [6] Michael Betancourt.  A conceptual introduction to Hamiltonian Monte Carlo.  arXiv preprint arXiv:1701.02434, 2017.
  * [7] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [8] Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus A Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell.  Stan: A probabilistic programming language.  Journal of Statistical Software, 76, 2017.
  * [9] Xi Chen, Paul N Bennett, Kevyn Collins-Thompson, and Eric Horvitz.  Pairwise ranking aggregation in a crowdsourced setting.  In Proceedings of the sixth ACM international conference on Web search and data mining, pages 193–202, 2013.
  * [10] Alexander Philip Dawid and Allan M Skene.  Maximum likelihood estimation of observer error-rates using the EM algorithm.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979.
  * [11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255, 2009.
  * [12] Mark A Espeland and Stanley L Handelman.  Using latent class models to characterize and assess relative error in discrete measurements.  Biometrics, pages 587–599, 1989.
  * [13] Kevin S Flannagan, Brian M Mills, and Robert L Goldstone.  The psychophysics of home plate umpire calls.  Scientific Reports, 14(1):2735, 2024.
  * [14] Anton K Formann.  Latent class model diagnostics—a review and some proposals.  Computational Statistics & Data Analysis, 41(3-4):549–559, 2003.
  * [15] Andrew Gelman, John B Carlin, Hal S Stern, David B Dunson, Aki Vehtari, and Donald B Rubin.  Bayesian Data Analysis.  Chapman and Hall/CRC, Third edition, 2013.
  * [16] Andrew Gelman, Xiao-Li Meng, and Hal Stern.  Posterior predictive assessment of model fitness via realized discrepancies.  Statistica Sinica, pages 733–760, 1996.
  * [17] Matthew D Hoffman, Andrew Gelman, et al.  The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo.  Journal of Machine Learning Research, 15(1):1593–1623, 2014.
  * [18] Balaji Lakshminarayanan and Yee Whye Teh.  Inferring ground truth from multi-annotator ordinal data: a probabilistic approach.  arXiv, 1305.0015, 2013.
  * [19] Paul F Lazarsfeld.  The logical and mathematical foundation of latent structure analysis.  In S. A. Stouffer, L. Guttman, A. Suchman, P.F. Lazarsfeld, A. Star, and J. A. Clausen, editors, Measurement and Prediction, volume IV of Studies in Social Psychology in World War II, pages 362–412. Princeton University Press, 1950.
  * [20] FM Lord, MR Novick, and Allan Birnbaum.  Statistical theories of mental test scores.  Addison-Wesley, 1968.
  * [21] R Duncan Luce.  Individual choice behavior: A theoretical analysis.  Courier Corporation, 1959.
  * [22] Mary L McHugh.  Interrater reliability: the kappa statistic.  Biochemia medica, 22(3):276–282, 2012.
  * [23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Keton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  * [24] Rebecca J Passonneau and Bob Carpenter.  The benefits of a model of annotation.  Transactions of the Association for Computational Linguistics, 2:311–326, 2014.
  * [25] Silviu Paun, Ron Artstein, and Massimo Poesio.  Statistical methods for annotation analysis.  Springer Nature, 2022.
  * [26] Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio.  Comparing Bayesian models of annotation.  Transactions of the Association for Computational Linguistics, 6:571–585, 2018.
  * [27] Robin L Plackett.  The analysis of permutations.  Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193–202, 1975.
  * [28] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  Advances in Neural Information Processing Systems, 36, 2024.
  * [29] Georg Rasch.  Studies in Mathematical Psychology: I. Probabilistic Models for Some Intelligence and Attainment Tests.  Nielsen & Lydiche, 1960.
  * [30] Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy.  Learning from crowds.  Journal of machine learning research, 11(4), 2010.
  * [31] Simon Rogers, Mark Girolami, and Tamara Polajnar.  Semi-parametric analysis of multi-rater data.  Statistics and Computing, 20:317–334, 2010.
  * [32] A Kimball Romney, Susan C Weller, and William H Batchelder.  Culture as consensus: A theory of culture and informant accuracy.  American Anthropologist, 88(2):313–338, 1986.
  * [33] Donald B Rubin.  Bayesianly justifiable and relevant frequency calculations for the applied statistician.  The Annals of Statistics, pages 1151–1172, 1984.
  * [34] Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl.  Corpus annotation through crowdsourcing: Towards best practice guidelines.  In The International Conference on Language Resources and Evaluation, pages 859–866, 2014.
  * [35] Hagit Shatkay, Andrey Rzhetsky, and W John Wilbur.  Searching for high-utility text in the biomedical literature.  BioLINK SIG: Linking Literature, Information and Knowledge for Biology, 2005.
  * [36] Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi.  Inferring ground truth from subjective labelling of Venus images.  Advances in neural information processing systems, 7, 1994.
  * [37] Rion Snow, Brendan O’connor, Dan Jurafsky, and Andrew Y Ng.  Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks.  In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254–263, 2008.
  * [38] James Surowiecki.  The wisdom of crowds.  Anchor, 2005.

##  Appendix A Marginalizing out latent categorization on the log scale

We start with the complete data likelihood for ratings $y$ and latent
categories $z$,

$p(y,z\mid\theta)=\prod_{i=1}^{I}p(z_{i}\mid\theta)\cdot\prod_{n=1}^{N}p(y_{n}\mid
z,\theta)$ (7)

and then rearrange terms by item,

$np(y,z\mid\theta)=\prod_{i=i}^{I}\left(p(z_{i}\mid\theta)\cdot\prod_{n:\textrm{item}_{n}=i}p(y_{n}\mid
z_{i},\theta)\right).$ (8)

On a per item basis, the marginalization is tractable, yielding Equation 3.

Computational inference requires a log likelihood. The log marginal likelihood
of the rating data is

$\begin{array}[]{rcl}\log
p(y\mid\theta)&=&\log\prod_{i=i}^{I}\sum_{z_{i}=0}^{1}\,p(z_{i}\mid\theta)\cdot\prod_{n:\textrm{item}_{n}=i}p(y_{n}\mid
z_{i},\theta).\\\\[4.0pt]
&=&\sum_{i=1}^{I}\textrm{logSumExp}_{z_{i}=0}^{1}\,\left(\log
p(z_{i}\mid\theta)+\sum_{n:\textrm{item}_{n}=i}\log p(y_{n}\mid
z_{i},\theta)\right),\end{array}$ (9)

where

$\textrm{logSumExp}_{n=1}^{N}\,\ell_{n}=\log\sum_{n=1}^{N}\exp(\ell_{n})$ (10)

is the numerically stable log-scale analogue of addition.

##  Appendix B Redundant parameter models

The following model likelihoods all involve redundant parameterizations, that
when identified, reduce to one of the existing models introduced in Section
4.4.

$\begin{array}[]{ccc}\textrm{Reductions}&\textrm{Probability Correct}\\\
\hline\cr
E&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha^{k}-\beta_{i}))\\\
DE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha-\beta_{i}))\\\
CE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot\alpha^{k})\\\
\hline\cr
CDE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\delta_{i}\cdot\alpha)\\\
BE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha^{k}-\beta_{i})\\\
BDE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha-\beta_{i})\\\
\hline\cr
BCE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha^{k})\\\
BCDE&\lambda_{i}+(1-\lambda_{i})\cdot\textrm{logit}^{-1}(\alpha)\\\
AE&\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha^{k}-\beta_{i}))\\\ \hline\cr
ADE&\textrm{logit}^{-1}(\delta_{i}\cdot(\alpha-\beta_{i}))\\\
ACE&\textrm{logit}^{-1}(\delta_{i}\cdot\alpha^{k})\\\
ACDE&\textrm{logit}^{-1}(\delta_{i}\cdot\alpha)\\\ \hline\cr
ABE&\textrm{logit}^{-1}(\alpha^{k}-\beta_{i})\end{array}$ (11)