# Harmless label noise and informative soft-labels in supervised
classification

Daniel Ahfock<EMAIL_ADDRESS>Geoffrey J. McLachlan

###### Abstract

Manual labelling of training examples is common practice in supervised
learning. When the labelling task is of non-trivial difficulty, the supplied
labels may not be equal to the ground-truth labels, and label noise is
introduced into the training dataset. If the manual annotation is carried out
by multiple experts, the same training example can be given different class
assignments by different experts, which is indicative of label noise. In the
framework of model-based classification, a simple, but key observation is that
when the manual labels are sampled using the posterior probabilities of class
membership, the noisy labels are as valuable as the ground-truth labels in
terms of statistical information. A relaxation of this process is a random
effects model for imperfect labelling by a group that uses approximate
posterior probabilities of class membership. The relative efficiency of
logistic regression using the noisy labels compared to logistic regression
using the ground-truth labels can then be derived. The main finding is that
logistic regression can be robust to label noise when label noise and
classification difficulty are positively correlated. In particular, when
classification difficulty is the only source of label errors, multiple sets of
noisy labels can supply more information for the estimation of a
classification rule compared to the single set of ground-truth labels.

##  1 Introduction

Many supervised learning algorithms operate on the assumption that the
training set labels are the ground-truth labels. In practice, ground-truth
labels may not be readily obtainable, and manual annotation is used to
construct the training dataset (Frénay and Verleysen, 2014). In medical
applications, clinicians will often classify patients into different groups on
the basis of preliminary examinations. In machine learning, it is convenient
to crowdsource labels for image and speech recognition tasks through an online
platform. Due to the subjective nature of the process, and the inherent
difficulty of classifying some observations, the manually collected labels may
not be equal to the ground-truth labels. This mismatch is often referred to as
label noise, and this phenomenon can have interesting statistical implications
(McLachlan, 1972; Bouveyron et al., 2019; Cannings et al., 2020). A primary
concern is the robustness of an estimated classification rule with respect to
label noise (Cappozzo et al., 2019; Vranckx et al., 2021). A closely related
issue is that when each member of a group of experts provides a class
assignment, the agreement is not necessarily unanimous. The situation when
there is heterogeneity amongst the supplied labels is referred to as soft-
labelling, as there is no definitive class assignment for each feature vector
(Quost et al., 2017). Extracting the maximum amount of information from
conflicting label sets is a challenging task in statistical machine learning
(Dawid and Skene, 1979; Smyth et al., 1995; Raykar et al., 2009; Yan et al.,
2010).

As motivating examples, Figure 1 shows histograms of vote counts for three
different binary classification datasets that have been manually labelled.
Each dataset contains $n$ entities in the training set that have each been
classified by $m$ individuals in a group (Mesejo et al., 2016; Welinder et
al., 2010; Ipeirotis et al., 2010). The $x$-axis represents the total number
of votes for the positive class from the $m$ individuals in the labelling
group. The $y$-axis represents the number of entities that have received a
certain number of positive votes. If there were unanimous group agreement on
the classification of an entity, there would either be $m$ positive votes in
total, or 0 positive votes in total. For the Gastroentology dataset in panel
(a), there were 32 entities which were classified as positive by all $m=7$
members in the labelling group. For the other type of unanimous agreement,
there were 4 entities that were classified as positive by $0$ members in the
labelling group. In each dataset there are entities where there is no strong
consensus, which may reflect uncertainty over the ground-truth label.

Figure 1: Vote counts for three binary classification datasets that have been
manually labelled. The $x$-axis represents the total number of votes for the
positive class from the $m$ individuals in the labelling group. The $y$-axis
represents the number of entities that have received a certain number of
positive votes. Unanimous group agreement implies $m$ votes for the positive
class or $0$ votes for the positive class. (a) Gastroentology dataset
$(n=76,m=7)$ (Mesejo et al., 2016). $m=7$ clinicians classify $n=76$
colnoscopic videos into malignant and benign classes . (b) Bird dataset
$(n=108,m=39)$ (Welinder et al., 2010). $m=39$ individuals report whether
$n=108$ images contain a bird or not. (c) Adult content dataset
$(n=496,m=100)$ (Ipeirotis et al., 2010). $m=100$ individuals report whether
$n=496$ websites contain adult material or not.

In the framework of model-based classification, the ground-truth labels are
commonly treated as latent variables in a finite-mixture model (Fraley and
Raftery, 2002; McLachlan and Peel, 2000). The data generating process for a
$g$-component mixture model can be represented as a hierarchical model,

$\displaystyle\bm{Z}$ $\displaystyle\sim\text{Multinomial}(1,\bm{\pi}),$
$\displaystyle\bm{Y}\mid\bm{Z}_{i}=1$ $\displaystyle\sim
f(\bm{y};\bm{\omega}_{i}),$

where $\bm{\pi}=(\pi_{1},\ldots,\pi_{g})^{\mathsf{T}}$ gives the mixing
proportions and
$\bm{\omega}=(\bm{\omega}_{1}^{\mathsf{T}},\ldots,\bm{\omega}_{g}^{\mathsf{T}})^{\mathsf{T}}$
is a vector of component specific parameters. The latent variable $\bm{Z}$ is
considered the ground-truth label as the feature vector $\bm{Y}$ is then
sampled from the $i$th class conditional distribution
$f(\bm{y};\bm{\omega}_{i})$ given that $\bm{Z}_{i}=1$ $(i=1,\ldots,g)$. We
propose to treat the manual label $\bm{Z}^{\prime}$ as a random variable to
model the manual labelling process. Label noise occurs when
$\bm{Z}^{{}^{\prime}}\neq\bm{Z}$.

We develop a probabilistic model for manual annotation where label noise is
positively associated with classification difficulty. Using this model, we
analyse the relative value of noisy manual labels compared to the ground-truth
labels in the context of logistic regression. As a starting point, Section 2.2
highlights that when the manual labels $\bm{Z}^{\prime}$ are sampled according
to the posterior probabilities of class membership, the use of noisy labels
can be as efficient as the use of the ground-truth labels $\bm{Z}$. In Section
2.3 we introduce a random-effects model for group labelling to account for
imperfect knowledge of the posterior class-probabilities. In Section 3 we
derive the asymptotic relative efficiency of logistic regression using the
noisy labels to logistic regression using the ground-truth labels. The key
finding is that multiple sets of noisy labels can supply more information for
the estimation of a classification rule relative to the single set of ground-
truth labels. Section 4 presents simulation results regarding the asymptotic
relative efficiency. In Section 5 we assess the proposed random effects model
on the Gastroentology dataset introduced in Figure (1) (a) and present the
results of experiments on the Wisconsin breast cancer dataset. Finally,
conclusions and directions for future work are given in Section 6.

##  2 Manual labelling

###  2.1 Prior work

A comprehensive survey of label noise in supervised learning is given in
Frénay and Verleysen (2014). Early statistical methods for manual labels
treated the ground-truth label as a latent variable to be estimated on the
basis of the observed noisy labels (Dawid and Skene, 1979; Smyth et al.,
1995). More recently, there has been a focus on constructing generative models
for the noisy labels in order train a classifier (Jin and Ghahramani, 2003;
Bouveyron and Girard, 2009; Yan et al., 2010; Raykar et al., 2009). In a
related branch of work on the analysis of crowdsourced labels, methods have
been developed for filtering out bad actors who deliberately mislabel
instances, and ranking annotators when domain expertise varies in the group
(Raykar and Yu, 2012; Hovy et al., 2013; Zhang et al., 2013). We will assume
that the annotators are of comparable skill level and none act maliciously.
This assumption will likely be violated in crowdsourcing applications with low
barriers to entry, but is more plausible when manual labels are collected from
qualified experts in a medical study.

Prior work on modelling group labelling typically makes the assumption that
label noise occurs uniformly within a class (Dawid and Skene, 1979; Raykar et
al., 2009; Jin and Ghahramani, 2003; Donmez et al., 2010; Song et al., 2020).
Feature dependent label noise is considered in Yan et al. (2010) to allow for
different experts in the group to specialise in separate areas of the feature
space. The behaviour of logistic regression has been studied under the
assumption of fixed class-conditional label noise rates. In Michalek and
Tripathi (1980) and Bi and Jeske (2010) it is demonstrated that with fixed
class-conditional noise, logistic regression is not necessarily consistent.
The increase in the expected error rate due to the noise is also derived.
Under class-conditional uniform noise, Song et al. (2020) show that the noisy
labels can be treated as the response in a modified generalized linear model
and compare the relative value of the noisy labels to the ground-truth labels.

The idea that label noise is more concentrated around classification decision
boundaries has been explored in the context of robust estimation (Xu et al.,
2006; Rebbapragada and Brodley, 2007; Blanchard et al., 2016). However, there
appears to be little statistical work modelling this phenomenon and
quantifying the information loss relative to the ground-truth labels. In
practice, information loss, manual adulteration, and data-entry errors can
introduce label noise into the dataset. In order to simplify the theoretical
analysis we make the assumption that only contributing factor to label errors
is classification difficulty as measured by the posterior probabilities of
class membership.

###  2.2 Label noise

Here we show that label noise does not necessarily lead to a loss in
statistical information. The mixing proportions
$\bm{\pi}=(\pi_{1},\ldots,\pi_{g})^{\mathsf{T}}$ and component specific
parameters
$\bm{\omega}=(\bm{\omega}_{1}^{\mathsf{T}},\ldots,\bm{\omega}_{g}^{\mathsf{T}})^{\mathsf{T}}$
can be put together into the single parameter
$\bm{\Psi}=(\bm{\pi}^{\mathsf{T}},\bm{\omega}^{\mathsf{T}})^{\mathsf{T}}\in\bm{\Omega}$.
We assume that the parameter space $\bm{\Omega}$ is such that the mixture
model is identifiable, suitable conditions for this to hold are discussed in
Cheng and Liu (2001). Let $\tau_{i}(\bm{y};\bm{\Psi})$ represent the posterior
probability of membership in the $i$th class

$\displaystyle\tau_{i}(\bm{y};\bm{\Psi})$
$\displaystyle=\dfrac{\pi_{i}f(\bm{y};\bm{\omega}_{i})}{\sum_{h=1}^{g}\pi_{h}f(\bm{y};\bm{\omega}_{h})}\quad(i=1,\ldots,g),$
(1)

and
$\bm{\tau}(\bm{y};\bm{\Psi})=(\tau_{1}(\bm{y};\bm{\Psi}),\ldots,\tau_{g}(\bm{y};\bm{\Psi}))^{\mathsf{T}}$
give the vector of $g$ posterior class-probabilities. To model the subjective
nature of manual labelling, suppose the annotated label $\bm{Z}^{\prime}$ is
distributed as a multinomial random variable according to the posterior
probabilities of class membership (1),

$\displaystyle\bm{Z}^{\prime}\mid\bm{Y}=\bm{y}$
$\displaystyle\sim\text{Multinomial}(1,\bm{\tau}(\bm{y};\bm{\Psi})).$ (2)

Under this labelling model, the probability of a labelling error is dependent
on the posterior class-probabilities $\bm{\tau}(\bm{y};\bm{\Psi})$.
Specifically,

$\displaystyle\text{pr}(\bm{Z}^{\prime}\neq\bm{Z}\mid\bm{Y}=\bm{y})$
$\displaystyle=1-\text{pr}(\bm{Z}^{\prime}=\bm{Z}\mid\bm{Y}=\bm{y})$
$\displaystyle=1-\sum_{i=1}^{g}\text{pr}(Z_{i}^{\prime}=1,Z_{i}=1\mid\bm{Y}=\bm{y})$
$\displaystyle=1-\sum_{i=1}^{g}\\{\tau_{i}(\bm{y};\bm{\Psi})\\}^{2}.$

Broadly speaking, the greater the classification difficulty, the greater the
probability of a labelling error. In the two-class problem, the probability of
a labelling error approaches 0.5 near the decision boundary.

The manual label model (2) is particularly interesting as the joint
distribution of the manual label $\bm{Z}^{\prime}$ and the feature vector
$\bm{Y}$ is the same as the joint distribution of the ground-truth label
$\bm{Z}$ and the feature vector $\bm{Y}$. The joint distributions
$(\bm{Y},\bm{Z}^{\prime})\sim g(\bm{y},\bm{z}^{\prime};\bm{\Psi})$ and
$(\bm{Y},\bm{Z})\sim f(\bm{y},\bm{z};\bm{\Psi})$ are equal as

$\displaystyle g(\bm{z}^{\prime},\bm{y};\bm{\Psi})$
$\displaystyle=f(\bm{y};\bm{\Psi})g(\bm{z}^{\prime}\mid\bm{y};\bm{\Psi})$
$\displaystyle=f(\bm{y};\bm{\Psi})f(\bm{z}^{\prime}\mid\bm{y};\bm{\Psi})$
$\displaystyle=f(\bm{z}^{\prime},\bm{y};\bm{\Psi}),$

where the substitution
$g(\bm{z}^{\prime}\mid\bm{y};\bm{\Psi})=f(\bm{z}^{\prime}\mid\bm{y};\bm{\Psi})$
is possible as the manual labels $\bm{Z}^{\prime}$ are sampled according to
the posterior probabilities of class membership (1). As such, the joint
distribution of $(\bm{Y},\bm{Z}^{\prime})$ is the same as the joint
distribution of $(\bm{Y},\bm{Z})$. Consequently, given $n$ independently and
identically distributed observations using the manual labelling model,
$\\{(\bm{Y}_{j},\bm{Z}_{j}^{{}^{\prime}})\\}_{j=1}^{n}$, the distribution of
the maximum likelihood estimate will be the same as if using a dataset with
the ground-truth labels $\\{(\bm{Y}_{j},\bm{Z}_{j})\\}_{j=1}^{n}$. It follows
that that the expected error rate of a classifier trained using
$\bm{Z}_{1}^{\prime},\ldots,\bm{Z}_{n}^{\prime}$ will be the same as a
classifier trained using the ground-truth labels
$\bm{Z}_{1},\ldots,\bm{Z}_{n}$. This is a proof of concept that label noise
can be harmless when the label noise process is governed by the posterior
probabilities of class membership.

Figure 2 shows $n=500$ observations from a simulated two-class dataset of two
normal distributions with equal identity covariance matrices and means $(1,0)$
and $(-1,0)$. Panel (a) shows the features and the ground-truth labels
$\bm{Z}_{1},\ldots,\bm{Z}_{n}$, and panel (b) shows the features and the
manual labels $\bm{Z}_{1}^{\prime},\ldots,\bm{Z}_{n}^{\prime}$ sampled from
model (2), where red triangles and blue squares represent class one and two
respectively. There were labelling errors
$(\bm{Z}_{j}\neq\bm{Z}_{j}^{\prime})$ for $24\%$ of the observations. The
errors are visibly concentrated around the decision boundary, which is
indicated by the dashed vertical line.

Figure 2: Simulated two-class normal dataset of $n=500$ observations. The
dashed vertical line represents the decision boundary. (a) Simulated features
$\bm{Y}_{1},\ldots,\bm{Y}_{n}$ and ground-truth labels
$\bm{Z}_{1},\ldots,\bm{Z}_{n}$ (b) Simulated features
$\bm{Y}_{1},\ldots,\bm{Y}_{n}$ and noisy manual labels
$\bm{Z}_{1}^{\prime},\ldots,\bm{Z}_{n}^{\prime}$ generated using the model
(2). 24% of the manual labels in (b) are not equal to the ground-truth labels
in (a).

###  2.3 Group labelling

Suppose that there are $m$ individuals in the labelling group, and they each
supply a label for each of the $n$ feature vectors in the dataset. In an
idealised scenario the ground-truth labels, feature vectors and manually
assigned labels are treated as $n$ independently and identically distributed
observations from the model:

$\displaystyle\bm{Z}_{j}$ $\displaystyle\sim\text{Multinomial}(1,\bm{\pi}),$
(3a) $\displaystyle\bm{Y}_{j}\mid Z_{ji}=1$ $\displaystyle\sim
f(\bm{y}_{j};\bm{\omega}_{i}),$ (3b)
$\displaystyle\bm{Z}_{jk}^{\prime}\mid\bm{Y}_{j}=\bm{y}_{j}$
$\displaystyle\sim\text{Multinomial}(1,\bm{\tau}(\bm{y}_{j};\bm{\Psi}))\quad(k=1,\ldots,m).$
(3c)

The random vector $\bm{Z}_{j}=(Z_{j1},\ldots,Z_{jg})^{\mathsf{T}}$ represents
the ground-truth label for the feature vector $\bm{y}_{j}$. The random vector
$\bm{Z}_{jk}^{\prime}=(Z_{jk1},\ldots,{Z}_{jkg})^{\mathsf{T}}$ represents the
label assigned to the feature vector $\bm{y}_{j}$ by the $k$th individual in
the group of experts. Let $S_{ji}=\sum_{k=1}^{m}\mathbbm{1}(Z_{jki}=1)$,
represent the number of votes that entity $j$ belongs to class $i$ over the
$m$ experts in the group, and $\bm{S}_{j}=(S_{j1},\ldots,S_{jg})^{\mathsf{T}}$
give the vector of vote counts for each of the $g$ classes. It follows from
the model (3a),(3b),(3c) that
$S_{ji}\sim\text{Binomial}(m,\tau_{i}(\bm{y}_{j};\bm{\Psi}))$ and the vector
of aggregated class counts $\bm{S}_{j}$ has the distribution

$\displaystyle\bm{S}_{j}$
$\displaystyle\sim\text{Multinomial}(m,\bm{\tau}(\bm{y}_{j};\bm{\Psi})).$ (4)

To relax the assumption about the accuracy of manual annotation, we adopt a
random effects model for the class label counts $\bm{S}_{j}$. The Dirichlet-
Multinomial model (Johnson et al., 1997) is an extension of the Multinomial
distribution that can be used to account for the fact that the group of
experts does not have exact knowledge of the posterior probabilities of class
membership $\bm{\tau}(\bm{y}_{j};\bm{\Psi})$. We introduce the overdispersion
parameter $\alpha_{0}\in\mathbb{R}^{+}$ to give the proposed Dirichlet-
Multinomial model for the aggregated manual labels

$\displaystyle\bm{S}_{j}$ $\displaystyle\sim\text{Dirichlet-
Multinomial}(m,\alpha_{0}\bm{\tau}(\bm{y}_{j};\bm{\Psi})).$ (5)

The model (5) has the hierarchical representation

$\displaystyle\bm{p}_{j}$
$\displaystyle\sim\text{Dirichlet}(\alpha_{0}\bm{\tau}(\bm{y}_{j};\bm{\Psi})),$
$\displaystyle\bm{S}_{j}\mid\bm{p}_{j}$
$\displaystyle\sim\text{Multinomial}(m,\bm{p}_{j}).$

The overdispersion parameter $\alpha_{0}$ controls how concentrated the draws
of $\bm{p}_{j}$ are around the true posterior class-probabilities
$\bm{\tau}(\bm{y}_{j};\bm{\Psi})$. As $\alpha_{0}\to\infty$ the excess
variance goes to zero, and the Dirichlet-Multinomial model (5) converges to
the multinomial distribution (4). The Dirichlet-Multinomial model still
enforces a relationship between the probability of a labelling error and
classification difficulty, but relaxes the assumptions on the accuracy of the
manual annotation compared to (4). An important difference between our model
and existing work is that we make an explicit connection between probability
of labelling error and classification difficulty using the posterior class-
probabilities.

##  3 Relative efficiency

We now consider the relative value of the aggregated noisy manual labels
$\bm{S}_{1},\ldots,\bm{S}_{n}$ compared to the ground-truth labels
$\bm{Z}_{1},\ldots,\bm{Z}_{n}$ for training a discriminative classifier (Ng
and Jordan, 2002). We will consider a binary classification problem where the
posterior probabilities of class membership $\bm{\tau}(\bm{y}_{j};\bm{\beta})$
are described by a logistic regression model with parameter
$\bm{\beta}=({\beta}_{0},\bm{\beta}^{\mathsf{T}})^{\mathsf{T}}$,

$\displaystyle\tau_{1}(\bm{y}_{j};\bm{\beta})$
$\displaystyle=\text{pr}(Z_{j1}=1\mid\bm{y}_{j};\bm{\beta})$
$\displaystyle=\dfrac{\exp(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})}{1+\exp(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})}.$
(6)

For example, this will be the case for two-class normal discrimination with
equal covariance matrices.

Let $\mathcal{Y}=\\{\bm{y}_{j}\\}_{j=1}^{n}$ represent the $n$ observed
feature vectors, $\mathcal{Z}=\\{\bm{z}_{j}\\}_{j=1}^{n}$ represent the
ground-truth labels,
$\mathcal{Z}^{\prime}=\\{\bm{z}_{jk}^{\prime}\\}_{j=1,\ldots,n}^{k=1,\ldots,m}$
represent the manual labels, and
$\mathcal{S}=\\{\bm{s}_{j}=\sum_{k=1}^{m}\bm{z}_{jk}^{\prime}\\}_{j=1}^{n}$
represent the class vote counts from manual annotation. The logistic
regression log-likelihood functions using the ground-truth labels
$\mathcal{Z}$ and the aggregated noisy manual labels $\mathcal{S}$ are
respectively,

$\displaystyle\log L_{LR}(\bm{\beta};\mathcal{Z},\mathcal{Y})$
$\displaystyle=\sum_{j=1}^{n}z_{j1}(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})-\log\left\\{1+\exp(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})\right\\},$
(7) $\displaystyle\log L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})$
$\displaystyle=\sum_{j=1}^{n}s_{j1}(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})-m\log\left\\{1+\exp(\beta_{0}+\bm{\beta}_{1}^{\mathsf{T}}\bm{y}_{j})\right\\}.$
(8)

We compare the use of $\log L_{LR}(\bm{\beta};\mathcal{Z},\mathcal{Y})$ and
$\log L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})$ for the estimation of
$\bm{\beta}$. Let $\widehat{\bm{\beta}}_{G}$ and $\widehat{\bm{\beta}}_{M}$ be
the estimated coefficients from the maximisation
$L_{LR}(\bm{\beta};\mathcal{Z},\mathcal{Y})$ and
$L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})$ respectively, and
$\widehat{R}_{G}$ and $\widehat{R}_{M}$ be the corresponding estimated
classification rules. The goal is to determine the asymptotic relative
efficiency

$\displaystyle\rm{ARE}$ $\displaystyle=\underset{n\to\infty}{\lim}\
\dfrac{\mathbb{E}\\{{\rm{err}}(\widehat{\bm{\beta}}_{G};\bm{\beta})\\}-\rm{err}(\bm{\beta};\bm{\beta})}{\mathbb{E}\\{{\rm{err}}(\widehat{\bm{\beta}}_{M};\bm{\beta})\\}-\rm{err}(\bm{\beta};\bm{\beta})},$

where $\text{err}(\bm{\beta}^{*};\bm{\beta})$ denotes the conditional error
rate when using the estimate $\bm{\beta}^{*}$ when the true parameter is
$\bm{\beta}$. The key assumptions for the analysis are summarised below.

######  Assumption 1.

The observations
$(\bm{Y}_{j}^{\mathsf{T}},\bm{Z}_{j}^{\mathsf{T}},\bm{S}_{j}^{\mathsf{T}})$
are independently and identically distributed for $j=1,\ldots,n$, and
$\text{cov}(\bm{Y}_{j})=\bm{\Sigma}$ for some symmetric positive definite
matrix $\bm{\Sigma}$. Furthermore, the conditional distribution of
$\bm{Z}_{j}$ given $\bm{Y}_{j}$ is given by the logistic regression model (6),
and the aggregated manual labels $\bm{S}_{j}$ are distributed according to the
Dirichlet-Multinomial model (5).

Using the ground-truth labels,
$\sqrt{n}(\widehat{\bm{\beta}}_{G}-\bm{\beta})\to N(\bm{0},\bm{I}_{LR}^{-1})$,
where $\bm{I}_{LR}$ is the Fisher information in the logistic regression
(Efron, 1975). Theorem 1 gives the asymptotic distribution of the estimator
using the noisy manual labels $\widehat{\bm{\beta}}_{M}$.

######  Theorem 1.

Suppose that the conditions of Assumption 1 are satisfied. Then
$\widehat{\bm{\beta}}_{M}$ is a consistent estimator of $\bm{\beta}$, and
$\sqrt{n}(\widehat{\bm{\beta}}_{M}-\bm{\beta})\to N(\bm{0},\bm{I}_{M}^{-1})$,
where $\bm{I}_{M}$ is the Godambe information matrix. The Godambe information
matrix is equal to

$\displaystyle\bm{I}_{M}$
$\displaystyle=\dfrac{m(1+\alpha_{0})}{m+\alpha_{0}}\bm{I}_{LR},$

where $\bm{I}_{LR}$ is the Fisher information matrix for the logistic
regression using the ground-truth labels, $\log
L_{LR}(\bm{\beta};\mathcal{Z},\mathcal{Y})$.

###### Proof.

Under the imperfect supervision model (5), the logistic regression on the
manual labels, $\log L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})$ is
misspecified. We first show that $\widehat{\bm{\beta}}_{M}$ remains a
consistent estimator. Under the Dirichlet-Multinomial model for the manual
labels,
$\mathbb{E}[\bm{S}_{j}\mid\bm{y}_{j}]=m\\{\bm{\tau}(\bm{y}_{j};\bm{\beta})\\}$
for all values of the overdispersion parameter $\alpha_{0}$. It follows that
the expected value of the score statistic is still zero despite the model
misspecification,

$\displaystyle\mathbb{E}_{\bm{Y},\bm{S}}\left[\dfrac{\partial}{\partial\bm{\beta}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\right]$
$\displaystyle=\mathbb{E}_{\bm{Y}}\left[\mathbb{E}_{\bm{S}\mid\bm{Y}}\left\\{\sum_{j=1}^{n}\\{S_{j1}-m\tau_{1}(\bm{y}_{j};\bm{\beta})\\}\bm{y}_{j}\mid\bm{Y}\right\\}\right]$
$\displaystyle=\mathbb{E}_{\bm{Y}}\left[\sum_{j=1}^{n}\\{m\tau_{1}(\bm{y}_{j};\bm{\beta})-m\tau_{1}(\bm{y}_{j};\bm{\beta})\\}\bm{y}_{j}\right]$
$\displaystyle=\mathbb{E}_{\bm{Y}}\left[\sum_{j=1}^{n}\\{0\\}\bm{y}_{j}\right]$
$\displaystyle=\bm{0}.$

Under the conditions of Assumption 1, we can apply Theorem 3 in Fahrmeir
(1990) for the asymptotic behaviour of M-estimators in misspecified
generalised linear models. It can be seen that the conditions of Assumption 1
are sufficient to meet the regularity conditions by appealing to Corollary 3
in Fahrmeir and Kaufmann (1985). By Theorem 3 in Fahrmeir (1990),
$\widehat{\bm{\beta}}_{M}$ is a consistent estimator of $\bm{\beta}$ and
symptomatically normally distributed. The asymptotic distribution of
$\widehat{\bm{\beta}}_{M}$ is given by
$\sqrt{n}(\widehat{\bm{\beta}}_{M}-\bm{\beta})\to N(\bm{0},\bm{I}_{M}^{-1})$,
where the Godambe information is given by
$\bm{I}_{M}=\bm{H}^{-1}\bm{G}\bm{H}^{-1}$, where

$\displaystyle\bm{H}$
$\displaystyle=-\mathbb{E}\left\\{\dfrac{\partial^{2}}{\partial\bm{\beta}\partial\bm{\beta}^{\mathsf{T}}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\right\\},$ $\displaystyle\bm{G}$
$\displaystyle=\text{cov}\left(\dfrac{\partial}{\partial\bm{\beta}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\right),$

where the expectation and variance are taken over a single feature vector
$\bm{Y}_{j}$ and vector of label counts $\bm{S}_{j}$. As $\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})$ has the form of a logistic
regression model,

$\displaystyle\bm{H}$
$\displaystyle=\mathbb{E}_{Y}\left[m\tau_{1}(\bm{y}_{j};\bm{\beta})\\{1-\tau_{1}(\bm{y}_{j};\bm{\beta})\\}\bm{y}_{j}\bm{y}_{j}^{\mathsf{T}}\right]=m\bm{I}_{LR},$
(9)

where $\bm{I}_{LR}$ is the Fisher information for $\bm{\beta}$ in the log-
likelihood $\log L_{LR}(\bm{\beta};\mathcal{Z},\mathcal{Y})$ that uses the
ground-truth labels. Under the Dirichlet-Multinomial model the variance of the
class count is

$\displaystyle\text{var}[S_{j1}\mid\bm{y}_{j}]$
$\displaystyle=m\tau_{1}(\bm{y}_{j};\bm{\beta})\\{1-\tau_{1}(\bm{y}_{j};\bm{\beta})\\}\dfrac{m+\alpha_{0}}{1+\alpha_{0}}.$

Using the law of total-variance,

$\displaystyle\bm{G}$
$\displaystyle=\text{cov}\left(\dfrac{\partial}{\partial\bm{\beta}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\right)$
$\displaystyle=\mathbb{E}_{Y}\left[\text{cov}\left(\dfrac{\partial}{\partial\bm{\beta}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\right)\mid\bm{Y}\right]+\text{cov}\left(\mathbb{E}_{S\mid
Y}\left[\dfrac{\partial}{\partial\bm{\beta}}\log
L_{LR}(\bm{\beta};\mathcal{S},\mathcal{Y})\mid\bm{Y}\right]\right)$
$\displaystyle=\mathbb{E}_{Y}\left[\text{var}(S_{j1})\bm{y}_{j}\bm{y}_{j}^{\mathsf{T}}\right]+\bm{0}$
$\displaystyle=\dfrac{m+\alpha_{0}}{1+\alpha_{0}}\mathbb{E}_{Y}\left[m\tau_{1}(\bm{y}_{j};\bm{\beta})\\{1-\tau_{1}(\bm{y}_{j};\bm{\beta})\\}\bm{y}_{j}\bm{y}_{j}^{\mathsf{T}}\right]$
$\displaystyle=\dfrac{m+\alpha_{0}}{1+\alpha_{0}}m\bm{I}_{LR}.$ (10)

Combining (9) and (10), the Godambe information $\bm{I}_{M}$ is given by

$\displaystyle\bm{I}_{M}$ $\displaystyle=\bm{H}\bm{G}^{-1}\bm{H}$
$\displaystyle=\dfrac{1+\alpha_{0}}{m+\alpha_{0}}m\bm{I}_{LR}\bm{I}_{LR}^{-1}\bm{I}_{LR}$
$\displaystyle=\dfrac{m(1+\alpha_{0})}{m+\alpha_{0}}\bm{I}_{LR}.$

∎

Under the Dirichlet-Multinomial model (5), logistic regression using the noisy
labels $\mathcal{S}$ provides a consistent estimator of ${\bm{\beta}}$. This
is not necessarily the case when label noise process is uniform within each
class (Michalek and Tripathi, 1980; Bi and Jeske, 2010). Theorem 1 connects
the Godambe information $\bm{I}_{M}$ from the set of noisy labels
$\mathcal{S}$ to the Fisher information $\bm{I}_{LR}$ from the the ground-
truth labels $\mathcal{Z}$. From this relationship we can derive the
asymptotic relative efficiency of $\widehat{R}_{M}$ compared to
$\widehat{R}_{G}$.

######  Theorem 2.

Suppose the conditions of Assumption 1 are satisfied. The asymptotic relative
efficiency of $\widehat{R}_{M}$ compared to $\widehat{R}_{G}$ is

$\displaystyle\rm{ARE}$ $\displaystyle=\dfrac{m(1+\alpha_{0})}{m+\alpha_{0}}.$

###### Proof.

The asymptotic excess error can be expanded as

$\displaystyle\underset{n\to\infty}{\lim}\
n\left[\mathbb{E}\\{\rm{err}(\widehat{\bm{\beta}};\bm{\beta})\\}-\rm{err}(\bm{\beta};\bm{\beta})\right]$
$\displaystyle={\rm{trace}}\\{\bm{J}(\bm{\beta})\bm{I}^{-1}(\bm{\beta})\\},$

where $\bm{I}(\bm{\beta})$ is the information matrix for a single observation,
and

$\displaystyle\bm{J}(\bm{\beta})$
$\displaystyle=\tfrac{1}{2}\left[\nabla\nabla^{\mathsf{T}}{\rm{err}}(\widehat{\bm{\beta}};\bm{\beta})\right]_{\widehat{\bm{\beta}}=\bm{\beta}}.$

The ARE can then be expressed as

$\displaystyle\rm{ARE}$ $\displaystyle=\underset{n\to\infty}{\lim}\
\dfrac{\mathbb{E}\\{{\rm{err}}(\widehat{\bm{\beta}}_{G};\bm{\beta})\\}-\rm{err}(\bm{\beta};\bm{\beta})}{\mathbb{E}\\{{\rm{err}}(\widehat{\bm{\beta}}_{M};\bm{\beta})\\}-\rm{err}(\bm{\beta};\bm{\beta})}$
$\displaystyle=\dfrac{{\rm{trace}}(\bm{J}(\bm{\beta})\bm{I}_{LR}^{-1}(\bm{\beta}))}{{\rm{trace}}(\bm{J}(\bm{\beta})\bm{I}_{M}^{-1}(\bm{\beta}))}.$

Using Theorem 1 and substituting
$\bm{I}_{M}=\tfrac{m(1+\alpha_{0})}{m+\alpha_{0}}\bm{I}_{LR}$,

$\displaystyle\rm{ARE}$
$\displaystyle=\dfrac{{\rm{trace}}(\bm{J}(\bm{\beta})\bm{I}_{LR}^{-1}(\bm{\beta}))}{\tfrac{m+\alpha_{0}}{m(1+\alpha_{0})}{\rm{trace}}(\bm{J}(\bm{\beta})\bm{I}_{LR}^{-1}(\bm{\beta}))}$
$\displaystyle=\dfrac{m(1+\alpha_{0})}{m+\alpha_{0}}.$

∎

######  Remark 1.

The limiting value of the Godambe information as $m\to\infty$ is

$\lim_{m\to\infty}\bm{I}_{M}=(1+\alpha_{0})\bm{I}_{LR}.$

The limiting value of the ARE as $m\to\infty$ is $(1+\alpha_{0})$. The
parameter $\alpha_{0}$ gives an upper bound on efficiency gain that can be
obtained given imperfect manual labels. Under the Dirichlet-Multinomial model,
the information supplied by the $m$ sets of noisy labels is bounded below by
$\bm{I}_{LR}$ as

$\lim_{\alpha_{0}\to 0^{+}}\bm{I}_{M}=\bm{I}_{LR}.$

Consequently the ARE is bounded below by one. ∎

Previous evidence for the benefits of incorporating multiple noisy labels in
supervised learning has largely been empirical (Sheng et al., 2008; Natarajan
et al., 2013; Raykar et al., 2009; Yan et al., 2010), Theorems 1 and 2 give
important supporting theory on the robustness and effectiveness of noisy
manual labels.

##  4 Simulation

We simulated data from a canonical form for two-class normal discrimination,
where $\bm{\Sigma}=\bm{I}$ and
$\bm{\mu}_{1}=(\Delta/2,0,\ldots,0)^{\mathsf{T}},\bm{\mu}_{2}=(-\Delta/2,0,\ldots,0)^{\mathsf{T}}$.
The coefficient vector
$\bm{\beta}=(\beta_{0},\bm{\beta}_{1}^{\mathsf{T}})^{\mathsf{T}}$ is given by

$\displaystyle\beta_{0}$
$\displaystyle=-\dfrac{1}{2}(\bm{\mu}_{1}+\bm{\mu}_{2})^{\mathsf{T}}\bm{\Sigma}^{-1}(\bm{\mu}_{1}-\bm{\mu}_{2})+\log(\pi_{1}/\pi_{2}),$
$\displaystyle\bm{\beta}$
$\displaystyle=\bm{\Sigma}^{-1}(\bm{\mu}_{1}-\bm{\mu}_{2}).$

We performed 1000 simulations at different combinations of $m,\Delta$, and
$\alpha_{0}$ to estimate the relative efficiency of $\widehat{R}_{M}$ compared
to $\widehat{R}_{G}$ with $n=500,p=2$. The class prior probabilities were
taken to be equal. In each replication we computed the exact conditional error
rate of $\widehat{\bm{\beta}}_{G}$ and $\widehat{\bm{\beta}}_{M}$.
Additionally, we computed the exact Bayes’ error rate
$\text{err}(\bm{\beta};\bm{\beta})$ for each value of $\Delta$. Table 1
reports the simulated relative efficiency of $\widehat{R}_{M}$ compared to
$\widehat{R}_{C}$. Bootstrap standard errors (Efron and Tibshirani, 1986) are
given in parentheses. The simulated relative efficiencies are close to the
theoretical values for each combination of $m,\Delta$, and $\alpha_{0}$. The
simulated relative efficiencies are not sensitive to $\Delta$. For each
combination of $m$ and $\alpha_{0}$, the differences in the results over
$\Delta=1,2,3$ and $4$ are within the standard errors.

Table 1: Simulated relative efficiency of $\widehat{R}_{M}$ compared to $\widehat{R}_{G}$ with $\pi_{1}=\pi_{2}$ for $n=500,p=2$. Standard errors are given in parentheses. | $\alpha_{0}$ | ARE | $\Delta=1$ | $\Delta=2$ | $\Delta=3$ | $\Delta=4$
---|---|---|---|---|---|---
$m=5$ | 1 | 1.67 | 1.58 (0.07) | 1.82 (0.08) | 1.68 (0.08) | 1.74 (0.08)
| 10 | 3.67 | 3.33 (0.15) | 3.74 (0.18) | 3.78 (0.19) | 3.51 (0.16)
| 100 | 4.81 | 4.65 (0.21) | 4.79 (0.23) | 4.87 (0.23) | 5.09 (0.22)
| 1000 | 4.98 | 4.57 (0.20) | 5.21 (0.24) | 5.03 (0.23) | 4.99 (0.21)
$m=10$ | 1 | 1.82 | 1.73 (0.08) | 1.97 (0.09) | 1.83 (0.09) | 1.89 (0.08)
| 10 | 5.50 | 5.10 (0.24) | 5.64 (0.26) | 5.47 (0.25) | 5.35 (0.24)
| 100 | 9.18 | 8.87 (0.40) | 9.18 (0.43) | 9.40 (0.44) | 9.69 (0.44)
| 1000 | 9.91 | 9.08 (0.41) | 10.41 (0.48) | 9.86 (0.44) | 9.99 (0.43)
$m=50$ | 1 | 1.96 | 1.86 (0.08) | 2.12 (0.09) | 2.00 (0.10) | 2.03 (0.09)
| 10 | 9.17 | 8.52 (0.38) | 9.38 (0.45) | 9.23 (0.45) | 8.84 (0.40)
| 100 | 33.67 | 32.50 (1.48) | 34.68 (1.62) | 34.53 (1.61) | 36.64 (1.67)
| 1000 | 47.67 | 43.28 (1.95) | 50.85 (2.31) | 48.85 (2.23) | 48.32 (2.22)

##  5 Data application

###  5.1 Gastroentology dataset

The Gastroentology dataset contain information on $n=76$ patients (Mesejo et
al., 2016) and is available from the UCI Machine learning repository (Dua and
Graff, 2017). For each patient, a regular colonoscopic video showed a
gastrointenstinal lesion. Seven clinicians reviewed the videos and made an
assessment on whether the lesions were benign or malignant, and these
classifications were taken to be the manual labels $\mathcal{Z}^{\prime}$. The
vote count totals $\mathcal{S}$ were then computed using
$\mathcal{Z}^{\prime}$. The dataset also contains the ground-truth
classifications $\mathcal{Z}$ obtained through histology and expert image
inspection. The full dataset contains $698$ variables. After standardizing
each variable to have zero mean and unit variance we selected a subset of
$p=5$ variables using sparse discriminant analysis (Clemmensen et al., 2011).
The $n=76$ observations on the variables $V113,V173,V475,V489$ and $V603$ were
taken to be the features $\mathcal{Y}$.

Assuming the logistic regression model is correctly specified, the model for
the manual labels

$\displaystyle\bm{S}_{j}$ $\displaystyle\sim\text{Dirichlet-
Multinomial}(m,\alpha_{0}\bm{\tau}(\bm{y}_{j};{\bm{\beta}}))\quad(j=1,\ldots,n),$
(11)

is a function of the true value of $\bm{\beta}$. As $\bm{\beta}$ is unknown, a
suitable estimate $\widehat{\bm{\beta}}$ can be used to construct a working
version of the manual label model,

$\displaystyle\bm{S}_{j}$ $\displaystyle\sim\text{Dirichlet-
Multinomial}(m,\alpha_{0}\bm{\tau}(\bm{y}_{j};\widehat{\bm{\beta}}))\quad(j=1,\ldots,n),$
(12)

From Theorem 1, $\widehat{\bm{\beta}}_{M}$ is a consistent estimator of
$\bm{\beta}$. As the ground-truth labels are also available we can also use
$\widehat{\bm{\beta}}_{G}$ as an estimate of $\bm{\beta}$. In many situations
the ground-truth labels will not be available, but they are useful here to
check for the consistency of results. To estimate $\alpha_{0}$ we propose to
maximise the likelihood of the model (12) with $\widehat{\bm{\beta}}$ fixed at
a suitable estimate. Using $\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{G}$ we
obtained the estimate $\widehat{\alpha}_{0}=5.47$, and using
$\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{M}$ we obtained the estimate
$\widehat{\alpha}_{0}=5.17$.

We now discuss some methods for assessing the suitability of the manual
labelling model. A key property of the Dirichlet-Multinomial model is that it
posits a relationship between the level of group agreement and classification
difficulty. Given the posterior probabilities of class membership
$\bm{\tau}(\bm{y}_{j};{\bm{\beta}})$ for $j=1,\ldots,n$, observations can be
divided into groups according to the number of votes that were received for
the positive class. A goodness of fit measure is then to compute within each
vote group the average posterior class-probability

$\displaystyle\overline{\bm{\tau}}_{v}(\bm{\beta})$
$\displaystyle={n_{v}}^{-1}\sum_{j\in\mathcal{N}_{v}}\bm{\tau}(\bm{y}_{j};{\bm{\beta}}),$

where the set $\mathcal{N}_{v}$ contains the indices of the $n_{v}$
observations which received $v$ positive votes $(v=0,\ldots,7)$. A
straightforward goodness of fit measure is to plot the observed vote counts
against the expected vote counts
$\mathbb{E}[\bm{S}_{j}\mid\bm{y}_{j};\bm{\beta}]=m\bm{\tau}(\bm{y}_{j};{\bm{\beta}})$.
Figure 3 plots goodness of fit diagnostics for the working version of the
manual label model (12). The top row shows results using the model based on
the estimate $\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{G}$ and the bottom
row shows the results using the model based on the estimate
$\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{M}$.

The first column in Figure 3 plots average posterior probabilities against the
number of positive votes $v$, the error bars represent plus or minus one
standard error of the mean. The top row displays
$\overline{\bm{\tau}}_{v}(\widehat{\bm{\beta}}_{G})$, and the bottom row
displays $\overline{\bm{\tau}}_{v}(\widehat{\bm{\beta}}_{M})$. As expected,
the average posterior class-probability is associated with the level of group
agreement. In panel 1(a), for the instances where there was unanimous
agreement, so $v=0$ or $v=7$, the average posterior class-probability is close
to zero or one respectively. There appears to be a positive relationship
between the number of votes and the average posterior class-probability.

Panels (1b) and (2b) in Figure 3 compare the observed vote counts to the
expected vote counts. The top row shows $\mathbb{E}[\bm{S}_{j}\mid\bm{y}_{j};\
\widehat{\bm{\beta}}_{G}]=m\bm{\tau}(\bm{y}_{j};\ \widehat{\bm{\beta}}_{G})$
and the bottom row shows $\mathbb{E}[\bm{S}_{j}\mid\bm{y}_{j};\
\widehat{\bm{\beta}}_{M}]=m\bm{\tau}(\bm{y}_{j};\ \widehat{\bm{\beta}}_{M})$
for $j=1,\ldots,n$. The red line gives the theoretical mean, and the predicted
trend in the vote counts is consistent with the observed votes $\mathcal{S}$.
The predicted model for $\bm{S}_{j}$ in (1b) was computed on the basis of the
ground-truth labels $\mathcal{Z}$. There is less deviation between the
observed and expected counts for $\bm{S}_{j}$ in (2b) compared to (1b) as
$\widehat{\bm{\beta}}_{M}$ was itself computed using the manual labels
$\mathcal{S}$.

The dashed lines in Figure 3 (1b) and (2b) show a conservative 95% prediction
interval for the counts using the estimated parameters. For the estimated
model using the ground-truth labels (1b), the majority of the data-points are
within the prediction bands, with the exception of some points with a large
posterior class-probability. The estimated model using the manual-labels in
(2b) has more observations falling within the prediction bands. There are no
outliers with large posterior class-probabilities.

Figure 3 (1c) and (2c) show histograms of 500 bootstrap estimates of
$\alpha_{0}$. Using $\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{G}$ the 95%
percentile bootstrap confidence interval is $(3.32,11.66)$. Using
$\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{M}$ the 95% percentile bootstrap
confidence interval is $(2.94,10.43)$. The random effects model for the manual
labels (11) appears to be more appropriate compared to the idealised
Multinomial model
$\bm{S}_{j}\sim\text{Multinomial}(m,\bm{\tau}(\bm{y}_{j};\bm{\beta}))$ as
there is little evidence in favour of a boundary estimate
$\alpha_{0}\to\infty$.

Figure 3: Assessment of the Dirichlet-Multinomial model for manual labelling
(12) on the gastroentology dataset. The top row shows results for the model
using the estimate $\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{G}$ computed
using the ground-truth labels $\mathcal{Z}$. The bottom row show results for
the model using the estimate $\widehat{\bm{\beta}}=\widehat{\bm{\beta}}_{M}$
computed using the manual labels $\mathcal{S}$. Column (a): Average estimated
posterior probability of positive class membership against number of positive
votes. Column (b): Comparison of observed and expected values for the number
of positive votes. Column (c): Bootstrap estimates of the overdispersion
parameter $\alpha_{0}$ in the Dirichlet-Multinomial model.

###  5.2 Wisconsin breast cancer dataset

We performed some experiments on the Wisconsin breast cancer dataset $(N=569,\
p=10)$ using simulated noisy labels. The dataset is available from the UCI
machine learning repository (Dua and Graff, 2017). We split the original
dataset of $N$ observations into a training set of $n$ observations and a test
set of $N-n$ observations. The maximum likelihood estimate of $\bm{\beta}$
using the full dataset of $N$ observations was taken to be the ground-truth
parameter value. The $N$ labels in the dataset were also assumed to be the
ground-truth labels. Manual labels were then simulated according to the
Dirichlet-Multinomial model for different values of the group size $m$ and
overdispersion parameter $\alpha_{0}$. We computed $\widehat{\bm{\beta}}_{G}$
and $\widehat{\bm{\beta}}_{M}$ using the training set of $n$ observations and
then computed the error rate on the test set of $N-n$ observations. This was
repeated for 100 random test-train splits. Table 2 reports the average test
set error of $\widehat{R}_{M}$ for each combination of $\alpha_{0}$ and $m$.
The value of the overdispersion parameter $\alpha_{0}$ has a strong impact on
the error rate. For $m=100$, the average error is $10.73\%$ for
$\alpha_{0}=1$, and is reduced to $5.54\%$ for $\alpha_{0}=1000$. The average
test set error of $\widehat{R}_{G}$ was 13.29% with standard error 0.37%. For
all combinations of $m$ and $\alpha_{0}$ the average test error using
$\widehat{R}_{M}$ was lower than $\widehat{R}_{G}$, which is consistent with
Theorem 2.

The value of $m$ has a larger impact on the error rates as $\alpha_{0}$
increases. As mentioned in Remark 1, the value of $\alpha_{0}$ gives an upper
bound on the information that can be obtained as the number of labellers $m$
increases,

$\displaystyle\lim_{m\to\infty}\bm{I}_{M}=(1+\alpha_{0})\bm{I}_{LR}.$

For $\alpha_{0}=1$, $\bm{I}_{M}$ is approaching $2\bm{I}_{LR}$ as $m$
increases. There is a small reduction in error rates comparing $m=5$ and
$m=100$ as there is a strong limit on the amount of information that can be
obtained by group size $m$. For $\alpha_{0}=1000$, $\bm{I}_{M}$ is approaching
$1001\bm{I}_{LR}$ as $m$ increases, and benefit of additional labellers is
much greater. The error rate approximately halves, dropping from 10.06% at
$m=5$ to 5.54% at $m=100$.

Table 2: Simulation results using the breast cancer dataset. Average test set error percentage of $\widehat{R}_{M}$ over 100 test-train splits using $n=50$ training samples. Standard errors of the mean are given in parentheses. The average test set error of $\widehat{R}_{G}$ was 13.28% $\alpha_{0}$ | $m=5$ | $m=10$ | $m=20$ | $m=50$ | $m=100$
---|---|---|---|---|---
1 | 11.63 (0.34) | 11.21 (0.34) | 11.01 (0.34) | 10.45 (0.31) | 10.73 (0.36)
10 | 9.78 (0.33) | 8.66 (0.30) | 7.83 (0.26) | 7.14 (0.19) | 6.99 (0.19)
100 | 9.45 (0.27) | 7.92 (0.23) | 6.78 (0.15) | 6.06 (0.11) | 5.85 (0.08)
1000 | 10.06 (0.34) | 8.00 (0.30) | 6.64 (0.18) | 5.87 (0.10) | 5.54 (0.05)

Table 3 reports the simulated relative efficiency of $\widehat{R}_{M}$
compared to $\widehat{R}_{G}$. The Bayes’ error rate was taken to be the
apparent error rate when using the logistic regression model trained on the
complete set of $N$ observations. The theoretical relative efficiency of
$\widehat{R}_{M}$ compared to $\widehat{R}_{G}$ using Theorem 2 is given in
parentheses in Table 3. The simulated relative efficiencies are smaller than
the corresponding theoretical values. This may be due to underestimation of
the Bayes’ error rate, and the fact that the true conditional error rates may
not be estimated perfectly with the limited test set of $N-n$ observations. In
the simulations in Section 4, it was possible to compute the exact conditional
error rate and the true Bayes’ error rate, and this could explain the better
agreement between the simulated relative efficiency and the asymptotic
relative efficiency there. Looking at the theoretical error rates, it is again
possible to see the stronger influence of $m$ for larger values of
$\alpha_{0}$. For $\alpha_{0}=1,\ m=100$ the ARE is 1.98, which is close to
the limiting value of 2 as $m\to\infty$. For $\alpha_{0}=1000,\ m=100$ the ARE
is 91.00 which is much closer to the nominal number of labellers $m=100$.

Table 3: Simulated relative efficiency of $\widehat{R}_{M}$ compared to $\widehat{R}_{G}$ on the breast cancer dataset. The theoretical ARE of $\widehat{R}_{M}$ to $\widehat{R}_{G}$ as given by Theorem 2 is given in parentheses. $\alpha_{0}$ | $m=5$ | $m=10$ | $m=20$ | $m=50$ | $m=100$
---|---|---|---|---|---
1 | 1.25 (1.67) | 1.34 (1.82) | 1.38 (1.90) | 1.53 (1.96) | 1.45 (1.98)
10 | 1.75 (3.67) | 2.30 (5.50) | 2.99 (7.33) | 4.01 (9.17) | 4.33 (10.00)
100 | 1.88 (4.81) | 2.90 (9.18) | 4.86 (16.83) | 8.52 (33.67) | 10.82 (50.50)
1000 | 1.65 (4.98) | 2.82 (9.91) | 5.32 (19.63) | 10.66 (47.67) | 18.65 (91.00)

##  6 Conclusion

Label noise is a practical consideration in supervised learning that can have
interesting statistical effects (McLachlan, 1972; Frénay and Verleysen, 2014;
Cannings et al., 2020; Song et al., 2020). A consequence of Bayes’ theorem is
that noisy labels sampled according to the posterior probabilities of class
membership supply as much information as the ground-truth labels. Based on
this observation we proposed a random effects model for manual labelling that
takes into account that the experts do not have perfect knowledge of the
posterior probabilities of class membership. Under the proposed model, the
sandwich information matrix associated with the use of the noisy manual labels
is a multiple of the Fisher information matrix associated with the use of the
ground-truth labels. We derived the asymptotic relative efficiency of logistic
regression using the noisy manual labels compared to logistic regression using
the ground-truth labels, and found that the classifier trained using multiple
sets of noisy manual labels can outperform the classifier trained using the
single set of ground-truth labels.

A limitation of the Dirichlet-Multinomial model is that the information in the
noisy labels $\bm{I}_{M}$ is bounded below by $\bm{I}_{LR}$. The model is not
flexible enough to allow for the $m$ sets of noisy labels to be less
informative than the single set of ground-truth labels. The Dirichlet
distribution is conjugate to the Multinomial, and was viewed as a natural
choice for the random effect to capture imperfect labelling that is
mathematically tractable. Extensions to this work could develop richer models
that allow for $\bm{I}_{M}$ to be less than $\bm{I}_{LR}$ with very noisy
labels. There is empirical evidence that multiple noisy labels can be less
informative than the single set of ground-truth labels (Sheng et al., 2008).

A second area where improvements could be made is allowing for correlated
labelling errors. In the present model, the manual labels are drawn
independently for each instance in the training set. It is plausible that
labelling errors will be correlated for similar instances in the training set.
Correlated labelling errors could be modelled by allowing for correlated
random effects in the Dirichlet-Multinomial model. Copulas could be used to
introduce the correlation. However, we expect that derivations of the Godambe
information and the asymptotic relative efficiency as in Section 3 will become
more challenging to calculate.

Modelling manual annotation by a group of experts is useful as it can allow
for more information to be extracted than by simply taking the majority vote
(Raykar et al., 2009; Yan et al., 2010; Song et al., 2020). When group
disagreement is related to classification difficulty, it is possible to use
this association to increase the accuracy of a classifier.

## Acknowledgements

This research was funded by the Australian Government through the Australian
Research Council (Project Numbers DP170100907 and IC170100035).

## References

  * Bi and Jeske (2010) Bi, Y. and Jeske, D. R. (2010) The efficiency of logistic regression compared to normal discriminant analysis under class-conditional classification noise.  Journal of Multivariate Analysis, 101, 1622–1637.
  * Blanchard et al. (2016) Blanchard, G., Flaska, M., Handy, G., Pozzi, S. and Scott, C. (2016) Classification with asymmetric label noise: Consistency and maximal denoising.  Electronic Journal of Statistics, 10, 2780–2824.
  * Bouveyron et al. (2019) Bouveyron, C., Celeux, G., Murphy, T. and Raftery, A. (2019) Model-Based Clustering and Classification for Data Science: With Applications in R.  Cambridge University Press.
  * Bouveyron and Girard (2009) Bouveyron, C. and Girard, S. (2009) Robust supervised classification with mixture models: Learning from data with uncertain labels.  Pattern Recognition, 42, 2649–2658.
  * Cannings et al. (2020) Cannings, T. I., Fan, Y. and Samworth, R. J. (2020) Classification with imperfect training labels.  Biometrika, 107, 311–330.
  * Cappozzo et al. (2019) Cappozzo, A., Greselin, F. and Murphy, T. B. (2019) A robust approach to model-based classification based on trimming and constraints.  Advances in Data Analysis and Classification, 14, 327–354.
  * Cheng and Liu (2001) Cheng, R. C. H. and Liu, W. B. (2001) The consistency of estimators in finite mixture models.  Scandinavian Journal of Statistics, 28, 603–616.
  * Clemmensen et al. (2011) Clemmensen, L., Hastie, T., Witten, D. and Ersbøll, B. (2011) Sparse discriminant analysis.  Technometrics, 53, 406–413.
  * Dawid and Skene (1979) Dawid, A. P. and Skene, A. M. (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 28, 20–28.
  * Donmez et al. (2010) Donmez, P., Carbonell, J. and Schneider, J. (2010) A probabilistic framework to learn from multiple annotators with time-varying accuracy.  In Proceedings of the 2010 SIAM International Conference on Data Mining, 826–837. SIAM.
  * Dua and Graff (2017) Dua, D. and Graff, C. (2017) UCI machine learning repository. University of California, Irvine, School of Information and Computer Sciences.
  * Efron (1975) Efron, B. (1975) The efficiency of logistic regression compared to normal discriminant analysis.  Journal of the American Statistical Association, 70, 892–898.
  * Efron and Tibshirani (1986) Efron, B. and Tibshirani, R. (1986) Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy.  Statistical Science, 1, 54–75.
  * Fahrmeir (1990) Fahrmeir, L. (1990) Maximum likelihood estimation in misspecified generalized linear models.  Statistics, 21, 487–502.
  * Fahrmeir and Kaufmann (1985) Fahrmeir, L. and Kaufmann, H. (1985) Consistency and asymptotic normality of the maximum likelihood estimator in generalized linear models.  The Annals of Statistics, 13, 342–368.
  * Fraley and Raftery (2002) Fraley, C. and Raftery, A. E. (2002) Model-based clustering, discriminant analysis, and density estimation.  Journal of the American Statistical Association, 97, 611–631.
  * Frénay and Verleysen (2014) Frénay, B. and Verleysen, M. (2014) Classification in the presence of label noise: a survey.  IEEE Transactions on Neural Networks and Learning Systems, 25, 845–869.
  * Hovy et al. (2013) Hovy, D., Berg-Kirkpatrick, T., Vaswani, A. and Hovy, E. (2013) Learning whom to trust with MACE.  In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1120–1130. Association for Computational Linguistics.
  * Ipeirotis et al. (2010) Ipeirotis, P. G., Provost, F. and Wang, J. (2010) Quality management on Amazon Mechanical Turk.  In Proceedings of the ACM SIGKDD Workshop on Human Computation, 64–67.
  * Jin and Ghahramani (2003) Jin, R. and Ghahramani, Z. (2003) Learning with multiple labels.  In Advances in Neural Information Processing Systems, 921–928.
  * Johnson et al. (1997) Johnson, N. L., Kotz, S. and Balakrishnan, N. (1997) Discrete multivariate distributions.  New York: Wiley.
  * McLachlan (1972) McLachlan, G. J. (1972) Asymptotic results for discriminant analysis when the initial samples are misclassified.  Technometrics, 14, 415–422.
  * McLachlan and Peel (2000) McLachlan, G. J. and Peel, D. (2000) Finite Mixture Models.  Wiley.
  * Mesejo et al. (2016) Mesejo, P., Pizarro, D., Abergel, A., Rouquette, O., Beorchia, S., Poincloux, L. and Bartoli, A. (2016) Computer-aided classification of gastrointestinal lesions in regular colonoscopy.  IEEE Transactions on Medical Imaging, 35, 2051–2063.
  * Michalek and Tripathi (1980) Michalek, J. E. and Tripathi, R. C. (1980) The effect of errors in diagnosis and measurement on the estimation of the probability of an event.  Journal of the American Statistical Association, 75, 713–721.
  * Natarajan et al. (2013) Natarajan, N., Dhillon, I. S., Ravikumar, P. K. and Tewari, A. (2013) Learning with noisy labels.  In Advances in Neural Information Processing Systems, 1196–1204.
  * Ng and Jordan (2002) Ng, A. Y. and Jordan, M. I. (2002) On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes.  In Advances in Neural Information Processing Systems, 841–848.
  * Quost et al. (2017) Quost, B., Denœux, T. and Li, S. (2017) Parametric classification with soft labels using the evidential EM algorithm: linear discriminant analysis versus logistic regression.  Advances in Data Analysis and Classification, 11, 659–690.
  * Raykar and Yu (2012) Raykar, V. C. and Yu, S. (2012) Eliminating spammers and ranking annotators for crowdsourced labeling tasks.  The Journal of Machine Learning Research, 13, 491–518.
  * Raykar et al. (2009) Raykar, V. C., Yu, S., Zhao, L. H., Jerebko, A., Florin, C., Valadez, G. H., Bogoni, L. and Moy, L. (2009) Supervised learning from multiple experts: whom to trust when everyone lies a bit.  In Proceedings of the 26th Annual International Conference on Machine Learning, 889–896.
  * Rebbapragada and Brodley (2007) Rebbapragada, U. and Brodley, C. E. (2007) Class noise mitigation through instance weighting.  In European Conference on Machine Learning, 708–715. Springer.
  * Sheng et al. (2008) Sheng, V. S., Provost, F. and Ipeirotis, P. G. (2008) Get another label? improving data quality and data mining using multiple, noisy labelers.  In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 614–622.
  * Smyth et al. (1995) Smyth, P., Fayyad, U. M., Burl, M. C., Perona, P. and Baldi, P. (1995) Inferring ground truth from subjective labelling of venus images.  In Advances in Neural Information Processing Systems, 1085–1092.
  * Song et al. (2020) Song, H., Dai, R., Raskutti, G. and Barber, R. F. (2020) Convex and non-convex approaches for statistical inference with class-conditional noisy labels.  Journal of Machine Learning Research, 21, 1–58.
  * Vranckx et al. (2021) Vranckx, I., Raymaekers, J., Ketelaere, B. D., Rousseeuw, P. J. and Hubert, M. (2021) Real-time discriminant analysis in the presence of label and measurement noise.  Chemometrics and Intelligent Laboratory Systems, 208, 104197.
  * Welinder et al. (2010) Welinder, P., Branson, S., Perona, P. and Belongie, S. J. (2010) The multidimensional wisdom of crowds.  In Advances in Neural Information Processing Systems, 2424–2432.
  * Xu et al. (2006) Xu, L., Crammer, K. and Schuurmans, D. (2006) Robust support vector machine training via convex outlier ablation.  In Proceedings of the 21st National Conference on Artificial Intelligence, AAAI’06, 536–542. AAAI Press.
  * Yan et al. (2010) Yan, Y., Rosales, R., Fung, G., Schmidt, M., Hermosillo, G., Bogoni, L., Moy, L. and Dy, J. (2010) Modeling annotator expertise: Learning when everybody knows a bit of something.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 932–939.
  * Zhang et al. (2013) Zhang, P., Cao, W. and Obradovic, Z. (2013) Learning by aggregating experts and filtering novices: a solution to crowdsourcing problems in bioinformatics.  BMC Bioinformatics, 14, 1–8.