Supervised learning depends on annotated examples, which are taken to be the ground truth. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits. Recent advances in supervised learning owe, in part, to the availability of large annotated datasets. For instance, the performance of modern image classifiers saturates only with millions of labeled examples. This poses an economic problem: Assembling such datasets typically requires the labor of human annotators. If we confined the labor pool to experts, this work might be prohibitively expensive. Therefore, most practitioners turn to crowdsourcing platforms such as Amazon Mechanical Turk (AMT), which connect employers with low-skilled workers who perform simple tasks, such as classifying images, at low cost.Compared to experts, crowd-workers provide noisier annotations, possibly owing to high variation in worker skill; and a per-answer compensation structure that encourages rapid answers, even at the expense of accuracy. To address variation in worker skill, practitioners typically collect multiple independent labels for each training example from different workers. In practice, these labels are often aggregated by applying a simple majority vote. Academics have proposed many efficient algorithms for estimating the ground truth from noisy annotations. Research addressing the crowd-sourcing problem goes back to the early 1970s. BID4 proposed a probabilistic model to jointly estimate worker skills and ground truth labels and used expectation maximization (EM) to estimate the parameters. BID27 ; ; BID29 proposed generalizations of the Dawid-Skene model, e.g. by estimating the difficulty of each example.Although the downstream goal of many crowdsourcing projects is to train supervised learning models, research in the two disciplines tends to proceed in isolation. Crowdsourcing research seldom accounts for the downstream utility of the produced annotations as training data in machine learning (ML) algorithms. And ML research seldom exploits the noisy labels collected from multiple human workers. A few recent papers use the original noisy labels and the corresponding worker identities together with the predictions of a supervised learning model trained on those same labels, to estimate the ground truth BID2 BID7 . However, these papers do not realize the full potential of combining modeling and crowd-sourcing. In particular, they are unable to estimate worker qualities when there is only one label per training example.This paper presents a new supervised learning algorithm that alternately models the labels and worker quality. The EM algorithm bootstraps itself in the following way: Given a trained model, the algorithm estimates worker qualities using the disagreement between workers and the current predictions of the learning algorithm. Given estimated worker qualities, our algorithm optimizes a suitably modified loss function. We show that accurate estimates of worker quality can be obtained even when only collecting one label per example provided that each worker labels sufficiently many examples. An accurate estimate of the worker qualities leads to learning a better model. This addresses a shortcoming of the prior work and overcomes a significant hurdle to achieving practical crowdsourcing without redundancy.We give theoretical guarantees on the performance of our algorithm. We analyze the two alternating steps: (a) estimating worker qualities from disagreement with the model, (b) learning a model by optimizing the modified loss function. We obtain a bound on the accuracy of the estimated worker qualities and the generalization error of the model. Through the generalization error bound, we establish that it is better to label many examples once than to label less examples multiply when worker quality is above a threshold. Empirically, we verify our approach on several multi-class classification datasets: ImageNet and CIFAR10 (with simulated noisy workers), and MS-COCO (using the real noisy annotator labels). Our experiments validate that when the cost of obtaining unlabeled examples is negligible and the total annotation budget is fixed, it is best to collect a single label per training example for as many examples as possible. We emphasize that although this paper applies our approach to classification problems, the main ideas of the algorithm can be extended to other tasks in supervised learning. We introduced a new algorithm for learning from noisy crowd workers. We also presented a new theoretical and empirical demonstration of the insight that when examples are cheap and annotations expensive, it's better to label many examples once than to label few multiply when worker quality is above a threshold. Many avenues seem ripe for future work. We are especially keen to incorporate our approach into active query schemes, choosing not only which examples to annotate, but which annotator to route them to based on our models current knowledge of both the data and the worker confusion matrices. Lemma A.2. Under the assumptions of Theorem 4.1, ∞ error in estimated confusion matrices π as computed in Equation (7), using n samples and a predictor function f with risk R ,D ≤ δ, is bounded by DISPLAYFORM0 with probability at least 1 − δ 1 .First we apply Lemma A.1 with P π computed using majority vote. We get a bound on the risk of function f computed in the first round. With this f , we apply Lemma A.2. When n is sufficiently large such that Equation (8) holds, the denominator in Equation FORMULA18 , 1/K − δ − 8 m log(4mK 2 /δ 1 )/(nr) ≥ 1/8. Therefore , in the first round, the error in confusion matrix estimation is bounded by , which is defined in the Theorem.For the second round: we apply Lemma A.1 with P π computed as the posterior distribution (5).Where ∞ error in π is bounded by . This gives the desired bound in (9). With this f , we apply Lemma A.2 and obtain ∞ error in π bounded by 1 , which is defined in the Theorem.For the given probability of error δ in the Theorem, we chose δ 1 in both the lemma to be δ/4 such that with union bound we get the desired probability of δ. DISPLAYFORM1 For ease of notation, we denote D W,π,r by D π . Similar to R ,D , risk of decision function f with respect to the modified loss function π is characterized by the following quantities: DISPLAYFORM2 2. Empirical π -risk on samples: R π ,Dπ (f ) : DISPLAYFORM3 i , wi ). With the above definitions, we have the following, DISPLAYFORM4 DISPLAYFORM5 where (19) follows from Equation FORMULA5 . FORMULA5 follows from the fact that f is the minimizer of R π ,Dπ as computed in FORMULA12 . FORMULA5 follows from the basic excess-risk bound. V is the VC dimension of hypothesis class F, and C is a universal constant.Following shows the inequality used in Equation (19). For binary classification , we denote the two classes by Y, −Y . DISPLAYFORM6 DISPLAYFORM7 where FORMULA5 follows from Equation FORMULA5 . FORMULA5 follows from the fact that for 0-1 loss function (f (X), Y ) + (f (X), −Y ) = 1. (24) follows from the definition of β π defined in Equation (12). When π is computed using weighted majority vote of the workers then (24) holds with β π replaced by α. α is defined in (14).Following shows the equality used in Equation FORMULA5 . Using the notations ρ π and τ π , in the following, for any function f ∈ F, we compute the excess risk due to the unbiasedness of the modified loss function π .
A new approach for learning a model from noisy crowdsourced annotations. This paper proposes a method for learning from noisy labels, focusing on the case when data isn't redundantly labeled with theoretical and experimental validation This paper focuses on the learning-from-crowds problem, where jointly updating the classifier weights and the confusion matrices of workers can help on the estimation problem with rare crowdsourced labels. Proposes a supervised learning algorithm for modeling label and worker quality and utilizes algorithm to study how much redundancy is required in crowdsourcing and whether low redundancy with abundant noise examples lead to better labels.