Supervised learning problems---particularly those involving social data---are often subjective. That is, human readers, looking at the same data, might come to legitimate but completely different conclusions based on their personal experiences. Yet in machine learning settings feedback from multiple human annotators is often reduced to a single ``ground truth'' label, thus hiding the true, potentially rich and diverse interpretations of the data found across the social spectrum. We explore the rewards and challenges of discovering and learning representative distributions of the labeling opinions of a large human population. A major, critical cost to this approach is the number of humans needed to provide enough labels not only to obtain representative samples but also to train a machine to predict representative distributions on unlabeled data. We propose aggregating label distributions over, not just individuals, but also data items, in order to maximize the costs of humans in the loop. We test different aggregation approaches on state-of-the-art deep learning models. Our results suggest that careful label aggregation methods can greatly reduce the number of samples needed to obtain representative distributions. This paper explores the problem of label aggregation in domains that are highly subjective, i.e., where different annotators may disagree for perfectly legitimate reasons. Such settings are common, if underacknowledged. Though increasingly, mass media provides stories about the unintended consequences of ignoring this diversity in machine learning.For example, Beauty.ai sponsored a worldwide beauty contest, judged by a machine learning algorithm. Though light-skinned entrants made up the majority of entrants, they nonetheless won a disproportionate number of contests. BID0 Tay, a Twitter-based learning agent, developed by Microsoft, was taught to tweet that the Holocaust was made up 2 (though the Holocaust factually existed, the same cybersocial dynamics of training bias found in subjective domains led to this outcome). ProPublica discovered that Northpointe risk assessment software-used to help judges determine sentence length for convicts-recommended longer sentences for African-American men than other groups, even when controlled for confounding factors. BID2 X:2 Fig. 1 . In this example, data items (black dots) are labeled by five human annotators each (left), where color indicates label choice, yielding an empirical label distribution y i for each data item i. By clustering similarly labeled objects, we pool together (right) the labels of all data items assigned to the same cluster k into a single, much larger sample θ k for all items in the cluster. Our research suggests that, in some cases, this larger sample (or a mixture of cluster samples) is a better representation of the true population distribution of beliefs about each data item in the cluster and can lead to better predictive supervised learning.Learning a distribution of beliefs about a data item, rather than a single "ground truth" label, poses unique challenges. It increases the dimensionality of the learning problem so that more data items may be needed. It also may require more labels per item to get a representative sample of the human populations' beliefs. And for most problems, labels are relatively expensive to obtain. Though crowdsourcing platforms have made this task convenient, they are frequently a resource bottleneck in supervised learning loops.Our main contribution is a method for minimizing the number of labels needed to learn to predict socially representative label distributions. It is based on the hypothesis that the sources are subjectivity are limited, and so the number of distinct distributions of beliefs over all data items is likewise limited. In other words, the label distributions are samples from a relatively small number of true, but hidden, distributions. See Figure 1 . These hidden distributions can be seen as latent classes representing population-level beliefs about the labels. According to this hypothesis, we can use unsupervised clustering algorithms to pool together the labels of data items with similar distributions into higher resolution distributions of beliefs shared commonly among all data items in the same cluster.In particular, we: (1) explore subjectivity as the problem of learning representative distributions from a target population of responses to target questions, BID1 propose clustering as a sensible means for pooling together labels from similar data items, to reduce the number of labels needed (3) test what we call our clustering hypothesis, that the label distributions of subjective data are clustered around a small number of underlying, true distributions (4) study how different label aggregation strategies and representations affect the performance of state-of-the art deep learning predictors.It would seem that bias is an inherent part of any information reduction process, such as those found in statistical learning BID29 . So it seems naive to expect that machines can learn unbiased models through unsupervised learning alone, or even for any supervised learning that assumes a singular, correct answer to most problems. We hope that this research sparks a broader debate about the best practices for machine learning with humans in the loop.The rest of this paper is organized as follows. Section 2 describes our experimental workflow, Section 3 presents our results, Section 4 discusses our study, Section 5 presents related work, and Section 6 is the conclusion. Figure 2 describes the basic experimental workflow in this study. We discuss each phase below. Note that there are two testing phases, one for determining how well each aggregation method fits the data and another for how well supervised learning algorithms trained by each aggregation strategy perform. Since these test phases share some methods, we discuss them together at the end of the section. Fig. 2 . The basic experimental workflow involves obtaining crowdsourced labels for raw data (yielding empirical label distributions for each data item), trying various strategies for aggregating and pooling those labels (including no aggregation), and finally testing how each method affects the accuracy of machine learning prediction. Note there are two testing phases: one for how well each aggregation strategy fits the data and one for machine learning performance. We also list important terms, keywords, and abbreviations associated with each phase of the workflow.
We study the problem of learning to predict the underlying diversity of beliefs present in supervised learning domains.