are given in [the appendix]{}.

![image](figs/dataanalysis){width="\textwidth"}

Annotation Performance Analysis
-------------------------------

\[sec:annotation-performance\] We analyzed crowd worker quality on the category labeling task by comparing to dedicated expert workers, see Fig. \[fig:workers\](a). We compared precision and recall of seven expert workers (co-authors of the paper) with the results obtained by taking the union of one to ten AMT workers. Ground truth was computed using majority vote of the experts. For this task recall is of primary importance as false positives could be removed in later stages. Fig. \[fig:workers\](a) shows that the union of 8 AMT workers, the same number as was used to collect our labels, achieved greater recall than any of the expert workers. Note that worker recall saturates at around 9-10 AMT workers.

Object category presence is often ambiguous. Indeed as Fig. \[fig:workers\](a) indicates, even dedicated experts often disagree on object presence, e.g. due to inherent ambiguity in the image or disagreement about category definitions. For any unambiguous examples having a probability of over 50% of being annotated, the probability all 8 annotators missing such a case is at most $.5^8 \approx .004$. Additionally, by observing how recall increased as we added annotators, we estimate that in practice over 99% 