The learning of hierarchical representations for image classification has experienced an impressive series of successes due in part to the availability of large-scale labeled data for training. On the other hand, the trained classifiers have traditionally been evaluated on a handful of test images, which are deemed to be extremely sparsely distributed in the space of all natural images. It is thus questionable whether recent performance improvements on the excessively re-used test sets generalize to real-world natural images with much richer content variations. In addition, studies on adversarial learning show that it is effortless to construct adversarial examples that fool nearly all image classifiers, adding more complications to relative performance comparison of existing models. This work presents an efficient framework for comparing image classifiers, which we name the MAximum Discrepancy (MAD) competition. Rather than comparing image classifiers on fixed test sets, we adaptively sample a test set from an arbitrarily large corpus of unlabeled images so as to maximize the discrepancies between the classifiers, measured by the distance over WordNet hierarchy. Human labeling on the resulting small and model-dependent image sets reveals the relative performance of the competing classifiers and provides useful insights on potential ways to improve them. We report the MAD competition results of eleven ImageNet classifiers while noting that the framework is readily extensible and cost-effective to add future classifiers into the competition. Large-scale human-labeled image datasets such as ImageNet (Deng et al., 2009 ) have greatly contributed to the rapid progress of research in image classification. In recent years, considerable effort has been put to designing novel network architectures (He et al., 2016; Hu et al., 2018) and advanced optimization algorithms (Kingma & Ba, 2015) to improve the training of image classifiers based on deep neural networks (DNNs), while little attention has been paid to comprehensive and fair evaluation/comparison of their model performance. Conventional model evaluation methodology for image classification generally follows a three-step approach (Burnham & Anderson, 2003) . First, pre-select a number of images from the space of all possible natural images (i.e., natural image manifold) to form the test set. Second, collect the human label for each image in the test set to identify its ground-truth category. Third, rank the competing classifiers according to their goodness of fit (e.g., accuracy) on the test set; the one with the best goodness of fit is declared the winner. A significant problem with this methodology is the apparent contradiction between the enormous size and high dimensionality of natural image manifold and the limited scale of affordable testing (i.e., human labeling, or verifying predicted labels, which is expensive and time consuming). As a result, a typical "large-scale" test set for image classification allows for only tens of thousands of natural images to be examined, which are deemed to be extremely sparsely distributed in natural image manifold. Model comparison based on a limited number of samples assume that they are sufficiently representative of the whole population, an assumption that has been proven to be doubtful in image classification. Specifically, Recht et al. (2019) found that a minute natural distribution shift leads to a large drop in accuracy for a broad range of image classifiers on both CIFAR-10 ( Krizhevsky, 2009) and ImageNet (Deng et al., 2009) , suggesting that the current test sets may far less suffice to represent hard natural images encountered in the real world. Another problem with the conventional model comparison methodology is that the test sets are pre-selected and therefore fixed. This leaves open the door of adapting classifiers to the test images, deliberately or unintentionally, via extensive hyperparameter tuning, raising the risk of overfitting. As a result, it is never guaranteed that image classifiers with highly competitive performance on such small and fixed test sets can generalize to real-world natural images with much richer content variations. In addition, recent studies on adversarial learning (Goodfellow et al., 2015; Madry et al., 2018) indicate that adversarial images produced to mislead a specific image classifier have strong transferability to fool other classifiers even if their design philosophies differ substantially, which further complicates the relative performance comparison of existing classifiers. In order to reliably measure the progress in image classification and to fairly test the generalizability of existing classifiers in a natural setting, we believe a much larger test set in the order of millions or even billions must be used. Apparently, the main challenge here is how to exploit such a large-scale test set under the constraint of the very limited budget for human labeling, knowing that collecting ground-truth labels for all images is extremely difficult, if not impossible. In this work, we propose an efficient and practical methodology, namely the MAximum Discrepancy (MAD) competition, to meet this challenge. Instead of trying to prove an image classifier to be correct using a small and fixed test set, MAD starts with a large-scale unlabeled image set and attempts to falsify a classifier by finding a set of images, whose predictions are in strong disagreements with the rest competing classifiers (See Figure 1) . A classifier that is harder to be falsified is considered better. The initial image set for MAD to explore can be made arbitrarily large provided that the cost of computational prediction for all competing classifiers is cheap. To quantify the discrepancy between two classifiers on one image, we propose a weighted distance over WordNet hierarchy (Miller, 1998) , which is more semantically aligned with human cognition compared with traditional binary judgment (agree vs. disagree). The set of model-dependent images selected by MAD are the most informative in discriminating the competing classifiers. Subjective experiments on the MAD test set reveal the relative strengths and weaknesses among the classifiers and identify the training techniques and architecture choices that improve the generalizability to natural image manifold. Moreover, careful inspection of the selected images may suggest potential ways to improve a classifier or to combine aspects of multiple classifiers. We apply the MAD competition to compare eleven ImageNet classifiers and find that MAD verifies the relative improvements achieved by recent DNN-based methods, with a minimal subjective testing budget. MAD is readily extensible, allowing future classifiers to be added into competition with little additional cost. Moreover, the application scope of MAD is far beyond image classification. It can be extended to many other research fields that expect discrete-valued outputs, and is especially useful when the sample space is enormous and the ground-truth measurement is expensive. We have presented a new methodology for comparing image classification models. MAD effectively mitigates the conflict between the prohibitively large natural image manifold that we have to evaluate against and the expensive human labeling effort that we aim to minimize. Much of our endeavor has been dedicated to selecting natural images that are optimal in term of distinguishing or falsifying classifiers. MAD requires explicit specification of image classifiers to be compared, and provides an effective means of exposing the respective flaws of competing classifiers. It also directly contributes to model interpretability and helps us analyze the models' focus and bias when making predictions. We have demonstrated the effectiveness of MAD competition on ImageNet classifiers, and concluded a number of interesting observations, which were not apparently drawn from the (often quite close) accuracy numbers on the small and fixed ImageNet validation set. MAD is widely applicable to computational models that produce discrete-valued outputs, and is particularly useful when the sample space is large and the ground-truth label being predicted is expensive to measure. Examples include medical and hyperspectral image classification (Filipovych & Davatzikos, 2011; Wang et al., 2014) , where signification domain expertise is crucial to obtain correct labels. MAD can also be applied towards spotting rare but fatal failures in high-cost and failure-sensitive applications, e.g., comparing perception systems of autonomous cars (Chen et al., 2015) in unconstrained real-world weathers, lighting conditions, and road scenes. In addition, by restricting the test set to some domain of interest, MAD allows comparison of classifiers in more specific applications, e.g., fine-grained image recognition (Xiao et al., 2015) . We also feel it important to note the current limitations of MAD. First, MAD cannot prove a model's predictions to be correct and therefore it should be viewed as complementary to, rather than a replacement for, the conventional accuracy comparison for image classification. Second, although the distance in Eq. (3) is sufficient to distinguish multiple classifiers in the current experimental setting, it does not yet fully reflect human cognition of image label semantics. Third, the computation of the confidence used to select images is not perfectly grounded. How to marry the MAD competition with Bayesian probability theory to model uncertainties during image selection is an interesting direction for future research. Our method arises as a natural combination of concepts drawn from two separate lines of research. The first explores the idea of model falsification as a model comparison. Wang & Simoncelli (2008) introduced the maximum differentiation competition for comparing computational models of continuous perceptual quantities, which was further extended by (Ma et al., 2019) . Berardino et al. (2017) developed a computational method for comparing hierarchical image representations in terms of their ability to explain perceptual sensitivity in humans. MAD, on the other hand, is tailored to applications with discrete model responses and relies on a semantic distance measure to compute model discrepancy. The second endeavour arises from machine learning literature on generating adversarial examples (Szegedy et al., 2013; Goodfellow et al., 2015; Madry et al., 2018) and evaluating image classifiers on new test sets (Geirhos et al., 2019; Recht et al., 2019; Hendrycks & Dietterich, 2019; . The images selected by MAD can be seen as a form of natural adversarial examples as each of them is able to fool at least one classifier (when Case I is eliminated). Unlike adversarial images with inherent transferability to mislead most classifiers, MAD-selected images emphasize on their discriminability of the competing models. Different from recently created test sets, the MAD-selected test set is adapted to the competing classifiers with the goal of minimizing human labeling effort. Disk brake is correlated to freewheel and spokes. Similar with how humans recognize objects, it would be reasonable for DNN-based classifiers to make predictions by inferring useful information from object relationships, only when their prediction confidence is low. However, this is not the case in our experiments, which show that classifiers may make high-confidence predictions by leveraging object relations only without really "seeing" the predicted object. Figure 10 : Examples of network bias to low-level visual features, such as color, shape and texture, while overlooking conflicting semantic cues. An ideal classifier is expected to utilize both low-level (appearance) and high-level (semantic) features when making predictions.
We present an efficient and adaptive framework for comparing image classifiers to maximize the discrepancies between the classifiers, in place of comparing on fixed test sets. Error spotting mechanism which compares image classifiers by sampling their "most disagreed" test set, measuring disagreement through a semantics-aware distance derived form WordNet ontology.