more challenging directions. The creation of ground truth stereo and optical flow datasets [@scharstein2002taxonomy; @baker2011database] helped stimulate a flood of interest in these areas. The early evolution of object recognition datasets [@Caltech101; @Caltech256; @Dalal] facilitated the direct comparison of hundreds of image recognition algorithms while simultaneously pushing the field towards more complex problems. Recently, the ImageNet dataset [@Imagenet] containing millions of images has enabled breakthroughs in both object classification and detection research using a new class of deep learning algorithms [@Hinton; @GirshickDDM13; @OverFeat].

Datasets related to object recognition can be roughly split into three groups: those that primarily address object classification, object detection and semantic scene labeling. We address each in turn.

[**Image Classification** ]{} The task of object classification requires binary labels indicating whether objects are present in an image; see Fig. \[fig:teaser\](a). Early datasets of this type comprised images containing a single object with blank backgrounds, such as the MNIST handwritten digits [@mnist] or COIL household objects [@nene1996columbia]. Caltech 101 [@Caltech101] and Caltech 256 [@Caltech256] marked the transition to more realistic object images retrieved from the internet while also increasing the number of object categories to 101 and 256, respectively. Popular datasets in the machine learning community due 