as belonging to a category, such as sky, chair, floor, street, etc. In contrast to the detection task, individual instances of objects do not need to be segmented, Fig. \[fig:teaser\](c). This enables the labeling of objects for which individual instances are hard to define, such as grass, streets, or walls. Datasets exist for both indoor [@NYUDepth] and outdoor [@shotton2009textonboost; @brostow2009semantic] scenes. Some datasets also include depth information [@NYUDepth]. Similar to semantic scene labeling, our goal is to measure the pixel-wise accuracy of object labels. However, we also aim to distinguish between individual instances of an object, which requires a solid understanding of each object’s extent.

A novel dataset that combines many of the properties of both object detection and semantic scene labeling datasets is the SUN dataset [@SUN] for scene understanding. SUN contains 908 scene categories from the WordNet dictionary [@wordnet] with segmented objects. The 3,819 object categories span those common to object detection datasets (person, chair, car) and to semantic scene labeling (wall, sky, floor). Since the dataset was collected by finding images depicting various scene types, the number of instances per object category exhibits the long tail phenomenon. That is, a few categories have a large number of 