Most classification and segmentation datasets assume a closed-world scenario in which predictions are expressed as distribution over a predetermined set of visual classes. However, such assumption implies unavoidable and often unnoticeable failures in presence of out-of-distribution (OOD) input. These failures are bound to happen in most real-life applications since current visual ontologies are far from being comprehensive. We propose to address this issue by discriminative detection 
 of OOD pixels in input data. Different from recent approaches, we avoid to bring any decisions by only observing the training dataset of the primary model trained to solve the desired computer vision task. Instead, we train a dedicated OOD model
 which discriminates the primary training set from a much larger "background" dataset which approximates the variety of the visual world. We perform our experiments on high resolution natural images in a dense prediction setup. We use several road driving datasets as our training distribution, while we approximate the background distribution with the ILSVRC dataset. We evaluate our approach on WildDash test, which is currently the only public test dataset with out-of-distribution images.
 The obtained results show that the proposed approach succeeds to identify out-of-distribution pixels while outperforming previous work by a wide margin. Development of deep convolutional models has resulted in tremendous advances of visual recognition. Recent semantic segmentation systems surpass 80% mIoU BID0 on demanding natural datasets such as Pascal VOC 2012 BID4 or Cityscapes BID1 . Such performance level suggests a clear application potential in exciting areas such as road safety assessment or autonomous driving. Unfortunately, most existing semantic segmentation datasets assume closed-world evaluation BID21 , which means that they require predictions over a predetermined set of visual classes. Closed-world datasets are very useful for promoting research, however they are poor proxies for real-life operation even in a very restricted scenario such as road driving. In fact, one can easily imagine many real-life driving scenes which give rise to image regions that can not be recognized by learning on the Cityscapes ontology. Some of those regions may be projected from objects which are foreign to Cityscapes (e.g. road works, water, animals). Other may appear unrelated to Cityscapes due to particular configurations being absent from the training dataset (e.g. pedestrians lying on the ground, crashed cars, fallen trees). Finally, some regions may be poorly classified due to different environmental conditions, acquisition setup, or geographical location BID23 .The simplest way to approach unrecognizable data is to improve datasets. For instance, the Vistas dataset BID16 proposes a richer ontology and addresses more factors of variation than Cityscapes. However , training on Vistas requires considerable computational resources while still being unable to account for the full variety of the recent WildDash dataset BID24 , as we show in experiments. Another way to approach this problem would be to design strategies for knowledge transfer between the training dataset and the test images BID23 . However , this is unsatisfactory for many real world applications where the same model should be directly applicable to a variety of environments.These examples emphasize the need to quantify model prediction uncertainty, especially if we wish to achieve reliable deployment in the real world. Uncertainty can be divided into two categories BID11 . Aleatoric uncertainty is caused by limitations of the model which cannot be reduced by supplying additional training data. For example, the quality of segmentation models on distant and small objects depends on the resolution on which inference is performed. On the other hand, epistemic uncertainty arises when the trained model is unable to bring the desired prediction given particular training dataset. In other words, it occurs when the model receives the kind of data which was not seen during training. Epistemic uncertainty is therefore strongly related to the probability that the model operates on an out-of-distribution sample.Recent work in image-wide out-of-distribution detection BID11 BID9 BID15 evaluates the prediction uncertainty by analyzing the model output. We find that these approaches perform poorly in dense prediction tasks due to prominence of aleatoric uncertainty. This means that total uncertainty can be high even on in-distribution pixels (e.g. on pixels at semantic borders, or very distant objects).A different approach attempts to detect out-of-distribution samples with GAN discriminators, whereby the GAN generator is used as a proxy for the out-of-distribution class BID14 BID20 . However, these approaches do not scale easily to dense prediction in high resolution images due to high computational complexity and large memory requirements.Therefore, in this paper we propose to detect out-of-distribution samples on the pixel level by a dedicated "OOD" model which complements the "primary" model trained for a specific vision task. We formulate the OOD model as dense binary classification between the training dataset and a much larger "background" dataset. The proposed formulation requires less computational resources than approaches with GAN-generated backgrounds, and is insensitive to aleatoric uncertainty related to semantic segmentation. Graceful performance degradation in presence of unforeseen scenery is a crucial capability for any real-life application of computer vision. Any system for recognizing images in the wild should at least be able to detect such situations in order to avoid disasters and fear of technology.We have considered image-wide OOD detection approaches which can be easily adapted for dense prediction in high resolution images. These approaches have delivered very low precision in our experiments because they unable to ignore the contribution of aleatoric uncertainty in the primary model output. We have therefore proposed a novel approach for recognizing the outliers as being more similar to some "background" dataset than to the training dataset of the primary model.Our experiments have resulted in a substantial improvement of OOD detection AP performance with respect to all previous approaches which are suitable for dense prediction in high resolution images. ILSVRC appears as a reasonable background dataset candidate due to successful OOD detection in negative WildDash images that are (at least nominally) not represented in ILSVRC (white wall, two kinds of noise, anthill closeup, aquarium, etc). Nevertheless, our study emphasizes the need for more comprehensive background datasets. Future work will address employing these results as a guide for better direction of the annotation effort as well as towards further development of approaches for recognizing epistemic uncertainty in images and video.Future work will address employing these results as a guide for better direction of the annotation effort as well as towards further development of approaches for recognizing epistemic uncertainty in images and video.
We present a novel approach for detecting out-of-distribution pixels in semantic segmentation. This paper addresses out-of-distribution detection for helping the segmentation process, and proposes an approach of training a binary classifier that distinguishes image patches from a known set of classes from those of an unknown. This paper aims to detect out-of-distribution pixels for semantic segmentation, and this work utilizes data from other domains to detect undetermined classes to model uncertainty better.