11institutetext:  University of Freiburg, Germany
11email<EMAIL_ADDRESS>
#  Diffusion for Out-of-Distribution Detection
on Road Scenes and Beyond

Silvio Galesso∗ Philipp Schröppel∗ Hssan Driss Thomas Brox
∗Equal contribution  0009-0007-0249-6029 0000-0003-0717-6896

###### Abstract

In recent years, research on out-of-distribution (OoD) detection for semantic
segmentation has mainly focused on road scenes – a domain with a constrained
amount of semantic diversity. In this work, we challenge this constraint and
extend the domain of this task to general natural images. To this end, we
introduce 1. the ADE-OoD benchmark, which is based on the ADE20k dataset and
includes images from diverse domains with a high semantic diversity, and 2. a
novel approach that uses Diffusion score matching for OoD detection (DOoD) and
is robust to the increased semantic diversity.

ADE-OoD features indoor and outdoor images, defines 150 semantic categories as
in-distribution, and contains a variety of OoD objects. For DOoD, we train a
diffusion model with an MLP architecture on semantic in-distribution
embeddings and build on the score matching interpretation to compute pixel-
wise OoD scores at inference time. On common road scene OoD benchmarks, DOoD
performs on par or better than the state of the art, without using outliers
for training or making assumptions about the data domain. On ADE-OoD, DOoD
outperforms previous approaches, but leaves much room for future improvements.
††ADE-OoD webpage: https://ade-ood.github.io ††DOoD code:
https://github.com/lmb-freiburg/diffusion-for-ood

###### Keywords:

OoD detection Segmentation Benchmark Diffusion

##  1 Introduction

Deep neural networks achieve remarkable results when applied to their training
domain, but their robustness and reliability on out-of-distribution (OoD) data
is often inadequate. Furthermore, they typically do not know that they do not
know, _i.e_. the complexity and overconfidence of neural networks makes it
difficult to reliably detect whether inputs are out-of-distribution. In
practice, this makes it hard to decide whether a model prediction can be
trusted or not. For deployment in the real world, and especially in safety
critical applications, this is a major problem; consequently OoD detection is
an active research area.

This work falls in the category of _out-of-distribution detection for semantic
segmentation_. In this task, a semantic segmentation training dataset is
given, consisting of images and segmentation ground truth maps, which assign
each pixel to a semantic class. The set of classes in the training set are
considered in-distribution. Given an image at inference time, the task is to
detect and localize entities that do not belong to the in-distribution classes
via per-pixel out-of-distribution scores. Several benchmarks exist to evaluate
performance on this task. The most common ones – such as RoadAnomaly [28],
SegmentMeIfYouCan [6], and Fishyscapes Lost&Found [4, 36] – are designed for
the application of autonomous driving. Therefore, all samples in these
benchmarks are from the constrained domain of road scenes and define the 19
road-specific classes (road, car, _etc_.) from Cityscapes [9] as in-
distribution.

In this work, we extend this setting to _general natural images_ , which
induces a higher complexity and semantic diversity. To this end, we introduce
a novel benchmark called ADE-OoD. ADE-OoD defines the 150 classes from ADE20k
[56] as in-distribution, which comprises a larger set of commonplace classes,
for example computer, streetlight, or river. The test images in ADE-OoD
feature diverse indoor and outdoor scenes, including a high variety of OoD
objects such as kites, pizza cutters, or disk golf baskets. This poses a
challenge for OoD detection methods, which need to model a more diverse in-
distribution semantic landscape in order to successfully recognize the
outliers. Fig. 1 shows samples from the proposed ADE-OoD benchmark.

Figure 1:  Samples from the ADE-OoD benchmark. The top row shows the input
images, which feature diverse indoor and outdoor scenes. The bottom row shows
the corresponding ground truth OoD segmentations (red indicates OoD), which
contain all regions not covered by the 150 classes from ADE20k that we define
as in-distribution.  Figure 2:  Overview of DOoD. We train a diffusion model
on features extracted with a pre-trained feature extractor from in-
distribution data. The diffusion model – a small MLP – is trained on
individual feature vectors, in order to discard harmful spatial correlations.
At inference time, we compute out-of-distribution scores by perturbing the
input $\mathbf{x}_{0}$ with noise $\mathbf{\epsilon}$ via forward diffusion,
estimating the gradient of the inlier log-density $s_{\Theta}$ with the
diffusion model, and using the directional error as OoD score.

In addition to that, we introduce a novel approach for OoD detection based on
diffusion models, which works particularly well for modeling the highly
diverse inlier distribution of the task at hand.

A few recent works have applied diffusion models to out-of-distribution
detection, albeit in different settings: classification, medical anomaly
segmentation, and industrial inspection. Most of these approaches [17, 30, 16,
48, 51, 54, 32, 15, 49] rely on reconstruction of the input data. This
requires sequential application of multiple reverse diffusion process steps,
which results in long runtimes. A few approaches [40, 31] build on the
connection of diffusion models and score matching and compute OoD scores based
on the estimated score from the diffusion model. In particular, Shin  _et al_.
[40] perturb the input data, estimate the score with the diffusion model, and
use the mean-squared error between perturbations and estimated scores as OoD
scores.

We follow a similar strategy, but propose to use the directional error between
perturbations and estimated scores as OoD scores. We validate this choice with
experiments and analyses, showing that it consistently outperforms previous
ones. Further, instead of images, we apply the diffusion model to the latent
distribution of features from _deep segmentation embedding spaces_ , which
have been shown to be well-suited for OoD detection [13]. Given the high
spatial correlation between segmentation features – which is not desirable for
out-of-distribution detection – we replace the common 2D U-Net with an MLP and
_learn to reconstruct individual features instead of spatial feature maps_.
This way, the diffusion model is faster and cannot easily reconstruct
anomalies by extrapolation.

We evaluate DOoD on the commonly used RoadAnomaly, Fishyscapes, and SMIYC-
Anomaly benchmarks, and on the proposed ADE-OoD benchmark. With the
evaluations on these benchmarks, we show that our approach can detect and
localize novel objects competitively or better than the state of the art. Our
approach is especially effective when the novel objects are semantically
anomalous, and when the test data has a large domain shift, such as in
RoadAnomaly and ADE-OoD. This validates that the proposed adaptations make
diffusion models effective for OoD detection on road scenes and beyond.

##  2 Related Work

In the following, we provide an overview of benchmarks and approaches for OoD
detection in semantic segmentation data, and of diffusion model based
approaches for other OoD detection tasks.

###  2.1 Out-of-Distribution Detection for Semantic Segmentation

Detecting semantic out-of-distribution objects in segmentation data has risen
in popularity in recent years, especially in the context of autonomous
driving.

Methods. Generative models have been used for out-of-distribution detection in
many settings. In semantic segmentation, some approaches use GANs for image
re-synthesis, and use reconstruction quality as a proxy for anomaly scores
[28, 52, 3]. Due to the implicit nature of GANs, these methods require
comparison networks trained with proxy outliers, whereas our method can
exploit the score estimate from the diffusion model.

A similar direction, but with non-parametric retrieval instead of parametric
generation, is taken by cDNP [13], which shows that strong OoD detection
results can be obtained with nearest-neighbor distances in certain deep
segmentation embedding spaces. We build on their findings but improve upon it
by using diffusion models as parametric density estimators instead of non-
parametric kNNs.

Many recent methods for dense OoD detection use the Mask2Former [8]
segmentation model. Its separate mask/class prediction system is equipped with
a rejection category, which was found to yield high quality anomaly scores
[33, 37, 1, 20], especially when exposed to outliers during training [21].
These works belong to a widespread practice of exploiting the prediction
confidence of semantic segmentation models as a proxy score for anomaly [24,
7, 18, 14, 5, 19, 45, 27, 12, 34].

Some approaches specifically target the driving domain: notably, JSRNet [47]
achieves strong results on driving benchmarks by modelling the road surface.
In contrast to ours, such methods are not applicable to other domains.

Benchmarks. Commonly used benchmarks are RoadAnomaly [28], SegmentMeIfYouCan
[6], and Fishyscapes Lost&Found [4, 36]. All of them focus on road scenes,
which are limited in terms of scene structure and semantic categories (see
overview in Sec. 3). Our proposed ADE-OoD benchmark uses the ontology of
ADE20k, a large scale scene segmentation dataset, to overcome this constraint.

###  2.2 Out-of-Distribution Detection with Diffusion Models

Multiple recent works have used diffusion models for OoD detection [17, 30,
16, 48, 51, 54, 32, 15, 49, 40, 31]. The approaches differ in terms of
downstream task and use of the diffusion model to derive OoD scores.

Downstream Tasks. One set of works [17, 30, 16, 15] applies diffusion models
for OoD classification, where the task is to classify whole images at
inference time as out-of-distribution. The approach DFDD [49] applies
diffusion to OoD object detection, where the task is to detect OoD objects in
form of bounding boxes.

Another set of works [48, 51, 54, 32, 15] applies diffusion models for OoD
segmentation, where the task is to predict a pixel-wise segmentation of OoD
regions. All of these works are either in a medical or an industrial
inspection domain.

In this work, we present the novel approach for OoD segmentation on semantic
segmentation data. In this task, a training dataset of images with semantic
segmentation annotations is given. At inference time, the task is to segment
OoD regions in images, where OoD is defined as not belonging to the set of
training classes. Our setting differs from industrial and medical anomaly
detection for two main reasons: the larger size of the input images
(unsuitable for direct application of diffusion models), and the higher visual
diversity and scene complexity which makes it more challenging to model the
in-distribution data.

OoD Score Computation. Most related works compute OoD scores based on
reconstruction errors [17, 30, 48, 51, 54, 32, 15]. This is done by corrupting
the input image, reconstructing it with the diffusion model, and using the
error between the input and the reconstruction as OoD score. Goodier  _et al_.
[16] estimate the likelihood of given input data and use it as OoD score. Most
related to our work, are methods [40, 31] that avoid expensive input
reconstructions by leveraging the theory of denoising score matching [23, 46,
42] and deriving OoD scores from estimated scores. Shin  _et al_. [40] do this
by perturbing the input data with Gaussian noise, estimating the score for the
perturbed data point, and using its $L_{2}$ distance to the perturbation as
OoD score. Mahmood  _et al_. [31] estimate the score for given data points and
use the norm of the estimated score. In contrast to this, our approach
computes OoD scores based on the directional difference of the estimated score
and the perturbation.

##  3 ADE-OoD: OoD Detection Beyond Road Scenes

In this section we introduce ADE-OoD, a novel benchmark for OoD detection on
semantic segmentation data. The key characteristic of the proposed benchmark
is higher diversity. It stands out by a much larger set of in-distribution
categories than the typically used 19 Cityscapes classes. Furthermore, it
contains images from more diverse indoors and outdoors scenes.

####  3.0.1 Benchmark Overview.

The benchmark uses the training set of the ADE20k [56] dataset, with its 150
categories, to define the in-distribution domain; for this reason, the
benchmark is named ADE-OoD. It consists of 111 real-world images, each
containing in-distribution and out-of-distribution segments. For each image, a
binary OoD mask is provided. A few examples are shown in Fig. 1 and Fig. 3. A
comparison to other established benchmarks is provided in the following table:

| Road- | Fishyscapes | Fishyscapes | SMIYC | SMIYC | ADE-OoD
---|---|---|---|---|---|---
| Anomaly | L&F | Static | Anomaly | Obstacle | (Ours)
Type | Real | Real | Synthetic | Real | Real | Real
(pasted cutouts) | (road only) |
Domain | Driving | Driving | Driving | Driving | Driving | In & outdoor
Num. Samples | 60 | 100+275 | 30+1000 | 10+100 | 327+30 | 111
In-dist. classes | 19 | 19 | 19 | 19 | 1 | 150

####  3.0.2 Benchmark Construction.

We obtained the images for the benchmark from the validation set of ADE20k and
from OpenImages [26]. The images were selected according to the following
criteria:  (1) presence of clear and unambiguous in- and out-of-distribution
entities only,  (2) presence of diverse in- and out-of-distribution
categories, as well as varying indoor and outdoor settings  (3) adequate scene
complexity to match the original ADE20k training data,  (4) collecting an
adequate number of samples for statistically significant results . For
annotation, we have set up a semi-automatic tool, based on the popular
promptable segmentation model SAM [25].

##  4 Diffusion Models Background

In this section, we provide background information on score matching and
diffusion models. Both are relevant for our approach DOoD, as we build on the
score matching interpretation of diffusion models to compute OoD scores.

###  4.1 Denoising Score Matching

Let $q(\mathbf{x})$ be an unknown data distribution and
$\\{\mathbf{x}_{i}\in\mathbb{R}^{D}\\}_{i=1}^{N}$ a dataset of samples from
that distribution. Let $p(\mathbf{x},\Theta)$ be a probability density model
that should fit the data distribution. Score matching [23] is an approach to
learn the parameters $\Theta$ such that the score
$\nabla_{\mathbf{x}}\log{p(\mathbf{x},\Theta)}$ of the model distribution
matches the score $\nabla_{\mathbf{x}}\log{q(\mathbf{x})}$ of the data
distribution. In the following, we denote the score of the model distribution
as
$s_{\Theta}(\mathbf{x})\coloneqq\nabla_{\mathbf{x}}\log{p(\mathbf{x},\Theta)}$.

As the score of the data distribution is usually not known, techniques exist
to match the score based on the given finite set of data samples [23, 46].
Denoising score matching [46] is one such technique. It perturbs data points
$\mathbf{x}$ according to a noise distribution
$q(\tilde{\mathbf{x}}\mid\mathbf{x})$ and matches the score of the resulting
perturbed distribution $q(\tilde{\mathbf{x}})=\int
q(\tilde{\mathbf{x}}\mid\mathbf{x})q(\mathbf{x})\mathrm{d}\mathbf{x}$. It has
been shown that matching the score of the perturbed distribution, amounts to
matching the score $\nabla_{\tilde{\mathbf{x}}}\log
q(\tilde{\mathbf{x}}\mid\mathbf{x})$ [46].

Typically, Gaussian noise is used as perturbation:
$q(\tilde{\mathbf{x}}\mid\mathbf{x})=\mathcal{N}(\tilde{\mathbf{x}}\mid\mathbf{x},\sigma^{2}\mathbf{I})$
[46, 42, 43]. Intuitively, this amounts to matching the score of a smoothed
data distribution where the amount of smoothing depends on the chosen
$\sigma$. With $\nabla_{\tilde{\mathbf{x}}}\log
q(\tilde{\mathbf{x}}\mid\mathbf{x})=-(\tilde{\mathbf{x}}-\mathbf{x})/\sigma^{2}$,
this leads to the following objective:

$\hat{\Theta}=\operatorname*{arg\,min}_{\Theta}\mathbb{E}_{q(\tilde{\mathbf{x}})}\bigg{[}\frac{1}{2}\left\lVert
s_{\Theta}(\tilde{\mathbf{x}})-\frac{-(\tilde{\mathbf{x}}-\mathbf{x})}{\sigma^{2}}\right\rVert_{2}^{2}\bigg{]}\,.$
(1)

Note that with a reparameterization
$\tilde{\mathbf{x}}=\mathbf{x}+\sigma\mathbf{\epsilon}$ with
$\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, one can reformulate
this as matching the score $s_{\Theta}(\tilde{\mathbf{x}})$ to
$-\mathbf{\epsilon}/\sigma$.

In the following, we provide background on Gaussian diffusion models, which
can be interpreted as denoising score matching over Gaussian perturbations of
different smoothing levels [22].

###  4.2 Diffusion Models and their Relation to Score Matching

Diffusion models [41] are a class of generative models that learn a data
distribution $q(\mathbf{x}_{0})$ by defining a forward diffusion process in
form of a Markov chain with steps $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})$ that
gradually transform the data distribution into a simple known distribution. A
model $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ with parameters $\theta$
is then trained to approximate the steps $q(\mathbf{x}_{t-1}|\mathbf{x}_{t})$
in the Markov chain of the reverse process.

In case of Gaussian diffusion models, the forward diffusion process gradually
replaces the data with Gaussian noise following a noise schedule
$\beta_{1},..,\beta_{T}$:

$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})\coloneqq\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})\,.$
(2)

Following this, $q(\mathbf{x}_{t}|\mathbf{x}_{0})$ is also a Gaussian that is
parameterized as:

$q(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I})\,\text{
with }\,\alpha_{t}\coloneqq 1-\beta_{t}\,\text{ and
}\,\bar{\alpha}_{t}\coloneqq\prod_{s=1}^{t}\alpha_{s}\,.$ (3)

DDPM [22] reparametrizes Eq. 3 to

$\mathbf{x}_{t}(\mathbf{x}_{0},\mathbf{\epsilon})=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}\text{
with }\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (4)

and trains a model $\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ to predict
the noise $\mathbf{\epsilon}$ with the following loss:

$\hat{\Theta}=\operatorname*{arg\,min}_{\Theta}\mathbb{E}_{\mathbf{x}_{0},\mathbf{\epsilon},t}\bigg{[}\left\lVert\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)-\mathbf{\epsilon}\right\rVert_{2}^{2}\bigg{]}\,.$
(5)

Based on the similarity of this loss and the denoising score matching
objective in Equation 1, an estimated
$\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ can be interpreted as estimated
score for the smoothed data distribution $q(\mathbf{x}_{t})$ that arises at
the diffusion timestep $t$:

$s_{\Theta}(\mathbf{x}_{t})=\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})=-\frac{\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)}{\sigma_{t}}\quad\text{with
}\,\sigma_{t}=\sqrt{1-\bar{\alpha}_{t}}\,.$ (6)

In this work, we train a diffusion model on in-distribution data. The
estimated score is hence trained to point towards regions of high-probability
in smoothed inlier distributions, where different diffusion timesteps
correspond to different amounts of smoothing. At inference time, we perturb
given data $\mathbf{x}_{0}$ for different diffusion timesteps $t$ via
$\mathbf{x}_{t}(\mathbf{x}_{0},\mathbf{\epsilon})=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}$
from Equation 4, estimate the score $s_{\theta}(\mathbf{x}_{t})$, and use the
directional difference of the estimated score $s_{\theta}(\mathbf{x}_{t})$ and
the perturbation $\mathbf{\epsilon}$ as OoD score.

##  5 DOoD: Diffusion Score Matching for OoD Detection

In the following, we describe DOoD, our proposed method for the detection and
localization of out-of-distribution entities in semantic segmentation
scenarios. As illustrated in Fig. 2, the method consists of two steps:  (1)
feature extraction and  (2) OoD score estimation . In the first step, we use a
semantic segmentation model trained on in-distribution data to extract feature
maps from the input images (Sec. 5.1). For estimating the OoD scores, we train
a diffusion model on the features extracted from the training dataset (Sec.
5.2). At inference time, we use this diffusion model to estimate the OoD
scores (Sec. 5.3).

###  5.1 Feature Extraction

To obtain feature representations suitable for OoD detection on segmentation
data, we apply a feature extractor that was pretrained on in-distribution data
of the given segmentation task. More specifically, we build on the findings
from [13], which show that the self-attention features of transformer encoders
induce meaningful distances between in-distribution and out-of-distribution
features. In particular, we employ the key features of the ViT [10] and MiT
[53] used as encoders in semantic segmentation models. These are trained with
a standard supervised loss on the respective predictions from the Segmenter
[44] or SETR [55] decoders.

After the feature extraction, our approach operates on dense feature maps of
dimensions $\mathbf{F}\in\mathbb{R}^{H{\times}W{\times}C}$, _i.e_. each
feature vector has dimensionality $C$ and corresponds to one of the
$H{\cdot}W$ square patches of the original image.

###  5.2 Diffusion Model Architecture and Training

In the second stage of the method, we train a diffusion model on the features
extracted from the training dataset of the in-distribution semantic
segmentation data. In the following, we describe the design of the diffusion
model in terms of the denoiser architecture and the training procedure.

####  5.2.1 Architecture.

Typical diffusion models for image generation use a convolutional 2D U-Net
[38] architecture for the denoiser network [22, 35]. This architecture works
well for image generation, because it embeds spatial context information in
the sampling process. For our application, however, information from
neighboring features is harmful. This is because our approach relies on the
diffusion model failing to estimate the score for OoD features; if the model
can exploit the strongly correlated neighboring features it can accomplish
this task regardless of whether the input is anomalous or not (as validated in
Fig. 5).

As a remedy, we propose to use a MLP-based denoiser network that has an input
dimensionality of $C$ and operates on each local feature individually. Our
diffusion model hence models the distribution $q(\mathbf{x}_{0})$ of
individual features $\mathbf{x}_{0}\in\mathbb{R}^{C}$. This is illustrated in
Figure 2. The architecture of our MLP denoiser network has the same structure
of residual and skip connections as the U-Net architecture in [35]. However,
it replaces the convolutional layers with linear ones, maintains a constant
hidden dimensionality, uses no attention layers, and uses a simplified time
embedding. Further details are provided in the Appendix.

####  5.2.2 Training.

Training of the denoiser is done with a DDPM formulation, _i.e_. with the same
noise schedule, model parametrization, and loss. We first extract feature maps
$\mathbf{F}$ from all images in the training dataset and disassemble them into
individual features $\mathbf{x}_{0}$. Those features represent samples of the
inlier feature distribution $q(\mathbf{x}_{0})$. We compute statistics across
all features and use it for normalization to a range from $-1$ to $1$ before
feeding data to the diffusion model. We use the same statistics for
normalization at inference time.

During the training, the features are perturbed with noise
$\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to
$\mathbf{x}_{t}(\mathbf{x}_{0},\mathbf{\epsilon})$ as stated in Eq. 4, and the
denoiser is trained to estimate the perturbation noise
$\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ using the MSE loss from Eq. 5.
As described in the Background section, this can be interpreted as the
denoiser learning to estimate the gradient of the smoothed data distribution
$\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)=-\sigma_{t}\nabla_{\mathbf{x}_{t}}\log
p(\mathbf{x}_{t})$ and the score $s_{\Theta}(\mathbf{x}_{t})$ can be computed
from the model output via
$s_{\Theta}(\mathbf{x}_{t})=-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)/\sigma_{t}$.

###  5.3 OoD Score Computation

####  5.3.1 OoD Score Computation for a Single Diffusion Timestep.

At inference time, given an input image, the goal is to estimate pixel-wise
OoD scores. First, we extract the feature map $\mathbf{F}$ from the image and
normalize it as described above. We disassemble the feature map to individual
features $\mathbf{x}_{0}$ and compute an OoD score for each feature
individually.

For computing the OoD score of a single feature $\mathbf{x}_{0}$, we perturb
the feature by applying the forward diffusion process up to a timestep $t$:
$\mathbf{x}_{t}(\mathbf{x}_{0},\mathbf{\epsilon})=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}\text{
with }\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (same as Eq.
3). We feed this perturbed feature
$\mathbf{x}_{t}(\mathbf{x}_{0},\mathbf{\epsilon})$ to our denoiser to estimate
the score
$s_{\Theta}(\mathbf{x}_{t})=-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)/\sigma_{t}$
with $\sigma_{t}=\sqrt{1-\bar{\alpha}_{t}}$. Intuitively, this score points
towards higher probability in the smoothed data distribution of the respective
timestep. Based on this, we compute the OoD score $e_{\measuredangle,t}$ as
directional error (_i.e_. cosine similarity) between the estimated score
$s_{\Theta}(\mathbf{x}_{t})$ and the perturbation $\mathbf{\epsilon}$:

$e_{\measuredangle,t}=\frac{s_{\Theta}(\mathbf{x}_{t})\cdot{\epsilon}}{\lVert
s_{\Theta}(\mathbf{x}_{t})\rVert\,\lVert{\epsilon}\rVert}=\frac{-\frac{1}{\sigma_{t}}\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\cdot{\epsilon}}{\lVert-\frac{1}{\sigma_{t}}\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\rVert\,\lVert{\epsilon}\rVert}=\frac{-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\cdot\epsilon}{\lVert\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\rVert\,\lVert\epsilon\rVert}\,.$
(7)

Finally, we reassemble the OoD scores $e_{\measuredangle,t}$ of each
individual feature to an OoD score map $\mathbf{E}_{\measuredangle,t}$. In
practice, the OoD scores for all features are computed in parallel as a single
batch.

####  5.3.2 OoD Score Aggregation Across Timesteps.

We compute OoD score maps for multiple diffusion timesteps, _i.e_. we compute
$\mathbf{E}_{\measuredangle,t}$ for each
$t\in\mathcal{T}=\\{t_{1},...,t_{N}\\}$. As the perturbations
$\sqrt{1-\bar{\alpha}_{t}}\mathbf{\epsilon}$ in each timesteps have different
expected magnitudes $\sqrt{1-\bar{\alpha}_{t}}$, we weight directional errors
in different timesteps accordingly. The final OoD score map
$\mathbf{E}_{\measuredangle}$ is therefore computed as a weighted sum as
follows:

$\mathbf{E}_{\measuredangle}=\sum\limits_{t\in\mathcal{T}}\sqrt{1-\bar{\alpha}_{t}}\mathbf{E}_{\measuredangle,t}\,.$
(8)

In practice, again, the OoD score maps $\mathbf{E}_{\measuredangle,t}$ of the
different timesteps are computed in parallel as a single batch.

Regarding reconstruction-based approaches, the advantage of this OoD score
computation is that all steps can be parallelized, which enables much faster
runtimes. The difference to previous score-based methods is that here the OoD
scores are computed from directional errors and aggregated across multiple
timesteps. In Sec. 6.4 we show experimentally that this improves performance
consistently.

####  5.3.3 Compound OoD Scores.

Following [13], we average the OoD scores $\mathbf{E}_{\measuredangle,t}$
obtained from the diffusion model with the uncertainty scores of the
segmentation model (in the form of the LogSumExp of the predicted logits). The
average is computed after standardization according to means and standard
deviations obtained on the training set.

##  6 Experiments

In this section, we provide experimental results for the presented method and
for the proposed benchmark. In Sec. 6.1, we compare our method to the state-
of-the-art in OoD detection for semantic segmentation on road scene
benchmarks, which are the current standard in the field. In Sec. 6.2, we
provide evaluations on the proposed ADE-OoD benchmark. Following that, we
validate the principal design choices of our approach, namely the use of an
MLP architecture for the denoiser network (Sec. 6.3), and the proposed OoD
score computation (Sec. 6.4).

####  6.0.1 Experimental Setup.

To train the segmentation models for feature extraction, we follow [13] or use
off-the-shelf pre-trained parameters whenever available. For the MLP
architecture of our diffusion model, we use a hidden dimension equal to the
input feature channel size. We use a learning rate of $5e\\!\\!-\\!\\!5$ and a
batch size of 4096. We train for 70k iterations, which takes ca. 5 hours on an
RTX 3090 GPU. We aggregate OoD scores across the last 25 timesteps, _i.e_.
$\mathcal{T}=\\{1,...,25\\}$. Please note that we use the same experimental
settings across all benchmarks.

We report the standard evaluation metrics Average Precision (AP) and FPR at
$95\%$TPR (FPR). As the results vary strongly across datasets, we additionally
report the average AP and FPR rank across benchmarks for each method.

###  6.1 Comparing DOoD to the State-of-the-Art on Road Scenes

In this section we evaluate DOoD on the common driving-oriented benchmarks
RoadAnomaly [28], Fishyscapes[4, 36] validation, and SegmentMeIfYouCan [6]
(SMIYC). In all cases, the Cityscapes dataset is used for training and the
Cityscapes categories are considered as in-distribution.

####  6.1.1 Compared Methods.

We compare with a diverse set of state-of-the-art methods. Notably, many are
similarly based on Mask2Former architecture (Rba, M2A, EAM, Maskomaly). Our
evaluation includes approaches that use external datasets as proxy outliers
for negative training (Outlier Exposure/OE) and approaches which are designed
specifically for driving (DS) scenarios. These methods build on data or
assumptions that make them less general (as detailed in Sec. 6.2) and
therefore not directly comparable. For example the DS method M2A suppresses
OoD scores for all “stuff” classes except “road”, and JSRNet explicitly models
the road surface. We therefore included them in the evaluation for
completeness, but excluded them from the ranking.

The compared methods build on diverse segmentation models. Following [13], we
apply our approach on the representation spaces of two different segmentation
model architectures: Segmenter-B and SETR-L. These are popular high-
performance models with pre-trained Cityscapes weights available off-the-
shelf. Since SMIYC is submission-based, we report results for one model per
dataset.

####  6.1.2 Results.

Quantitative results are provided in Tab. 1. DOoD performs as good or better
than the state of the art on the RoadAnomaly, SMIYC-Anomaly, and Fishyscapes
Static benchmarks. Further, on RoadAnomaly and SMIYC-Anomaly, our approach
challenges most methods that are driving-specific or use outlier exposure,
even though they require additional data or are less general.

On Fishyscapes Lost&Found the performance of our method varies depending on
which segmentation model is used as feature extractor. With the SETR-L feature
extractor, our approach is competitive on the AP metric with all comparable
approaches. We attribute the higher FPR metric of our approach to the fact
that our estimated OoD score maps are of lower resolution (as we operate on
transformer feature maps), which is unfavorable with the small-sized obstacles
featured in Lost&Found, resulting in a higher FPR. The same applies to SMIYC-
Obstacle, which also features small-sized anomalies.

Overall, we observe that no method is capable to prevail on all benchmarks;
the ones which score excellently on some, perform poorly on others. In
comparison with others, the results of DOoD are more consistent and give best
average performance across all benchmarks, as indicated by the Average Rank.

Table 1:  State-of-the-art comparison on common road scene benchmarks. While
no approach is superior on all benchmarks and metrics, our approach DOoD often
performs better or on-par with the state-of-the-art, which is indicated by the
strong Average Rank across benchmarks. “OE” (Outlier Exposure) and “DS”
(Driving Specific) methods have higher data requirements or stricter
assumptions, which makes them less general and therefore not directly
comparable (as shown in Sec. 6.2). Nevertheless, our approach is competitive
with them on RoadAnomaly and SMIYC-Anomaly.

|  |  | Road- | Fishysc. | Fishysc. | SMIYC | SMIYC LF | SMIYC | Average
---|---|---|---|---|---|---|---|---|---
|  |  | Anomaly | L&F val. | Static val. | Anomaly | NoKnown | Obstacle | Rank
Method | OE | DS | AP | FPR | AP | FPR | AP | FPR | AP | FPR | AP | FPR | AP | FPR | AP | FPR
GMMSeg [27] |  |  | 57.7 | 44.3 | 50.0 | 12.6 | - | - | - | - | - | - | - | - | 5.0 | 4.5
NFlowJS [18] |  |  | - | - | 39.4 | 9.0 | - | - | 56.9 | 34.7 | 89.3 | 0.7 | 85.5 | 0.4 | 3.5 | 2.0
Maskomaly [1] |  |  | 80.8 | 12.0 | - | - | 68.8 | 15.0 | 93.4 | 6.9 | - | - | - | - | 2.67 | 2.67
RbA [33] |  |  | 78.5 | 11.8 | 61.0 | 10.6 |  †59.1 |  †17.7 | 86.1 | 15.9 | - | - | 87.9 | 3.3 | 3.6 | 3.4
cDNP [13] |  |  | 85.6 | 9.8 | 66.2 | 27.8 |  †63.1 |  †19.3 | 88.9 | 11.4 | - | - | 72.7 | 1.4 | 2.8 | 3.2
EAM [20] | ✓ |  | 69.4 | 7.7 | 81.5 | 4.2 | - | - | 93.8 | 4.1 | - | - | 92.9 | 0.5 | - | -
M2A [37] | ✓ | ✓ | 79.7 | 13.5 | 46.0 | 4.4 | 90.5 | 1.2 | 88.7 | 14.6 | - | - | 92.9 | 0.5 | - | -
RbA-OE [33] | ✓ |  | 85.4 | 6.9 | 70.8 | 6.3 | 75.5 | 3.5 | 94.5 | 4.6 | - | - | 95.1 | 0.1 | - | -
RPL [29] | ✓ |  | 71.6 | 17.7 | 70.6 | 2.5 | 92.5 | 0.9 | 83.5 | 11.7 | - | - | 85.9 | 0.6 | - | -
JSRNet [47] |  | ✓ | 94.4 | 9.2 | - | - | - | - | 33.6 | 43.9 | 74.2 | 6.6 | 28.1 | 28.9 | - | -
Ours Segm.-B |  |  | 89.1 | 8.8 | 43.0 | 30.2 | 75.0 | 6.2 | 90.8 | 8.4 | - | - | - | - | 2.5 | 2.5
Ours SETR-L |  |  | 88.6 | 13.1 | 64.4 | 27.7 | 79.3 | 6.5 | - | - | 79.5 | 3.7 | 64.3 | 2.6 | 2.2 | 3.4

  * •

Ranking: best, second, third $\dagger$ Computed using official code and model.

###  6.2 ADE-OoD Evaluation

In this section we provide evaluations on the proposed ADE-OoD benchmark.

####  6.2.1 Evaluated Methods.

We report results for a diverse set of methods, including the hybrid
generative-discriminative GMMSeg, the Mask2Former based RbA and M2A, and cDNP.
For M2A category partitioning we follow ADE20k’s stuff/things split. We use
off-the-shelf ADE20k weights for all methods, matching the architectures used
in the respective paper for driving data. We use SETR-ViT-L for our approach,
as it is compatible to our previous experiments and available for download.
Note that it is difficult to evaluate many methods that use outlier exposure
or are specific to driving scenes, as they require additional outlier data or
their assumptions do not apply.

Figure 3: ADE-OoD Evaluation. In the Table we provide results on the proposed
benchmark for a diverse set of approaches, including parametric generative
(GMMSeg), retrieval-based (cDNP), and Mask2Former-based (RbA, M2A, Maskomaly).
Our approach DOoD performs second best in terms of AP, and best in terms of
FPR. Overall there is much room for improvement for all methods, which
testifies the challenge posed by ADE-OoD. In the Figure we show three samples
from the benchmark, along with OoD score maps from GMMSeg, RbA, and our
approach.

Method | AP | FPR
---|---|---
M2A | 18.23 | 86.53
Maskomaly | 28.23 | 82.68
GMMSeg | 47.6 | 43.5
cDNP | 62.35 | 39.20
RbA | 66.82 | 82.42
Ours | 63.03 | 36.50

Image

OoD Mask

GMMSeg

RbA

Ours

####  6.2.2 Results.

Quantitative results are reported in the Table in Fig. 3. In terms of AP, the
best approaches are RbA and our approach DOoD, followed by cDNP. M2A has a
lower performance: its strategy of restricting its scope to foreground classes
works well in the driving domain, but is undesirable here. In terms of FPR our
approach performs best, and surprisingly RbA ranks last.

In Fig. 3 we show some samples from the benchmark, along with predicted OoD
scores from GMMSeg, RbA and our method. Neither approach perfectly segments
the anomalies, showcasing the difficulty of the benchmark. The examples show
how RbA’s predictions are smooth within objects, but they incur in more
critical failure cases, such as the false negatives in the last example. On
the other hand, our method’s outputs are of lower resolution and noisier.

###  6.3 Analysis of the Diffusion Model Architecture

As described in Sec. 5.2, we use an MLP architecture for the diffusion model.
In Fig. 5, we provide a comparison to the commonly used U-Net architecture.

Image

G.T.

U-Net

MLP

| RoadAnomaly | FS-L&F
---|---|---
Arch. | AP | FPR | AP | FPR
U-Net | 67.3 | 19.8 | 25.9 | 43.0
MLP | 84.7 | 15.3 | 31.5 | 41.1

Figure 5:  Comparison of MLP and U-Net diffusion model architectures.

The OoD scores from the U-Net architecture can be correct on boundaries of OoD
objects, but too low within the object. We attribute this to the receptive
field of the U-Net, which allows it to extrapolate information from neighbor
features and as such successfully denoise OoD inputs, which is not desirable.
The MLP architecture operates on each feature individually and does not suffer
from this problem. Indeed, the proposed MLP architecture outperforms the
common U-Net architecture qualitatively and quantitatively by a large margin.

###  6.4 Analysis of the OoD Score Computation

As described in Sec. 2.2, we propose to compute OoD scores from the
directional error between a perturbation and the estimated score. Moreover, we
propose to aggregate OoD scores across multiple timesteps. A natural question
that arises is which timesteps should be used for the aggregation. Related
works either face similar questions [17, 54, 16], or work with a fixed noise
magnitude [40].

In the following, we analyze these two aspects, which shows that the proposed
OoD score $\mathbf{E}_{\measuredangle}$ outperforms others and is more robust
to the choice of timesteps.

Setup. We compare three different OoD scores. For all scores, the input
feature $\mathbf{x}_{0}$ is first perturbed with noise $\mathbf{\epsilon}$ to
$\mathbf{x}_{t}$ via the forward diffusion process up to a timestep $t$. The
OoD scores are computed from the model output
$\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ as:

  * •

$\text{MSE}_{\text{recon}}$: applies reverse diffusion process to obtain a
reconstructed $\hat{\mathbf{x}}_{0}$, uses
$\text{MSE}(\mathbf{x}_{0},\hat{\mathbf{x}}_{0})$ as OoD score [51]. Requires
$t$ _sequential_ forward passes.

  * •

$\text{MSE}_{\text{score}}$: uses
$\text{MSE}(\mathbf{\epsilon},\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t))$
as OoD score [40]. Requires 1 forward pass.

  * •

$\mathbf{E}_{\measuredangle,t}$: uses
$\frac{-\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\cdot\epsilon}{\lVert\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\rVert\,\lVert\epsilon\rVert}$
as OoD score (Ours). Requires 1 forward pass.

To analyze the effects of different diffusion timesteps, we report
quantitative results APt=1 for using only the first timestep and
AP${}_{\text{best}}$ for the best timestep.

We report results in Fig. 6 and analyze them in the following.

Figure 6: Analysis of OoD scores and diffusion timesteps. In the table, we
report the AP on three benchmarks for different ways to compute OoD scores and
different diffusion timesteps. In the figure, we plot the APt on RoadAnomaly
for the different OoD scores depending on the timestep.

| RoadAnomaly | FS-Static | ADE-OoD
---|---|---|---
|  APt=1 |  AP${}_{\text{best}}$ |  APt=1 |  AP${}_{\text{best}}$ |  APt=1 |  AP${}_{\text{best}}$
$\text{MSE}_{\text{recon}}$ | 52.5 | 84.6 | 58.7 | 71.1 | 32.5 | 42.4
$\text{MSE}_{\text{score}}$ | 78.9 | 86.7 | 74.2 | 75.0 | 49.2 | 49.2
$\mathbf{E}_{\measuredangle}$ (Ours) | 86.7 | 86.7 | 76.5 | 76.5 | 50.4 | 50.4

Comparison of OoD scores: Overall, we observe that OoD scores
$\text{MSE}_{\text{score}}$ and $\mathbf{E}_{\measuredangle}$, computed from
the score estimate of the diffusion model, outperform the reconstruction-based
OoD score $\text{MSE}_{\text{recon}}$. Further, we observe a consistently
better performance of the proposed $\mathbf{E}_{\measuredangle}$ score
compared to the $\text{MSE}_{\text{score}}$ score.

Effect of the diffusion timestep: We observe that the performance of the
$\text{MSE}_{\text{score},t}$ score worsens sensibly for smaller timesteps,
showing that it requires a carefully chosen timestep. The proposed
$\mathbf{E}_{\measuredangle,t}$ score has a better and more consistent
performance.

###  6.5 Computational Costs

We provide a comparison of computational costs on the RoadAnomaly benchmark
for cDNP, RbA, and our approach (all on a RTX 3090 GPU):

Method | Time (s) | Mem. (GB) | AP (RA)
---|---|---|---
cDNP Segm.-B | 0.170 | 4.58 | 85.6
RbA | 0.124 | 1.30 | 78.5
Ours Segm.-B $|\mathcal{T}|=5$ | 0.125 | 1.87 | 86.1
Ours Segm.-B $|\mathcal{T}|=10$ | 0.126 | 1.87 | 87.9
Ours Segm.-B $|\mathcal{T}|=25$ | 0.322 | 1.87 | 89.1

One can observe that RbA, whose runtime basically consists of a Mask2Former
forward pass, is the cheapest. Our approach, while having a higher inference
time than cDNP, it has a smaller GPU memory footprint. An advantage of our
approach is that it offers the possibility to trade of runtime and performance
via changing the number of diffusion timesteps that are aggregated in the OoD
score computation. For example, aggregating five timesteps
$\mathcal{T}=\\{1,...,5\\}$ enables a comparable runtime as previous methods
but with better performance.

##  7 Conclusion

In this work we considered the problem of out-of-distribution detection in
semantic segmentation data. We introduced, to the best of our knowledge, the
first diffusion model based approach for this task. We explored design
possibilities in terms of model architecture and OoD scoring functions, the
best of which resulted in an approach which is competitive on the major
driving-oriented OoD detection benchmarks, and achieves state of the art
results on RoadAnomaly and SMIYC-Anomaly without outlier exposure and domain-
specific priors.

Furthermore, we proposed a novel benchmark based on the ADE20k dataset, with
the goal of assessing the quality of OoD detection approaches in settings with
higher diversity, _i.e_. beyond 19 in-distribution categories and driving
scenarios. We provide evaluations of state-of-the-art approaches on the
benchmark and show that our proposed method copes well with the increased
diversity.

####  7.0.1 Acknowledgements.

The research leading to these results is funded by the Deutsche
Forschungsgemeinschaft (DFG, German Research Foundation) under the project
numbers 401269959 and 417962828, and by the German Federal Ministry for
Economic Affairs and Climate Action within the project “NXT GEN AI METHODS –
Generative Methoden für Perzeption, Prädiktion und Planung". The authors would
like to thank the consortium for the successful cooperation.

## References

  * [1] Ackermann, J., Sakaridis, C., Yu, F.: Maskomaly: Zero-shot mask anomaly segmentation. In: BMVC (2023)
  * [2] Bevandić, P., Krešo, I., Oršić, M., Šegvić, S.: Simultaneous semantic segmentation and outlier detection in presence of domain shift. In: Fink, G.A., Frintrop, S., Jiang, X. (eds.) Pattern Recognition. pp. 33–47. Springer International Publishing, Cham (2019)
  * [3] Biase, G.D., Blum, H., Siegwart, R.Y., Cadena, C.: Pixel-wise anomaly detection in complex driving scenes. In: CVPR (2021)
  * [4] Blum, H., Sarlin, P.E., Nieto, J., Siegwart, R., Cadena, C.: The fishyscapes benchmark: Measuring blind spots in semantic segmentation. IJCV (2021)
  * [5] Cen, J., Yun, P., Cai, J., Wang, M.Y., Liu, M.: Deep metric learning for open world semantic segmentation. In: ICCV (2021)
  * [6] Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Fua, P., Salzmann, M., Rottmann, M.: SegmentMeIfYouCan: A benchmark for anomaly segmentation. arXiv:2104.14812 (2021)
  * [7] Chan, R., Rottmann, M., Gottschalk, H.: Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation. In: ICCV (2021)
  * [8] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR (2022)
  * [9] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [11] Elfwing, S., Uchibe, E., Doya, K.: Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks 107, 3–11 (2018)
  * [12] Franchi, G., Bursuc, A., Aldea, E., Dubuisson, S., Bloch, I.: One versus all for deep neural network for uncertainty (ovnni) quantification. IEEE Access 10, 7300–7312 (2022). https://doi.org/10.1109/ACCESS.2021.3138978
  * [13] Galesso, S., Argus, M., Brox, T.: Far away in the deep space: Dense nearest-neighbor-based out-of-distribution detection. In: ICCVW (October 2023)
  * [14] Galesso, S., Bravo, M.A., Naouar, M., Brox, T.: Probing contextual diversity for dense out-of-distribution detection. In: ECCV (2022)
  * [15] Gao, R., Zhao, C., Hong, L., Xu, Q.: Diffguard: Semantic mismatch-guided out-of-distribution detection using pre-trained diffusion models. In: ICCV (2023)
  * [16] Goodier, J., Campbell, N.D.: Likelihood-based out-of-distribution detection with denoising diffusion probabilistic models. In: BMVC (2023)
  * [17] Graham, M.S., Pinaya, W.H., Tudosiu, P.D., Nachev, P., Ourselin, S., Cardoso, J.: Denoising diffusion models for out-of-distribution detection. In: CVPR (2023)
  * [18] Grcić, M., Bevandić, P., Šegvić, S.: Dense anomaly detection by robust learning on synthetic negative data. arXiv:2112.12833 (2021)
  * [19] Grcic, M., Bevandi’c, P., vSegvi’c, S.: Densehybrid: Hybrid anomaly detection for dense open-set recognition. In: ECCV (2022)
  * [20] Grcić, M., Šarić, J., Šegvić, S.: On advantages of mask-level recognition for outlier-aware segmentation. In: CVPRW. pp. 2936–2946 (June 2023)
  * [21] Hendrycks, D., Mazeika, M., Dietterich, T.: Deep anomaly detection with outlier exposure. In: ICLR (2019)
  * [22] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
  * [23] Hyvärinen, A., Dayan, P.: Estimation of non-normalized statistical models by score matching. JMLR 6(4) (2005)
  * [24] Jung, S., Lee, J., Gwak, D., Choi, S., Choo, J.: Standardized max logits: A simple yet effective approach for identifying unexpected road obstacles in urban-scene segmentation. In: ICCV (2021)
  * [25] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: ICCV (2023)
  * [26] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., Duerig, T., Ferrari, V.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV (2020)
  * [27] Liang, C., Wang, W., Miao, J., Yang, Y.: GMMSeg: Gaussian mixture based generative semantic segmentation models. In: NeurIPS (2022)
  * [28] Lis, K.M., Nakka, K.K., Fua, P., Salzmann, M.: Detecting the unexpected via image resynthesis. In: ICCV (2019)
  * [29] Liu, Y., Ding, C., Tian, Y., Pang, G., Belagiannis, V., Reid, I., Carneiro, G.: Residual pattern learning for pixel-wise out-of-distribution detection in semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1151–1161 (2023)
  * [30] Liu, Z., Zhou, J.P., Wang, Y., Weinberger, K.Q.: Unsupervised out-of-distribution detection with diffusion inpainting. In: ICML (2023)
  * [31] Mahmood, A., Oliva, J., Styner, M.: Multiscale score matching for out-of-distribution detection. In: ICLR (2021)
  * [32] Mousakhan, A., Brox, T., Tayyub, J.: Anomaly detection with conditioned denoising diffusion models (2023)
  * [33] Nayal, N., Yavuz, M., Henriques, J.F., Güney, F.: Rba: Segmenting unknown regions rejected by all. In: ICCV (2023)
  * [34] Nekrasov, A., Hermans, A., Kuhnert, L., Leibe, B.: UGainS: Uncertainty Guided Anomaly Instance Segmentation. GCPR (2023)
  * [35] Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: ICML (2021)
  * [36] Pinggera, P., Ramos, S., Gehrig, S., Franke, U., Rother, C., Mester, R.: Lost and found: detecting small road hazards for self-driving vehicles. In: IROS (2016)
  * [37] Rai, S.N., Cermelli, F., Fontanel, D., Masone, C., Caputo, B.: Unmasking anomalies in road-scene segmentation. In: ICCV (2023)
  * [38] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [39] Shao, Z.: Multilabel remote sensing image retrieval based on fully convolutional network
  * [40] Shin, W., Lee, J., Lee, T., Lee, S., Yun, J.P.: Anomaly detection using score-based perturbation resilience. In: ICCV (2023)
  * [41] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (July 2015)
  * [42] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: NeurIPS (December 2019)
  * [43] Song, Y., Ermon, S.: Improved techniques for training score-based generative models. In: NeurIPS (2020)
  * [44] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: ICCV (2021)
  * [45] Tian, Y., Liu, Y., Pang, G., Liu, F., Chen, Y., Carneiro, G.: Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes. In: ECCV (2022)
  * [46] Vincent, P.: A connection between score matching and denoising autoencoders. Neural Computation (2011)
  * [47] Vojir, T., Šipka, T., Aljundi, R., Chumerin, N., Reino, D.O., Matas, J.: Road anomaly detection by partial image reconstruction with segmentation coupling. In: ICCV (2021)
  * [48] Wolleb, J., Bieder, F., Sandkühler, R., Cattin, P.C.: Diffusion models for medical anomaly detection. In: MICCAI (2022)
  * [49] Wu, A., Chen, D., Deng, C.: Deep feature deblurring diffusion for detecting out-of-distribution objects. In: ICCV (2023)
  * [50] Wu, Y., He, K.: Group normalization. In: ECCV (2018)
  * [51] Wyatt, J., Leach, A., Schmon, S.M., Willcocks, C.G.: Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In: CVPR (2022)
  * [52] Xia, Y., Zhang, Y., Liu, F., Shen, W., Yuille, A.L.: Synthesize then compare: Detecting failures and anomalies for semantic segmentation. In: ECCV (2020)
  * [53] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. In: NeurIPS (2021)
  * [54] Zhang, X., Li, N., Li, J., Dai, T., Jiang, Y., Xia, S.T.: Unsupervised surface anomaly detection with diffusion probabilistic model. In: ICCV (2023)
  * [55] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: CVPR (2021)
  * [56] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: CVPR (2017)

##  Appendix 0.A Additional Information on the ADE-OoD Benchmark

In this section, we provide additional information on the ADE-OoD benchmark.
We provide details on the construction process and show samples from the
benchmark alongside predictions from segmentation models. We analyze the
predictions regarding the challenges of the benchmark.

###  0.A.1 Construction Details

####  0.A.1.1 Image Selection.

As described in Sec. 5 of the main paper, the images for the benchmark were
selected from the ADE20k [56] validation set and from OpenImages [26]
according to the following criteria:

  1. 1.

Presence of clear and unambiguous in- and out-of-distribution entities only.

  2. 2.

Presence of diverse in- and out-of-distribution categories, as well as varying
indoor and outdoor settings.

  3. 3.

Adequate scene complexity to match the original ADE20k training data.

  4. 4.

Adequate number of samples for statistically significant results.

More specifically, we selected images by  (1.) automatically selecting ADE20k
images with large unlabeled regions, or manually selecting images from
OpenImages (via the OpenImages web visualizer) that contain semantic classes
that are not labeled in ADE20k,  (2.) manually discarding images that do not
fit to the described criteria (_e.g_. images that contain ambiguous entities),
(3.) validating the selected images by applying SOTA ADE20k segmentation
models and manually inspecting their outputs.

In total we selected 84 images from OpenImages, and 27 from ADE20k. The
complete benchmark data is provided on https://ade-ood.github.io and the
source of each image is indicated by its filename.

####  0.A.1.2 Annotation Process.

For the 27 images selected from ADE20k, we used the available ground truth
segmentation annotations, using the “unlabelled” class mask as OoD (after
checking for problematic cases as described above).

The annotation process for the 84 images selected from OpenImages was carried
out in a semi-automatic fashion using the Segment Anything Model (SAM) [25], a
promptable object segmentation model trained on very large data collections,
which we prompted using hand-selected points. For optimal mask coherence and
quality, objects and object parts were segmented individually before merging
all resulting masks. After obtaining binary masks for each image, we refined
them using dilation and erosion operations. This allowed to remove small noise
elements (points and holes) in the masks.

####  0.A.1.3 Benchmark Size and Statistical Significance.

To ensure that results on the benchmark are meaningful and reliable and that
the number of samples in the benchmark is adequate, we tested its statistical
robustness and compared it with that of other benchmarks. To this end, we
computed results on 10 random subsets of the test data, each containing 90% of
the original samples. The mean and standard deviation of the obtained
performances, computed for three benchmarks (RoadAnomaly, FS-Static,
ADE20k-OoD), are reported in Tab. 2.

Table 2: Mean and standard deviation results for 10-fold bootstrapped evaluation on RoadAnomaly, Fishyscapes Static, and ADE-OoD. | RoadAnomaly | FS-Static | ADE-OoD
---|---|---|---
| AP | FPR | AP | FPR | AP | FPR
cDNP | 85.2$\pm$0.9 | 13.2$\pm$1.7 | 62.8$\pm$4.0 | 18.9$\pm$1.7 | 61.6$\pm$1.6 | 37.9$\pm$1.0
RbA | 78.3$\pm$1.0 | 12.2$\pm$1.1 | 59.8$\pm$3.2 | 17.0$\pm$3.7 | 66.8$\pm$1.7 | 82.6$\pm$1.8
Ours | 89.2$\pm$0.7 | 9.1$\pm$0.7 | 74.8$\pm$2.7 | 6.1$\pm$0.6 | 62.9$\pm$2.1 | 35.9$\pm$0.8

The standard deviations are in a similar range for all benchmarks, and are
generally low, indicating that ADE-OoD has a similar level of stability and
statistical significance as other benchmarks.

###  0.A.2 Examples and Analysis of Segmentation Model Predictions

####  0.A.2.1 Examples.

A collection of samples (images and ground truth OoD masks) from the ADE-OoD
benchmark is shown in Fig. 7.

wall, towel,
---
mirror
windowpane,
wall, person

grass, sky,
---
fence
grass, person,
sky

tent, flag,
---
person
bed, sky,
person

floor, wall
---
grandstand
floor, wall
windowpane

person, wall,
---
floor
person, curtain,
table

building,
---
sculpture
tree, building

grass, person,
---
minibike
person, grass,
earth

sculpture,
---
wall, animal
tree, wall,
person

tray, floor,
---
wall
floor, table,
wall

truck, tree,
---
building
building, earth,
road

earth, building,
---
person
earth, door,
building

wall, box,
---
bottle
wall, person,
table

Figure 7: Samples from the proposed ADE20k-OoD Benchmark. We show sample
images with their respective annotation masks (red=OoD). Additionally, we
indicate the the top-3 classes that two state-of-the-art models (SETR and
Mask2Former) predict for the OoD regions. The first four rows contain examples
from the OpenImages dataset, while the samples in the last two rows are
sourced from ADE20k. The images featured in the benchmark depict both indoor
and outdoor scenes and feature diverse out-of-distribution objects of
different semantic classes.

####  0.A.2.2 Analysis of Segmentation Model Predictions.

Additionally, Fig. 7 indicates the most prominent predicted classes for the
OoD regions obtained from SETR and Mask2Former models.

The predictions give us useful information about how the trained networks
handle unknown objects, and therefore about the challenges of the current
models on the proposed benchmark. We can make the following observations:

  * •

Several times the OoD objects are classified as a spatially neighboring and
frequent class, though this is not visually related to the object (e.g. wall,
person, building). This is a form of extrapolation from the context that both
models do, often with the same categories.

  * •

The remaining predicted classes are visually similar to the OoD objects (e.g.
tent for the parachute in the second row, minibike for the segways in the
fourth row), or would likely feature in the surrounding environment (towel in
the restroom in the first row, fence in the outdoor scene in the first row).
These “good guess” predictions differ between the two models, confirming that
the chosen objects are interesting for OoD detection.

##  Appendix 0.B Diffusion Model Architecture

Figure 8: Overview of the diffusion model architecture. The architecture is
composed of several residual blocks, split into input blocks and output
blocks. In the usual U-Net fashion, skip-connections are present between the
input and output blocks. Each block receives the timestep embedding as input.

The architecture of the denoiser in our diffusion model, illustrated in Fig.
8, is based on the commonly used U-Net architecture, but replaces the
convolutional layers with linear layers. The network is composed of
consecutive sets of _residual blocks_ , structured into input blocks and
output blocks. Each block receives features and timestep embedding as input.
Skip connections forward input block features directly to the corresponding
output block, like in a convolutional U-Net. In our experiments we use 6 input
blocks and 6 output blocks.

####  0.B.0.1 Residual Block Architecture.

As shown in Fig. 9, each residual block first processes incoming features with
a sequence of a normalization layer (GroupNorm [50]), a non-linearity (SiLU
[11]), and a linear layer. Then the timestep embedding is added and the result
is processed by another sequence of a normalization layer, a non-linearity and
a linear layer. The original input is summed to the output to form a residual
connection. We use a constant hidden dimension throughout all linear layers.

Figure 9: Residual Block Architecture. Features are processed individually,
then summed with the timestep embedding, and then processed by output layers.
A residual connection spans the whole block, from input to output.

####  0.B.0.2 Lightweight Timestep Embedding.

As common in diffusion models, the current diffusion timestep is supplied to
the network in form of a timestep embedding. In order to reduce the
computational costs, we compute the timestep embedding as follows:

  * •

Instead of using transformer-style periodic sinusoidal time embeddings, as
done in DDPM and ViT works [22, 10], we directly use the scalar time embedding
$t$.

  * •

In each residual block, instead of further processing the timestep embedding
as done for example in DDPM, we directly add the scalar time embedding to the
incoming features (as shown in Fig. 9).

With these timestep embedding simplifications, we observe on average a 17%
inference time speedup and no decrease in OoD detection performance.

##  Appendix 0.C Qualitative Examples and Analysis

In Fig. 10 we show qualitative examples showcasing the differences between the
predictions of our approach DOoD, Mask2Anomaly (M2A)[37], and RbA [33]. The
examples highlight the respective advantages and disadvantages of the
approaches, all of which are high-performing, despite being based on very
different paradigms.

Image

Ground Truth

OoD scores - M2A

OoD scores - RbA

OoD scores - Ours

Figure 10: Qualitative comparison of our approach, Mask2Anomaly (M2A), and
RbA. The examples show images from RoadAnomaly, along with the OoD ground
truth (red=OoD) and the predicted OoD scores for each method. Purple indicates
low and red indicates high OoD scores. In all examples the methods produce
some false positive predictions (i.e. the scores are high but the ground truth
is black). However while our method mostly still produces lower scores than
for real anomalies, M2A and RbA produce very high OoD scores,
indistinguishable from the scores of true anomalies. This is, for example,
visible in the street sign (first row), the guard rail (second row), the
unpaved road and the building (third row). While all these examples are
undoubtedly challenging, a good OoD detection system should still assign them
lower scores. The last example shows a failure of our approach, which produces
false positives for several small objects.

The predictions from M2A and RbA are both based on Mask2Former, which
decouples the prediction of masks and categories, as opposed to the typical
per-pixel classification paradigm. For this reason, the scores are very
uniform over the pixel sets corresponding to the same mask, and have smooth
edges. In contrast to this, the predictions from our method are strictly local
(as they are computed on each patch individually) and have no enforced spatial
correlation.

Even though the predictions of M2A and RbA are more “tidy”, the approaches
have limitations. In the examples shown, it can be seen that where all
approaches produce false positives, the two Mask2Former-based approaches have
– mistakenly – high OoD scores. This can happen also for large sets of pixels,
if they are predicted to belong to the same mask by Mask2Former, resulting in
situations that are harmful for the benchmark results, and critical in real
life applications.

Our approach DOoD, on the other hand, can struggle with small objects: in the
last example the (out-of-distribution) street cones are assigned OoD scores
which are partly surpassed by the scores of other small rare objects.

##  Appendix 0.D Ablation: Compounding OoD Scores from the Diffusion Model and
the Segmentation Model

As described in the main paper (Sec. 4.3), we compound OoD scores obtained
from the diffusion model with the segmentation model uncertainty (using the
LogSumExp scoring function). In this section we compare the results from both
individually, and the compound ones.

Table 3: OoD detection results before and after compounding. | RoadAnomaly | Fishyscapes Static | ADE-OoD
---|---|---|---
Scores | AP$\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$ | AP$\uparrow$ | FPR$\downarrow$
Segmentation uncertainty | 63.5 | 33.3 | 24.1 | 23.6 | 45.9 | 55.3
Diffusion only | 87.1 | 9.0 | 75.2 | 3.9 | 48.6 | 44.3
Compound | 89.1 | 8.8 | 79.3 | 6.5 | 63.0 | 36.5

Quantitative results are shown in Tab. 3 on three benchmarks and qualitative
examples are shown in Fig. 11 for samples from RoadAnomaly and ADE-OoD.

The quantitative results clearly show that:

  1. 1.

Diffusion scores alone are always better than segmentation uncertainty.

  2. 2.

Compounding has a positive effect on the overall score, with its results being
better than both components.

Image |  |  |  |  |  |
---|---|---|---|---|---|---
OoD |  Mask |  |  |  |  |
Diffusion |  scores |  |  |  |  |
Compound |  scores |  |  |  |  |
Figure 11: Qualitative comparison between pure diffusion scores and compound
scores. In the examples above, taken from the RoadAnomaly and ADE20k-OoD
benchmarks, it can be seen that the diffusion scores already detect the
anomalous objects before compounding. Compounding usually has a positive
effect, noticeable for example in the example in the second column where it
reduces the score magnitudes for background pixels and increases the ones
corresponding to the vehicle. In some cases the effect is negative, such as in
the example in the third column, where some in-distribution background objects
provoke high uncertainty in the segmentation model.

##  Appendix 0.E Results on Dynamic Environmental Conditions

We tested our approach DOoD on WD-Pascal [2], a benchmark containing images of
road scenes in adverse environmental conditions. Quantitative results, along
with an example image from the benchmark and our method’s prediction for it,
are shown below:

Figure 12: Results on dynamic environmental conditions.

| AP | FPR
---|---|---
cDNP | 68.55 | 12.05
RbA | 73.00 | 9.58
DOoD (Ours) | 81.56 | 6.75

|
---|---

##  Appendix 0.F Results on Remote Sensing Data

The focus of our work is out-of-distribution detection for semantic
segmentation on general natural images. For completeness, in the following, we
demonstrate the applicability of our method DOoD to even more general data
domains via the example of remote sensing data. We constructed a benchmark by
training on WHDLD [39] and testing on DLRSD [39], using classes only found in
the latter (airplane, car, court, sand, ship, tank) as OoD. A qualitative
example and early results on this benchmark are shown below:

Figure 13: Early results on remote sensing data.

|  |
---|---|---

| AP | FPR
---|---|---
cDNP | 39.7 | 45.8
DOoD (Ours) | 48.4 | 42.5