# A Baseline for Detecting Out-of-Distribution Examples in Image Captioning

Gabi Shalev  Bar-Ilan UniversityIsrael<EMAIL_ADDRESS>,  Gal-Lev Shalev
Bar-Ilan UniversityIsrael<EMAIL_ADDRESS>and  Joseph Keshet
TechnionIsrael<EMAIL_ADDRESS>
(2022)

###### Abstract.

Image captioning research achieved breakthroughs in recent years by developing
neural models that can generate diverse and high-quality descriptions for
images drawn from the same distribution as training images. However, when
facing out-of-distribution (OOD) images, such as corrupted images, or images
containing unknown objects, the models fail in generating relevant captions.

In this paper, we consider the problem of OOD detection in image captioning.
We formulate the problem and suggest an evaluation setup for assessing the
model’s performance on the task. Then, we analyze and show the effectiveness
of the caption’s likelihood score at detecting and rejecting OOD images, which
implies that the relatedness between the input image and the generated caption
is encapsulated within the score.

out-of-distribution detection, image captioning, uncertainty estimation,
anomaly detection

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price: 15.00††doi:
10.1145/3503161.3548340††isbn: 978-1-4503-9203-7/22/10††ccs: Computing
methodologies Computer vision tasks

##  1\. Introduction

Figure 1. Generated captions for out-of-distribution images by a Top-Down
model trained on COCO dataset. The first image contains an object that does
not appear in training (unknown object), without a scene in the background.
Second is an image containing a random noise. Third is an image from Google’s
cartoon set, and forth is an image of unknown object (dolphin) in a natural
scene. Attention maps are added for the first example, and the predicted
probability of each token is mentioned in brackets.

Deep Neural Networks (DNNs) have gained lots of success after enabling several
breakthroughs in notably challenging problems such as image classification,
object detection, and language generation. However, despite the tremendous
progress driven by DNNs, they were found as vulnerable to OOD inputs. For
example, image classification models were shown to fail by predicting the
wrong class with high confidence when given images containing unknown objects
(Nguyen et al., 2015). Another example would be in the image captioning (IC)
task, where the models are able to output high-quality and diverse sentences
when given images drawn from the same distribution as the training set;
however, when facing images in the wild, these models are shown to generalize
poorly (Tran et al., 2016).

OOD detection task has been formulated as the task of detecting whether input
data is drawn from a distribution different from the training distribution.
The task has been studied for a long, and recently, a baseline for detecting
OOD images by neural image classification models has been proposed in
(Hendrycks and Gimpel, 2017). The authors investigated the OOD detection rate
of image classification models by considering the confidence of the predicted
class and set baseline results for the task. The baseline became widely
adopted, and many advanced approaches have been suggested for the task (Koner
et al., 2021; Hsu et al., 2020; Lee et al., 2018b; Li et al., 2021; Shalev et
al., 2018). Yet, despite the field’s progress, detecting OOD images remains a
difficult problem, as they are not limited to some particular type of images.
Instead, OOD images can have infinite forms, such as images of objects that do
not appear in the training set (unknown objects), corrupted images of known
objects, or even random noise images.

We focus on detecting OOD instances in image captioning (IC). While in the
image classification task, a model is given an image and is expected to
correctly classify it to one of $K$ possible classes, in IC, the model is
expected to generate a sentence that describes the scene in the image. Since
most OOD image detection research works considered the detection in an image
classification task environment, adopting these methods to IC is not
straightforward. A possible way to reject OOD images in the IC task
environment is by designing a two-phase system. First, a pre-trained image
classifier is applied to reject OOD images, and then, if the image was not
rejected, it is fed into the IC model to generate a caption. However, such a
method has several caveats: (1) It requires training and maintaining two
separate models, which adds to the time and space complexity. (2) When the
training data is composed of only images and their corresponding captions,
training an image classifier can be difficult as it requires heuristics for
generating a ground-truth class. But while these caveats are manageable, there
is a bigger question of whether image-classification-based models are
applicable to images used to train image captioning models, as the images are
different from those used to train image classification models. For example,
while image classification models are given images containing a single main
dominant object, IC models are given images containing complex scenes with
multiple objects interacting with each other.

This paper proposes a simple yet effective method for detecting OOD instances
by IC systems. Using an empirical study on several widespread IC models, we
show that the generated caption’s likelihood score can capture the relatedness
between the image and the generated caption and effectively allow the
rejection of OOD images. Furthermore, we find that while image-classification-
based methods express their uncertainty using a single score (such as the
predicted class probability), IC models express their uncertainty majorly in
visually grounded tokens (such as Nouns, Verbs, and Adjectives) and outperform
in detecting OOD images with a complex scene. Moreover, to facilitate OOD
detection research in IC, we construct benchmarks on top of widely used image
datasets and set baseline results for this task.

As the research in IC achieved breakthroughs in the field by developing end-
to-end IC models that are able to output high-quality and diverse sentences,
an increasing number of industrial companies integrated IC models into their
applications (Dognin et al., 2022; Sadeh et al., 2019a). However, in real-
world scenarios, it is most likely that input instances will contain objects
that the model did not see during training, which will result in unexpected
behavior by the model. A research work (MacLeod et al., 2017), explored how
blind and visually impaired people experience generated image captions. The
authors showed that people trust incorrect AI-generated captions, filling in
details to reconcile incongruencies rather than suspecting the caption may be
wrong. Therefore, with the increasing adoption of IC models in real-life
applications, it is crucial to focus on rejecting OOD inputs without
outputting the generated captions for these instances.

Neural generative IC models are encoder-decoder networks, trained end-to-end,
and achieve state-of-the-art results in the field. Training the models is
commonly done by maximum likelihood estimation (MLE), which maximizes the
probability of a ground-truth sentence given its corresponding training image.
During inference time, a decoding strategy is applied to search and output the
most probable describing sentence for the input image. These training and
inference procedures allow models to have rich image descriptiveness ability
and achieve high scores in IC metrics (such as Cider (Vedantam et al., 2015)
and BLEU (Papineni et al., 2002)) when given in-distribution images, but their
ability of ”expressing” their uncertainty for OOD images is yet unknown.

In Fig-1 we show captions generated by Top-Down model (Anderson et al., 2018)
for several different types of OOD images. The figure shows that the sentences
are vibrant and natural but also loosely related to the given image. Ideally,
when the IC model is facing an OOD image that it cannot describe, the decoded
sentence should be assigned with a low probability to allow the rejection of
the instance. However, MLE training does not guarantee to capture the
relatedness between the generated caption and the given image. Here, we
explore the effectiveness of the captions’ likelihood at expressing
uncertainty in the caption when given OOD images.

In summary, the contribution of this paper is as follows:

  * •

We define the task of out-of-distribution detection in image captioning and
explore the performance of several top-performing image captioning models.

  * •

We construct benchmarks that include multiple types of out-of-distribution
image sets and suggest evaluation metrics.

  * •

We demonstrate the efficiency of the generated captions’ likelihood at
detecting out-of-distribution images without requiring any additional data or
external knowledge and show that the uncertainty is expressed through the
probability of the visually grounded tokens.

##  2\. Related Work

##### Out-of-distribution detection

Since DNNs are ubiquitous, present in nearly all segments of the technology
industry, from search engines (Sadeh et al., 2019b) to critical applications
such as self-driving cars (Do et al., 2018) and healthcare (Shahid et al.,
2019; Granovsky et al., 2018), it becomes critical to design models that can
express uncertainty when predicting OOD inputs. OOD detection task has been
studied for a long, and recently, a baseline for detecting OOD images by
neural image classification models has been proposed in (Hendrycks and Gimpel,
2017). The authors examined the OOD detection performance of image
classification models using a confidence score derived from the network (max
softmax probability). Later, several studies proposed improving the baseline
by modifying the model’s architecture (Shalev et al., 2018; Koner et al.,
2021), inference procedure (Liang et al., 2018; Hsu et al., 2020) and training
criterion (Lee et al., 2018a, b).

OOD detection methods were also developed for safety-critical image
applications (Venkatakrishnan et al., 2020), and also into other modalities
such as natural language processing (NLP) and speech processing. (Hendrycks et
al., 2020) explored the task of OOD text detection in NLP classification tasks
(sentiment analysis and textual entailment). The authors showed that the
output confidence scores of pre-trained Transformers can effectively indicate
whether the sample is OOD. They also concluded that pre-trained Transformers
have an improved OOD detection rate compared to RNN and convolution-based text
classification models. More recently, (Li et al., 2021) suggested improving
the OOD detection by combining a self-supervised training method with an
ensemble of text classifiers, and the method sets the state-of-the-art
results.

##### Novel object description

Some of the first attempts at IC (Farhadi et al., 2010; Kuznetsova et al.,
2012) relied heavily on the outputs of object detectors and attribute
classifiers to describe images. Since then, a large amount of end-to-end IC
models have been developed, eliminating the need for external object detector
(Anderson et al., 2018; Vinyals et al., 2015; Herdade et al., 2019; Li et al.,
2019; Cornia et al., 2020; Shalev et al., 2021; Ji et al., 2021). However,
several works (Agrawal et al., 2019; Lu et al., 2018; Tran et al., 2016) also
leveraged the output of a pre-trained object detector to tackle the task of
novel object description. In this task, the IC model is given an image
containing an unknown object, and a caption describing the object with the
presented scene should be generated. Most commonly, external knowledge from a
pre-trained object detector is facilitated for describing the unknown objects.

The task of novel object description overlaps with the task of OOD detection
in the sense that both tasks need to deal with unknown objects. However, OOD
image definition is not limited to images containing unknown objects but
rather to a broad range of types such as corrupted and low-quality images or
even images containing random noise. Moreover, in OOD detection, the model is
expected to express its uncertainty in the caption to allow the rejection of
the instance. In contrast, in novel object description, the generated caption
is expected to include the unknown object.

While the proposed methods for novel object description gained success at
describing images containing unknown objects, they mainly rely on the output
of an object detector. Detecting whether an image is OOD (Hendrycks and
Gimpel, 2017) or contains novel objects (Agrawal et al., 2019) based on the
outputs of the object detector is not sufficient for real-world applications
since the number of classes that can be detected by the object detector is
finite and limited, even when trained on massive image classification
datasets. Additionally, object detectors include in their mechanism a
classification module which by itself is vulnerable to OOD examples (Hendrycks
and Gimpel, 2017), causing the entire process to be error-prone to both
classification and detection errors made by the external detection model.
Therefore, we find the task of OOD detection in IC an important complementary
task to the novel object detection task.

##  3\. Notations and Definitions

IC is the task of generating a natural language sentence that accurately
describes a given image. Neural generative IC models typically consist of a
CNN or visual Transformer, which is responsible for encoding the input image,
and an RNN or Transformer that decodes descriptive sentences word-by-word
conditioned on the image encoding.

Commonly, the models trained as follows: assume a training dataset,
$D_{train}=\\{(I_{i},S_{i})\\}^{N}_{i=1}$, composed of image-sentence pairs,
where $I_{i}$ is the i-th image and $S_{i}$ is the corresponding descriptive
sentence. The parameters of the model, denoted by $\theta$, are found by MLE,
which directly maximizes the log-likelihood of the ground-truth description
given the image, formally:

(1) $\theta^{*}=\operatorname*{argmax}_{\theta}\sum_{(S_{i},I_{i})\in
D_{train}}\log P(S_{i}|I_{i};\theta)$

Since $S_{i}$ composed of a sequence of words $S_{i1},...,S_{ij}$, where $j$
is the length of $S_{i}$, a chain rule is applied to model the joint
probability over the sentence words (we omit $\theta$ for simplicity):

$\log P(S_{i}|I_{i})=\sum_{z=1}^{z=j}\log P(S_{iz}|I,S_{i1},...,S_{iz-1})$

This optimization requires assigning the highest probability to the ground-
truth sentence, which is the sum of the log probabilities of its composing
words.

At inference, a decoding strategy is applied, searching for the best
describing sentence. A greedy decoding is the one that, at each decoding step,
chooses the most probable next word. Another widely spread heuristic decoding
strategy is Beam-search-K (BS-K), which maintains the $K$ most probable
partial sequences until each decoding step. Notice that the greedy decoding is
a special case of BS-K, where $K=1$. The BS-K does not guarantee finding the
most probable sentence and leads to degenerated sentences when $K$ is large
(Holtzman et al., 2020). Several sampling-based decoding algorithms have been
proposed. Top-K decoding (Fan et al., 2018), is a method that samples at each
timestamp a word from the K most probable next words. Later, Nucleus sampling
(NS-p) decoding strategy has been proposed (Holtzman et al., 2020). NS-p
suggests sampling from the top p portion of the probability mass instead of
relying on a fixed top K candidate pool.

##  4\. Problem Formulation

In this paper, we are interested in the problem of out-of-distribution
detection in IC: can we detect whether the given image is from a different
distribution than the training data? Does the generated caption relate to the
OOD image?

Denote by $\mathcal{S}$ the infinite set of all possible sentences and by
$\mathcal{I}$ the infinite set of all possible images. Assume each training
example, $(S_{i},I_{i})\in\mathcal{S}\times\mathcal{I}$, is drawn from a fixed
but unknown distribution $\rho$. In OOD detection, a model is expected to
perform well on unseen images drawn from the same distribution $\rho$ (in-
distribution images) and identify images drawn from a different distribution,
$\mu$, to prevent the generation of unrelated captions. Since we train the
model to maximize the probability of the ground-truth captions for images
drawn from $\rho$, its behavior on images drawn from $\mu$ is unexpected.
Therefore, we are interested in exploring whether MLE training allows models
to capture the relatedness between the generated caption and the given image.
If the model learned to factor in the relatedness, it would be reasonable to
assume that in-distribution images will result in higher likelihood captions
than the OOD images. Meaning:

(2) $\log P(S_{I_{\rho}}|I_{\rho})>\log P(S_{I_{\mu}}|I_{\mu})$

Where $I_{\rho}$, $I_{\mu}$ are in- and out- of-distribution images,
respectively, and $S_{I_{\rho}}$, $S_{I_{\mu}}$ are the corresponding
generated captions.

In- and out-of-distribution images are not a strict dichotomy. There are
various types of OOD images, such as images containing unknown objects (that
are not appearing in the training set) or random noise. Identifying images
containing unknown objects, which the model cannot accurately describe, is
more challenging than detecting random noise images due to their relative
visual closeness to in-distribution images.

We find that OOD images can be effectively detected using the generated
captions’ likelihood score. This finding implies that even though MLE training
maximizes the sentence likelihood for in-distribution images only, it still
allows the model to express its uncertainty when given OOD images by
generating low probability captions. On the application side, the generated
captions for OOD instances can be rejected based on the captions’ likelihood
to prevent the output of irrelevant captions.

Figure 2. Images from the cropped unknown object set with captions generated
by Top-Down model.

##  5\. Experiments

Figure 3. Comparison between captions generated for (left to right): in-
distribution image, and out-of-distribution corrupted images using: salt-and-
pepper noise, JPEG compression, cartoon corruption, and snowflakes corruption

In this section, we demonstrate the effectiveness of the likelihood score at
OOD detection by evaluating the performance of six widespread IC models. We
also train an image classification model and compare its performance by
applying two OOD detection methods. We start by describing the models in
section 5.1. Then, in section 5.2 we describe the experimental setup including
the OOD datasets. Evaluation metrics are described in 5.3. Lastly, in section
5.4, we show the detection results and analyze the generated captions of both
in- and out- of-distribution images.

###  5.1. Baseline and Evaluated Models

As our captioning models, we trained two of the most widespread RNN-based and
four Transformer-based IC models. RNN-based models: (1) Top-Down (Anderson et
al., 2018) (2) Show-Tell (Vinyals et al., 2015). We set a pre-trained
ResNet-101 as the image encoder component in both models. Transformer models:
(1) ORT (Herdade et al., 2019) (2) ETA (Li et al., 2019) (3) M2 (Cornia et
al., 2020) (4) GET (Ji et al., 2021).

To compare the performance of the IC models with OOD detection methods based
on image-classification models, we trained a classification baseline model
(ResNet-101) on objects appearing in the training datasets (we list the labels
in the appendix). For the classification model we use two popular OOD
detection methods: (1) Max probability score (MSP) (Hendrycks and Gimpel,
2017) (2) Generalized ODIN (GODIN)(Hsu et al., 2020). Notice that the
classification and the captioning models are not exposed to OOD objects during
training for a fair comparison.

For the captioning models, since the decoding strategies search for the most
probable sentence differently, we evaluate captions generated by several
decoding strategies to examine the effectiveness of the likelihood score at
OOD detection. The strategies we compare are Greedy, BS-10, Top-10, and
NS-0.8.

We train and evaluate each of the models on - COCO (Lin et al., 2014) and
Flickr-8K (Hodosh et al., 2013) datasets.

###  5.2. Experimental Setup

For evaluating OOD detection in IC, we adopt the popular experimental settings
of OOD detection in image (Hendrycks and Gimpel, 2017) and text (Hendrycks et
al., 2020) classification. In the proposed setup, models are required to
distinguish between a set of in-distribution samples and multiple OOD sets in
the following setup: First, a model is trained over a training dataset
$D^{IN}_{train}$ (without the exposure to OOD instances). Afterward, the model
is evaluated over the test set $D^{IN}_{test}$, and the predicted probability
for each sample is recorded. Then, to assess the detection ability, the model
is evaluated over several OOD datasets, denoted as $D^{OUT}_{1}$, …,
$D^{OUT}_{p}$, and the predicted probability for each sample in the sets is
recorded. Lastly, a measure of the separation between the recorded
probabilities for $D^{IN}_{test}$ and each of the OOD sets $D^{OUT}_{i}$ is
calculated.

We follow this setup and conduct several experiments wherein we create two
groups. The first group contains the likelihood scores of captions generated
for in-distribution images. This group is shared across all the experiments.
The second group contains the likelihood scores of captions generated for OOD
images. We consider eight types of OOD image sets in various degrees of
difficulty, described later in this section. We compare each of the groups
created from the OOD sets with the group created from the in-distribution set
and measure the separation between the two using the evaluation metrics
described in section 5.3. An IC model with a perfect OOD detection ability
will allow setting a fixed threshold of $T$, where all captions generated for
in-distribution images will have higher likelihood scores, and captions for
OOD images will be assigned with lower scores.

For evaluating the OOD detection methods by the classification model, we
follow the same setup but consider the predicted model’s confidence as
described in the original papers.

####  5.2.1. Out-of-Distribution Image Sets

We evaluate the performance on eight different OOD image sets. Two of the sets
composed of images with no object appearing in them. Additional two sets
composed of images containing unknown objects (samples can be seen in Fig-1
and Fig-2). The last four sets are composed of algorithmically corrupted in-
distribution images. The details of the sets are as follows:

Unknown objects: For creating this set we collected images containing a scene
with objects that are not appearing the training sets. The images are sourced
from the Open Images V4 (Krasin et al., 2017), which is a publicly available
human-annotated object detection dataset, containing 14.6M bounding boxes of
600 objects in ~$1.7M$ images. From this set, we manually collected 38 objects
which are not appearing in the COCO set and constructed a set of 4067 images
containing unknown objects in natural scenes. The full list of the extracted
objects can be found in the appendix.

Cropped unknown objects: Since Open images V4 composed of natural images where
the unknown objects are appearing with a scene in the background, we isolate
the objects from the presented scene and create a set of images containing a
single object only. For creating this set, we consider images from the Unknown
objects set, and crop the unknown objects using the corresponding ground-truth
bounding boxes. Examples can be seen in Fig-2

Google’s cartoon set: A publicly available dataset of 10K 2D cartoon avatar
images111https://google.github.io/cartoonset/. The cartoons vary in 10 artwork
categories, 4 color categories, and 4 proportion categories, with a total of
~$10^{13}$ possible combinations.

Random noise: We generate 5K random noise images by randomly drawing each
pixel’s value.

Corrupted image sets: Since COCO and Flickr training sets contain high-quality
images, we apply on the corresponding test sets four types of algorithmically
generated corruptions to create the following OOD sets: (1) JPEG corruption:
consisting of images corrupted by JPEG compression. (2) Salt-and-pepper
corruption: consisting of images added with salt-and-pepper noise. (3) Snow
corruption: consisting of images added with white noise, imitating images
taken in snowy weather (4) Cartoon corruption: consisting of images converted
to cartoon style. In Fig-3 we demonstrate the considered corruptions.

These corruptions are a subset of the corruptions used to benchmark models’
robustness to common corruptions and perturbations (Hendrycks and Dietterich,
2019; Shalev et al., 2020). The corruptions are generated using ImgAug (Jung
et al., 2019) library. We normalize the OOD images using the same
normalization parameters used for normalizing the training set.

Figure 4. The upper figure compares the log-likelihood scores of captions generated for in- and out-of-distribution images. The lower figure shows the ROC plot for the distributions above and compares it against a random detector. The captions are produced by the Top-Down model trained on the COCO dataset. Table 1. IC evaluation metrics results for models trained on COCO dataset. We evaluate each metric on (from left to right): “Karpathy” test split (in bold), JPEG compressed set, salt-and-pepper set, snowflakes set ,and cartoon-corruption set. Model | Cider | BLEU-4 | ROUGE-L
---|---|---|---
Top-Down |  1.16/ 0.09/ 0.18/ 0.41/ 0.33 |  0.35/ 0.05/ 0.09/ 0.17/ 0.13 |  0.53/ 0.29/ 0.33/ 0.36/ 0.37
Show-Tell |  0.98/ 0.17/ 0.19/ 0.36/ 0.37 |  0.31/ 0.07/ 0.06/ 0.13/ 0.14 |  0.51/ 0.30/ 0.31/ 0.38/ 0.39
ORT |  1.28/ 0.08/ 0.16/ 0.42/ 0.38 |  0.38/ 0.06/ 0.06/ 0.11/ 0.12 |  0.58/ 0.27/ 0.31/ 0.36/ 0.37
ETA |  1.26/ 0.13/ 0.17/ 0.39/ 0.38 |  0.38/ 0.06/ 0.06/ 0.12/ 0.13 |  0.58/ 0.29/ 0.32/ 0.36/ 0.37
M2 |  1.30/ 0.14/ 0.18/ 0.38/ 0.37 |  0.38/ 0.07/ 0.08/ 0.13/ 0.13 |  0.58/ 0.30/ 0.31/ 0.36/ 0.37
GET |  1.30/ 0.14/ 0.19/ 0.39/ 0.38 |  0.38/ 0.08/ 0.06/ 0.12/ 0.13 |  0.58/ 0.30/ 0.31/ 0.36/ 0.38

###  5.3. Evaluation Metrics

For evaluating the detection rate, we compare the likelihood scores of the
generated sentences for the in-distribution image with the scores of the
generated sentences for each of the OOD sets and measure how well they can be
separated. For measuring the separation, we use the following metrics:

Area under the Receiver Operating Characteristic curve (AUROC): The metric was
adopted by (Hendrycks and Gimpel, 2017) for OOD detection in image
classification task. The ROC curve depicts the relationship between true
positive rate and false positive rate. The metric is a threshold-independent
performance evaluation that can be interpreted as the probability that a
positive example has a greater detector score/value than a negative example. A
perfect detector corresponds to AUROC of 1.

Area under the Precision-Recall curve: As the AUROC is not ideal when the
positive and negative classes have greatly differing base rates, another
threshold independent metric was proposed for OOD detection (Hendrycks and
Gimpel, 2017). The PR curve is a graph showing the precision and recall
against each other. The metrics PRin and PRout in Tables denote the area under
the precision-recall curve where in- and out- of-distribution images are
specified as positives, respectively. A perfect detector has an AUPR of 1.

Bhattacharyya distance: The distance metric was introduced by (Bhattacharyya,
1943) and later adopted by (Reyes-Aldasoro and Bhalerao, 2006) for feature
selection. Bhattacharyya Distance (BD) measures the similarity of two discrete
or continuous distributions.

We use this metric to measure the amount of overlap between the distributions
of the likelihood scores of caption generated for in- and out-of-distribution
images. Higher BD value means that the overlap between the distributions is
smaller. Hence, perfect OOD detection will result in $BD=\infty$.

Table 2. Results for in- and out-of-distribution detection over the aggregation of the OOD sets. (COCO as an in-distribution set) Model | ROC$\uparrow$ | PRin$\uparrow$ | PRout$\uparrow$ | BD$\uparrow$
---|---|---|---|---
Top-Down | 0.84 | 0.87 | 0.85 | 0.45
Show-Tell | 0.82 | 0.82 | 0.84 | 0.39
ORT | 0.90 | 0.89 | 0.91 | 0.51
ETA | 0.89 | 0.89 | 0.90 | 0.50
M2 | 0.88 | 0.87 | 0.90 | 0.49
GET | 0.90 | 0.90 | 0.90 | 0.53
MSP | 0.73 | 0.72 | 0.74 | 0.10
GODIN | 0.76 | 0.74 | 0.78 | 0.18
Table 3. Results for in- and out-of-distribution detection over the aggregation of the OOD sets. (Flickr-8K as an in-distribution set) Model | ROC$\uparrow$ | PRin$\uparrow$ | PRout$\uparrow$ | BD$\uparrow$
---|---|---|---|---
Top-Down | 0.82 | 0.81 | 0.84 | 0.41
Show-Tell | 0.81 | 0.80 | 0.81 | 0.36
ORT | 0.89 | 0.88 | 0.90 | 0.50
ETA | 0.87 | 0.88 | 0.86 | 0.49
M2 | 0.88 | 0.87 | 0.89 | 0.51
GET | 0.88 | 0.87 | 0.89 | 0.51
MSP | 0.74 | 0.73 | 0.75 | 0.19
GODIN | 0.78 | 0.77 | 0.78 | 0.23

###  5.4. Results

We start by aggregating all the OOD image sets and evaluating the detection
rate. Tables 2 and 3 summarize the results for models trained on COCO and
Flickr-8k datasets, respectively. As results indicate, the generated captions’
likelihood scores can detect OOD instances significantly better than the
methods based on image-classification models. Since the captions generated for
the OOD images are assigned with a lower likelihood than unseen in-
distribution images, they can be rejected with high rates by thresholding the
likelihood score. Meaning that instead of applying two separate models where
the first decide whether an image is OOD or not, and the second is the IC
model responsible for generating a relevant caption, a single IC model can be
applied. We also notice that Transformer-based IC models can detect OOD
instances with higher rates than RNN-based models. These results are aligned
with (Hendrycks et al., 2020) that showed that Transformer-based models used
for _text classification_ are better OOD detectors. Next, we analyze the
detection of each OOD set independently. We start by examining the corrupted
image sets, and afterward, we analyze the unknown object, random noise, and
cartoon image sets.

Corrupted image sets: recall that these sets consist of corrupted in-
distribution images taken from the corresponding test sets. To demonstrate the
severity of the corruptions, we generate captions for each of the corrupted
sets and measure the standard IC evaluation metrics against the corresponding
ground-truth captions. Results, summarized in Table-1, show a significant drop
in all metrics when models are evaluated over the corrupted sets since the
generated captions are irrelevant to the given images. For comparison,
randomly generated captions (created by randomly drawing tokens from the
vocabulary without considering the image) achieve the following scores: a
Cider of 0.06, BLEU-4 of 0.04, and ROUGE-L of 0.24. However, despite the
severe drop caused by these sets, we find that the corrupted images can be
effectively detected by considering the captions’ likelihood scores. The
detection results of the models trained on the COCO dataset appear in Table-4
(results for Flickr data sets are consistent).

In Fig-4, we demonstrate the likelihood of the JPEG compression set against
COCO’s test set. The ROC curve for the JPEG compression set shows that when
80% of the compressed images are rejected, less than 3% of in-distribution
images are falsely rejected (at the point where the TPR and FPR are 97%, 20%
respectively). For comparison, a random detector that randomly assigns a score
for a caption would reject only 3% of the OOD images while falsely rejecting
3% of the in-distribution images (at the point where the TPR and FPR are both
97%).

Table 4. Results for in- and out-of-distribution detection over the corrupted OOD image sets. Model | OOD set |  ROC$\uparrow$ |  PRin$\uparrow$ |  PRout$\uparrow$ |  BD$\uparrow$
---|---|---|---|---|---
ORT | Salt-and-pepper | 0.924 | 0.915 | 0.918 | 0.523
JPEG compression | 0.931 | 0.937 | 0.929 | 0.503
Snow flakes | 0.829 | 0.801 | 0.842 | 0.184
Cartoon corruption | 0.836 | 0.822 | 0.851 | 0.210
ETA | Salt-and-pepper | 0.909 | 0.922 | 0.900 | 0.511
JPEG compression | 0.939 | 0.944 | 0.923 | 0.555
Snow flakes | 0.801 | 0.789 | 0.832 | 0.134
Cartoon corruption | 0.833 | 0.813 | 0.836 | 0.202
Top-Down | Salt-and-pepper | 0.911 | 0.910 | 0.909 | 0.455
JPEG compression | 0.961 | 0.962 | 0.968 | 0.797
Snow flakes | 0.778 | 0.739 | 0.791 | 0.111
Cartoon corruption | 0.803 | 0.785 | 0.809 | 0.149
Show-Tell | Salt-and-pepper | 0.863 | 0.849 | 0.851 | 0.292
JPEG compression | 0.870 | 0.868 | 0.851 | 0.321
Snow flakes | 0.752 | 0.693 | 0.772 | 0.103
Cartoon corruption | 0.725 | 0.682 | 0.712 | 0.053
MSP | Salt-and-pepper | 0.891 | 0.901 | 0.854 | 0.498
JPEG compression | 0.822 | 0.854 | 0.812 | 0.349
Snow flakes | 0.712 | 0.735 | 0.701 | 0.118
Cartoon corruption | 0.852 | 0.856 | 0.829 | 0.219
GODIN | Salt-and-pepper | 0.843 | 0.810 | 0.866 | 0.432
JPEG compression | 0.849 | 0.841 | 0.850 | 0.422
Snow flakes | 0.739 | 0.726 | 0.744 | 0.127
Cartoon corruption | 0.741 | 0.765 | 0.704 | 0.089
Table 5. Results for in- and out-of-distribution detection over the non-corrupted OOD image sets. Model | OOD set |  ROC$\uparrow$ |  PRin$\uparrow$ |  PRout$\uparrow$ |  BD$\uparrow$
---|---|---|---|---|---
ORT | Unknown objects | 0.769 | 0.771 | 0.763 | 0.132
Random noise | 0.985 | 0.971 | 0.992 | 1.691
Google’s cartoon set | 0.986 | 0.988 | 0.979 | 1.198
Cropped unknown objects | 0.767 | 0.764 | 0.782 | 0.131
ETA | Unknown objects | 0.755 | 0.764 | 0.701 | 0.119
Random noise | 0.981 | 0.976 | 0.984 | 1.531
Google’s cartoon set | 0.978 | 0.969 | 0.984 | 1.101
Cropped unknown objects | 0.781 | 0.771 | 0.793 | 0.163
Top-Down | Unknown objects | 0.736 | 0.737 | 0.716 | 0.115
Random noise | 0.965 | 0.981 | 0.899 | 1.682
Google’s cartoon set | 0.980 | 0.985 | 0.964 | 1.181
Cropped unknown objects | 0.759 | 0.762 | 0.732 | 0.128
Show-Tell | Unknown objects | 0.727 | 0.722 | 0.704 | 0.097
Random noise | 0.954 | 0.969 | 0.921 | 0.995
Google’s cartoon set | 0.930 | 0.943 | 0.910 | 0.595
Cropped unknown objects | 0.722 | 0.721 | 0.699 | 0.092
MSP | Unknown objects | 0.641 | 0.656 | 0.629 | 0.099
Random noise | 0.899 | 0.903 | 0.886 | 0.349
Google’s cartoon set | 0.912 | 0.922 | 0.909 | 0.498
Cropped unknown objects | 0.702 | 0.689 | 0.711 | 0.094
GODIN | Unknown objects | 0.706 | 0.699 | 0.711 | 0.102
Random noise | 0.969 | 0.984 | 0.973 | 1.544
Google’s cartoon set | 0.963 | 0.977 | 0.965 | 1.001
Cropped unknown objects | 0.786 | 0.782 | 0.789 | 0.171

Figure 5. Comparison between the average predicted probability for noun, verb,
and adjective tokens. On the left panel we show the results of Show-Tell model
with NS-0.8. On the right panel we show the results of Top-Down model with
BS-10.

Non-corrupted image sets: moving forward to the next four OOD sets. These
sets, composed of images containing unknown objects, cartoons, and noise, can
also be detected at high rates using the likelihood score. Table-5 summarizes
the detection rates of the models. As can be seen, unsurprisingly, images
containing random noise or cartoons are the easiest to detect, while images
containing unknown objects are the hardest due to their visual closeness to
in-distribution images.

Here as before, we find the Transformer-based models outperform. In all tested
sets, the models outperform a random detector222AUROC is equal to 0.5, BD is
equal to 0, and AUPR is equal to the precision and the classification-based
models. To gain an intuition for why the classification models achieve lower
results, we checked their confidence scores and found that they tend to be
high even for objects that are not part of the training object (overconfidence
when wrong). This was also observed in previous works (Guo et al., 2017).

Recall that training the captioning models is done by MLE, which does not
guarantee to produce less likely captions for OOD images. To better understand
why exactly the generated captions for OOD images tend to have a lower
likelihood, we extract the predicted probabilities of their visual counterpart
tokens. More specifically, we consider the part-of-speech of each token in the
generated caption and focus on the predicted probabilities of noun, verb, and
adjective tokens, as they should be visually grounded in the given image.
Interestingly, in all models, the noun, verb, and adjective tokens are
predicted with lower probabilities for OOD images compared to in-distribution
images (see Fig-5). In contrast, tokens from other part-of-speech groups such
as determiner (a, an, the, etc.), adverbs (very, there, where, etc.), and
adposition (in, to, during, etc.) are predicted with the same average
probability for both in- and out -of-distribution images. Our intuition is
that since the predicted visual tokens should match the image and fit the
textual context, the model can learn the in-distribution from a richer signal
compared to the classification model. We also noticed, in preliminary results,
that as we increased the weight of the loss of visually grounded tokens during
training (by multiplying the loss with a constant), the OOD detection rates
increased while at the same time, the performance on in-distribution remained
unchanged. We leave the exploration of this direction to future work.

Table 6. Results for in- and out-of-distribution detection of Top-Down model using different decoding strategies. OOD set | decoding |  ROC$\uparrow$ |  PRin$\uparrow$ |  PRout$\uparrow$ |  BD$\uparrow$
---|---|---|---|---|---
Salt-and- | Greedy | 0.831 | 0.842 | 0.809 | 0.230
pepper | Top-10 | 0.597 | 0.614 | 0.562 | 0.025
corruption | N-0.8 | 0.837 | 0.833 | 0.826 | 0.238
JPEG | Greedy | 0.935 | 0.939 | 0.928 | 0.571
compression | Top-10 | 0.657 | 0.690 | 0.604 | 0.061
corruption | N-0.8 | 0.892 | 0.897 | 0.877 | 0.379
Snow | Greedy | 0.706 | 0.678 | 0.703 | 0.091
flakes | Top-10 | 0.603 | 0.610 | 0.574 | 0.013
corruption | N-0.8 | 0.696 | 0.675 | 0.685 | 0.076
Cartoon | Greedy | 0.705 | 0.689 | 0.715 | 0.070
corruption | Top-10 | 0.590 | 0.585 | 0.571 | 0.019
| N-0.8 | 0.757 | 0.736 | 0.755 | 0.105
Unknown | Greedy | 0.691 | 0.690 | 0.670 | 0.068
objects | Top-10 | 0.555 | 0.566 | 0.531 | 0.013
| N-0.8 | 0.676 | 0.669 | 0.659 | 0.057
Random | Greedy | 0.809 | 0.863 | 0.766 | 0.720
noise | Top-10 | 0.657 | 0.685 | 0.610 | 0.058
| N-0.8 | 0.911 | 0.924 | 0.891 | 0.470
Google’s | Greedy | 0.892 | 0.922 | 0.842 | 0.515
cartoon | Top-10 | 0.566 | 0.609 | 0.531 | 0.028
set | N-0.8 | 0.865 | 0.884 | 0.830 | 0.336
Cropped | Greedy | 0.708 | 0.713 | 0.679 | 0.082
unknown | Top-10 | 0.549 | 0.559 | 0.525 | 0.012
objects | N-0.8 | 0.760 | 0.761 | 0.734 | 0.130
Table 7. Results for in- and out-of-distribution detection of Show-Tell model using different decoding strategies. OOD set | decoding |  ROC$\uparrow$ |  PRin$\uparrow$ |  PRout$\uparrow$ |  BD$\uparrow$
---|---|---|---|---|---
Salt-and- | Greedy | 0.755 | 0.761 | 0.721 | 0.139
pepper | Top-10 | 0.561 | 0.584 | 0.537 | 0.017
corruption | N-0.8 | 0.682 | 0.688 | 0.656 | 0.064
JPEG | Greedy | 0.761 | 0.777 | 0.742 | 0.148
compression | Top-10 | 0.635 | 0.649 | 0.607 | 0.042
corruption | N-0.8 | 0.750 | 0.754 | 0.719 | 0.242
Snow | Greedy | 0.698 | 0.666 | 0.687 | 0.087
flakes | Top-10 | 0.586 | 0.588 | 0.571 | 0.012
corruption | N-0.8 | 0.645 | 0.630 | 0.644 | 0.069
Cartoon | Greedy | 0.676 | 0.653 | 0.674 | 0.044
corruption | Top-10 | 0.588 | 0.591 | 0.572 | 0.011
| N-0.8 | 0.672 | 0.653 | 0.661 | 0.038
Unknown | Greedy | 0.657 | 0.679 | 0.627 | 0.044
objects | Top-10 | 0.547 | 0.563 | 0.523 | 0.013
| N-0.8 | 0.645 | 0.647 | 0.622 | 0.039
Random | Greedy | 0.796 | 0.860 | 0.684 | 0.521
noise | Top-10 | 0.630 | 0.657 | 0.590 | 0.038
| N-0.8 | 0.815 | 0.851 | 0.809 | 0.310
Google’s | Greedy | 0.863 | 0.877 | 0.843 | 0.333
cartoon | Top-10 | 0.611 | 0.632 | 0.584 | 0.033
set | N-0.8 | 0.836 | 0.853 | 0.814 | 0.257
Cropped | Greedy | 0.649 | 0.667 | 0.617 | 0.049
unknown | Top-10 | 0.548 | 0.564 | 0.531 | 0.013
objects | N-0.8 | 0.641 | 0.642 | 0.616 | 0.041

Decoding strategies: lastly, we investigate the influence of the decoding
strategy on the OOD detection rates. We compare three additional types of
decoding strategies: (1) greedy. (2) top-10 sampling. (3) nucleus sampling
(N-0.8). Results presented in Table-6 and Table-7 indicate that the influence
of the decoding strategy has a high impact on the overall results and that
beam search is the most effective for this task. We find Top-K as the least
effective decoding method for this task, as it has higher chances of choosing
low confidence tokens even for in-distribution samples. As some decoding
strategies yield significantly superior results for OOD detection than others,
we think that developing new techniques that consider the relatedness might
boost the results further.

##  6\. Conclusions and Future Work

To conclude, we formulated the task of OOD detection in image captioning and
presented evaluation datasets and metrics. We demonstrated that IC models are
able to express their uncertainty when facing OOD images by generating low
probability captions. While the captions’ likelihood is effective for
detecting OOD images and is able to capture the image-caption relatedness,
results indicate there is still room for improvement. We hope to inspire
future work on developing new models, training methods, and decoding
strategies for tackling the important task of ”knowing” when the model cannot
describe the given image.

## References

  * (1)
  * Agrawal et al. (2019) Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019\.  nocaps: novel object captioning at scale. In _Proceedings of the IEEE International Conference on Computer Vision_. 8948–8957.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6077–6086.
  * Bhattacharyya (1943) Anil Bhattacharyya. 1943\.  On a measure of divergence between two statistical populations defined by their probability distributions.  _Bull. Calcutta Math. Soc._ 35 (1943), 99–109.
  * Cornia et al. (2020) Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020.  Meshed-memory transformer for image captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10578–10587.
  * Do et al. (2018) Truong-Dong Do, Minh-Thien Duong, Quoc-Vu Dang, and My-Ha Le. 2018\.  Real-Time Self-Driving Car Navigation Using Deep Neural Network. In _2018 4th International Conference on Green Technology and Sustainable Development (GTSD)_. IEEE, 7–12.
  * Dognin et al. (2022) Pierre Dognin, Igor Melnyk, Youssef Mroueh, Inkit Padhi, Mattia Rigotti, Jarret Ross, Yair Schiff, Richard A Young, and Brian Belgodere. 2022.  Image captioning as an assistive technology: Lessons learned from vizwiz 2020 challenge.  _Journal of Artificial Intelligence Research_ (2022).
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Melbourne, Australia, 889–898.  https://doi.org/10.18653/v1/P18-1082
  * Farhadi et al. (2010) Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Peter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. 2010\.  Every picture tells a story: Generating sentences from images. In _European conference on computer vision_. Springer, 15–29.
  * Granovsky et al. (2018) Lena Granovsky, Gabi Shalev, Nancy Yacovzada, Yotam Frank, and Shai Fine. 2018.  Actigraphy-based sleep/wake pattern detection using convolutional neural networks.  _arXiv preprint arXiv:1802.07945_ (2018).
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017\.  On calibration of modern neural networks. In _International Conference on Machine Learning_. PMLR, 1321–1330.
  * Hendrycks and Dietterich (2019) Dan Hendrycks and Thomas Dietterich. 2019.  Benchmarking Neural Network Robustness to Common Corruptions and Perturbations.  _Proceedings of the International Conference on Learning Representations_ (2019).
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks.  _Proceedings of International Conference on Learning Representations_ (2017).
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020\.  Pretrained Transformers Improve Out-of-Distribution Robustness. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 2744–2751.  https://doi.org/10.18653/v1/2020.acl-main.244
  * Herdade et al. (2019) Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. 2019.  Image captioning: Transforming objects into words.  _arXiv preprint arXiv:1906.05963_ (2019).
  * Hodosh et al. (2013) Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013\.  Framing image description as a ranking task: Data, models and evaluation metrics.  _Journal of Artificial Intelligence Research_ 47 (2013), 853–899.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2020\.  The curious case of neural text degeneration.  _Proceedings of International Conference on Learning Representations_ (2020).
  * Hsu et al. (2020) Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. 2020\.  Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10951–10960.
  * Ji et al. (2021) Jiayi Ji, Yunpeng Luo, Xiaoshuai Sun, Fuhai Chen, Gen Luo, Yongjian Wu, Yue Gao, and Rongrong Ji. 2021\.  Improving image captioning by leveraging intra-and inter-layer global representation in transformer network. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 1655–1663.
  * Jung et al. (2019) Alexander B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving, et al. 2019\.  imgaug.  https://github.com/aleju/imgaug.  Online; accessed 25-Sept-2019.
  * Koner et al. (2021) Rajat Koner, Poulami Sinhamahapatra, Karsten Roscher, Stephan Günnemann, and Volker Tresp. 2021\.  Oodformer: Out-of-distribution detection transformer.  _BMVC_ (2021).
  * Krasin et al. (2017) Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. 2017\.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  _Dataset available from https://github. com/openimages_ 2, 3 (2017), 2–3.
  * Kuznetsova et al. (2012) Polina Kuznetsova, Vicente Ordonez, Alexander C Berg, Tamara L Berg, and Yejin Choi. 2012.  Collective generation of natural image descriptions. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_. Association for Computational Linguistics, 359–368.
  * Lee et al. (2018a) Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018a.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  _Proceedings of International Conference on Learning Representations_ (2018).
  * Lee et al. (2018b) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018b.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  _Advances in neural information processing systems_ 31 (2018).
  * Li et al. (2019) Guang Li, Linchao Zhu, Ping Liu, and Yi Yang. 2019\.  Entangled transformer for image captioning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 8928–8937.
  * Li et al. (2021) Xiaoya Li, Jiwei Li, Xiaofei Sun, Chun Fan, Tianwei Zhang, Fei Wu, Yuxian Meng, and Jun Zhang. 2021\.  $k$ Folden: $k$-Fold Ensemble for Out-Of-Distribution Detection.  _EMNLP_ (2021).
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _Proceedings of International Conference on Learning Representations_ (2018).
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014.  Microsoft coco: Common objects in context. In _European conference on computer vision_. Springer, 740–755.
  * Lu et al. (2018) Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018\.  Neural baby talk. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 7219–7228.
  * MacLeod et al. (2017) Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. 2017.  Understanding blind people’s experiences with computer-generated captions of social media images. In _Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems_. ACM, 5988–5999.
  * Nguyen et al. (2015) Anh Nguyen, Jason Yosinski, and Jeff Clune. 2015\.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 427–436.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002\.  BLEU: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_. Association for Computational Linguistics, 311–318.
  * Reyes-Aldasoro and Bhalerao (2006) Constantino Carlos Reyes-Aldasoro and Abhir Bhalerao. 2006.  The Bhattacharyya space for feature selection and its application to texture segmentation.  _Pattern Recognition_ 39, 5 (2006), 812–826.
  * Sadeh et al. (2019a) Gil Sadeh, Lior Fritz, Gabi Shalev, and Eduard Oks. 2019a.  Generating Diverse and Informative Natural Language Fashion Feedback.  _CVPR Workshop on Language and Vision_ (2019).
  * Sadeh et al. (2019b) Gil Sadeh, Lior Fritz, Gabi Shalev, and Eduard Oks. 2019b.  Joint visual-textual embedding for multimodal style search.  _CVPR Workshop on Language and Vision_ (2019).
  * Shahid et al. (2019) Nida Shahid, Tim Rappon, and Whitney Berta. 2019.  Applications of artificial neural networks in health care organizational decision-making: A scoping review.  _PloS one_ 14, 2 (2019).
  * Shalev et al. (2018) Gabi Shalev, Yossi Adi, and Joseph Keshet. 2018.  Out-of-distribution detection using multiple semantic label representations. In _Advances in Neural Information Processing Systems_. 7375–7385.
  * Shalev et al. (2020) Gabi Shalev, Gal-Lev Shalev, and Joseph Keshet. 2020\.  Redesigning the classification layer by randomizing the class representation vectors.  _arXiv preprint arXiv:2011.08704_ (2020).
  * Shalev et al. (2021) Gal-Lev Shalev, Gabi Shalev, and Joseph Keshet. 2021\.  On Randomized Classification Layers and Their Implications in Natural Language Generation. In _Proceedings of the Third Workshop on Multimodal Artificial Intelligence_. 6–11.
  * Tran et al. (2016) Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, and Chris Sienkiewicz. 2016\.  Rich image captioning in the wild. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_. 49–56.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015\.  Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4566–4575.
  * Venkatakrishnan et al. (2020) Abinav Ravi Venkatakrishnan, Seong Tae Kim, Rami Eisawy, Franz Pfister, and Nassir Navab. 2020\.  Self-supervised out-of-distribution detection in brain CT scans.  _arXiv preprint arXiv:2011.05428_ (2020).
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3156–3164.

###  .1. Unknown Objects

For creating the unknown object and the cropped unknown object sets we
collected images containing 38 objects which are not appearing in COCO
training sets. The objects we fetched are: bomb, camel, calculator, dolphin,
lion, beetle, chime, dumbbell, hammer, belt, alpaca, dice, balloon, frog,
cheetah, hippopotamus, fork, fax, kangaroo, flute, cart, binoculars, insect,
armadillo, helicopter, grape, coin, handgun, otter, beaker, bat, bee, lizard,
hamster, cello, axe, barrel, egg. Examples can be seen in Fig-2. Images are
sourced from open images V4.

###  .2. Baseline Model

The baseline model was trained on COCO dataset labels: person, umbrella, tie,
backpack, handbag, suitcase, bicycle, motorcycle, bus, truck, truck, airplane,
train, boat, bench, stop sign, traffic light, fire hydrant, parking meter,
zebra, elephant, sheep, dog, bird, cat, horse, cow, bear, giraffe, surfboard,
baseball glove, kite, snowboard, frisbee, skis, sports ball, baseball bat,
skateboard, tennis racket, bowl, knife, cup, bottle, wine glass, fork, spoon,
donut, hot dog, broccoli, sandwich, banana, apple, orange, carrot, pizza,
cake, dining table, potted plant, chair, couch, bed, toilet, keyboard, mouse,
tv, laptop, remote, cell phone, refrigerator, toaster, microwave, oven, sink,
toothbrush, teddy bear, vase, book, clock, scissors, and hair drier.