# A Picture May Be Worth a Hundred Words
for Visual Question Answering

Yusuke Hirota  Osaka UniversityOsakaJapan<EMAIL_ADDRESS>,  Noa
Garcia  Osaka UniversityOsakaJapan<EMAIL_ADDRESS>,  Mayu Otani
CyberAgent, Inc.TokyoJapan<EMAIL_ADDRESS>,  Chenhui Chu  Kyoto
UniversityKyotoJapan<EMAIL_ADDRESS>,  Yuta Nakashima  Osaka
UniversityOsakaJapan<EMAIL_ADDRESS>,  Ittetsu Taniguchi  Osaka
UniversityOsakaJapan<EMAIL_ADDRESS>and  Takao Onoye  Osaka
UniversityOsakaJapan<EMAIL_ADDRESS>
###### Abstract.

How far can we go with textual representations for understanding pictures? In
image understanding, it is essential to use concise but detailed image
representations. Deep visual features extracted by vision models, such as
Faster R-CNN, are prevailing used in multiple tasks, and especially in visual
question answering (VQA). However, conventional deep visual features may
struggle to convey all the details in an image as we humans do. Meanwhile,
with recent language models’ progress, descriptive text may be an alternative
to this problem. This paper delves into the effectiveness of textual
representations for image understanding in the specific context of VQA. We
propose to take description-question pairs as input, instead of deep visual
features, and fed them into a language-only Transformer model, simplifying the
process and the computational cost. We also experiment with data augmentation
techniques to increase the diversity in the training set and avoid learning
statistical bias. Extensive evaluations have shown that textual
representations require only about a hundred words to compete with deep visual
features on both VQA 2.0 and VQA-CP v2.

††ccs: Computing methodologies Computer vision††ccs: Computing methodologies
Natural language processing††ccs: Computing methodologies Image
representations

##  1\. Introduction

Figure 1. Model overview. Our language-only model takes a question and a
description as the input of a language-only Transformer model and predicts an
answer accordingly.

Model Overview

The English proverb A picture is worth a thousand words is used to convey the
idea that, for humans, the meaning of a complex verbal description is often
better understood with a single static image. However, identifying,
understanding, and reasoning over the concepts contained in an image is still
highly challenging for computer vision systems. Whereas standard deep visual
features based on vectors may present some limitations to capture the rich
semantic content from a picture (Agrawal et al., 2018, 2016; Goyal et al.,
2017), descriptive representations based on text may be a good alternative,
especially with the recent emergence of powerful language models. This paper
aims to provide a deep analysis of textual representations for image
understanding tasks, and compare them with existing deep visual features
techniques.

A practical task to evaluate image understanding is visual question answering
(VQA). VQA aims to answer questions about an image’s visual content, requiring
a machine to understand both the question and the image. For describing the
visual content, the way in which images are represented is essential, as it
highly impacts the performance of the downstream task. Due to the bottom-up
attention’s success (Anderson et al., 2018; Kim et al., 2018; Jiang et al.,
2018), deep visual features extracted by object recognition models (Ren et
al., 2015) have been used as the de-facto standard for representing visual
content.

On the other hand, the recent progress of Transformer language models (Vaswani
et al., 2017; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019)
produced outstanding advances in several natural language processing (NLP)
tasks, e.g., question answering (Rajpurkar et al., 2016) or sentiment analysis
(Maas et al., 2011a). These language models can simultaneously represent
inter-relationships between two consecutive sentences and intra-relationships
between the individual words in a sentence. With the development of
Transformer language models, research on vision-and-language tasks (Lu et al.,
2019; Tan and Bansal, 2019; Chen et al., 2020a; Su et al., 2020; Li et al.,
2019b) has shifted to explore pre-training methods to learn cross-modal
representations on image-text pairs. These pre-trained Transformers are fine-
tuned on the downstream vision-and-language tasks, outperforming standard
methods based on convolutional neural networks. Within this context, one
interesting question arises here: according to the popular proverb, humans may
need a thousand words to describe an image; but how many words are necessary
for Transformer language models?

In this paper, we explore the effectiveness of textual representations of
images and explore if they are competitive with current deep visual features.
Specifically, we conduct VQA experiments by representing images using text,
instead of deep visual features. Thus, we change the input of a VQA model from
image-question pairs to image description-question pairs. We use a RoBERTa
(Liu et al., 2019), a state-of-the-art Transformer language-only model, as a
our VQA model. Then, the input description-question pairs are jointly fed into
the model to predict an answer (Figure 1). Besides, with the success of data
augmentation methods on both VQA (Gokhale et al., 2020; Chen et al., 2020b)
and NLP (Wei and Zou, 2019; Sennrich et al., 2016; Yu et al., 2018) tasks, we
investigate the use of synthetic samples on language-only representations. As
the aim of the study is to explore the viability of language-only
representations in VQA, we rely on already annotated descriptions from two
standard datasets (Chen et al., 2015; Pont-Tuset et al., 2020). Automatically
generating the image descriptions, although a necessary future step, is out of
the scope of this paper, and would require a whole paper on its own. We will
carefully study image description generation in our future research.

Extensive experiments and analysis, including qualitative and quantitative
results, have shown the effectiveness of textual representation of images. We
have found that well-described descriptions can outperform deep visual
features on two standard VQA datasets, VQA 2.0 and VQA-CP. Additionally, we
validated that it does not take a thousand words to describe an image, but
about a hundred.

The key contributions of our work are summarized as follows.

  * •

Different from previous VQA models, we use text as image representation. Our
setup allows us to study how well textual representation works in VQA. We show
that textual representations with about a hundred words are competitive with
state-of-the-art deep visual features. This observation promotes more
interpretable representations of images in that it gives visibility to what
humans easily understand.

  * •

To increase the diversity on the training set, we explore multiple data
augmentation techniques. We propose single-modality adaptations to novel VQA
data augmentation techniques, as well as we apply standard existing techniques
on the NLP community. We find that creating synthetic data consistently
improves our language-only model performance. Prominently, back translation
for questions boosts the performance by a large margin, not only in our
language-only setting but also for deep visual feature-based models.

##  2\. Related work

Image Representations for VQA. In vision and language tasks, visual features
have played a key role in leveraging the visual content of images. Most VQA
methods (Yang et al., 2016; Anderson et al., 2018; Kim et al., 2018; Jiang et
al., 2018; Cadène et al., 2019a, b; Clark et al., 2019) use deep visual
features extracted by object detectors, such as Faster R-CNN (Ren et al.,
2015). These deep visual features are a set of pooled convolutional feature
vectors, in which each vector represents a prominent image region. Also, some
models that use deep visual features extracted by CNN models, such as Resnet
(He et al., 2016), can achieve impressive results (Jiang et al., 2020; Huang
et al., 2021). However, VQA models using deep visual features tend to learn
superficial linguistic correlations (Agrawal et al., 2018, 2016; Goyal et al.,
2017).

Building on top of the recent progress in language models, some work has also
adapted Transformer models to fuse visual and textual information. Recent
studies (Lu et al., 2019; Tan and Bansal, 2019; Chen et al., 2020a; Su et al.,
2020; Li et al., 2019b) achieve high performance on vision-and-language tasks,
such as VQA, by pre-training multi-layer Transformers with image-caption
pairs. As a result, these models can learn general cross-modal
representations. Even though they leverage captions of images for pre-
training, they rely on deep visual features for image representation.

Similarly, image captioning (You et al., 2016; Yao et al., 2017) can be seen
as a related task to text representation for images. Yet, converting images
into text is different from our purpose of using text for representing images
on VQA. Recent work (Garcia and Nakashima, 2020) has shown that textual
representations for video question answering could outperform models based on
deep visual features. Based on these results, we present a rigorous analysis
on textual representations for VQA.

Data Augmentation for VQA. Recent work (Chen et al., 2020b; Gokhale et al.,
2020) proposed data augmentation techniques for VQA to deal with language bias
(Agrawal et al., 2018, 2016; Goyal et al., 2017). They generated
counterfactual training samples by masking, removing, or changing critical
parts of the input. Training with these synthesized samples makes models focus
on the crucial parts in an image and question, leading to alleviate the
problem. Inspired by these studies, we propose data augmentation techniques
for textual representations.

Data Augmentation for NLP. Various techniques of data augmentation have been
proposed to improve NLP tasks. A well-known method is back translation
(Sennrich et al., 2016; Yu et al., 2018; Ott et al., 2019) which generates new
data by translating sentences to another language and back into the original
language. Other techniques such as EDA (Wei and Zou, 2019) boosted the
performance on text classification tasks (Socher et al., 2013; Hu and Liu,
2004; Pang and Lee, 2004; Li and Roth, 2002; Ganapathibhotla and Liu, 2008).
In addition to the data augmentation techniques for VQA, we adopt these
methods for our language-only representations, which enables us to check if
data augmentation for NLP is also effective for VQA.

##  3\. Approach

Figure 2. Generated synthetic samples using our proposed data augmentation for
VQA techniques.

Data augmentation examples

We introduce the text-based model with which we investigate the potential of
language-only representations for VQA. An overview is shown in Figure 1. The
input, consisting of a question and a detailed description of the image, is
encoded through a Transformer language-only model with several self-attention
layers. Then, the output of the Transformer is fed into a classifier to
predict an answer. We additionally propose the use of data augmentation
techniques to increase the size and diversity of the training set.

###  3.1. Language-Only Data

The data for our language-only VQA framework consists of: (1) questions and
answers from standard VQA datasets, (2) image descriptions representing image
content, and (3) synthetic data obtained from data augmentation techniques.

Questions and Answers. For the questions and answers, we use VQA-CP (Agrawal
et al., 2018) and VQA 2.0 (Antol et al., 2015) datasets. VQA 2.0 contains
$1.1$M question-answer pairs of about $204$K images, whereas VQA-CP consists
of $603$K question-answer pairs of about $219$K images. All images of VQA 2.0
and VQA-CP are from COCO dataset (Lin et al., 2014). Whereas VQA 2.0 has been
the de-facto standard for natural image VQA, it has also been shown to contain
strong statistical biases on its training distribution (Agrawal et al., 2018,
2016; Goyal et al., 2017), by which models obtain high accuracy by only using
the first few words of a question. VQA-CP addresses this problem by re-
organizing the training and validation splits to have a different answer
distribution.

Image Descriptions We obtain image descriptions from two different corpus:
COCO captions (Chen et al., 2015) and Localized Narratives (Pont-Tuset et al.,
2020). COCO captions contains five short captions for each image in COCO
dataset (Lin et al., 2014), with an average length of $10.5$ words per
caption. Captions are obtained by asking annotators to describe the important
parts of the scene, without mentioning unimportant details. Localized
Narratives provides multimodal image annotations for multiple image datasets
(COCO, Flickr30k (Young et al., 2014), ADE20K (Zhou et al., 2019), and Open
Images (Kuznetsova et al., 2020)). The narratives are generated by asking
annotators to describe an image out loud while simultaneously hovering the
mouse over the region they are describing, representing the entire image,
including minor objects as opposed to COCO captions. This results in an
average of $42.9$ words per narrative. We only use narratives corresponding to
COCO images.

Synthetic Data Additionally, we generate synthetic samples using data
augmentation techniques to increase the size and diversity of the training
data in our language-only setting. We explore multiple techniques, which can
be grouped into two main categories: Data Augmentation for VQA (Section 3.2)
and Data Augmentation for Language (Section 3.3). In particular, Data
Augmentation for VQA (DAV) are techniques based on the latest multimodal data
augmentation methods (Gokhale et al., 2020; Chen et al., 2020b) which propose
to generate new images and new questions to increase the diversity in the
dataset. Since we do not use the images directly, we propose single modality
methods that generate synthetic descriptions that help mitigate the effects of
language bias. As for Data Augmentation for Language (DAL), we leverage
various established methods within the NLP community for improving language-
only tasks (Wei and Zou, 2019; Sennrich et al., 2016; Yu et al., 2018; Ott et
al., 2019).

###  3.2. Data Augmentation for VQA

We adapt data augmentation techniques for VQA (Gokhale et al., 2020; Chen et
al., 2020b) to our language-only input setting. The aim is to generate new
samples that force the model to react to essential parts in the input and
provide an answer without relying on the language bias. While image data
augmentation methods for VQA (Teney et al., 2020; Agarwal et al., 2020; Chen
et al., 2020b; Gokhale et al., 2020) mask objects or employ GANs (Goodfellow
et al., 2014) to change the scene in an image, which is computationally
expensive, we only require word replacement.

Let $s$ denote a training triplet, i.e., ${s=(\mathbf{q},\mathbf{d},A)}$.
$\mathbf{q}=[q_{1},\cdots,q_{Q}]$ is a sequence input question with $Q$ tokens
and associated with a question type $t$, ${\mathbf{d}=[d_{1},\cdots,d_{D}]}$
is a sequence input description with $D$ tokens, and
$A=\\{a_{1},\cdots,a_{N}\\}$ is the set of $N$ ground truth answers, where $N$
is at most 10 for the VQA 2.0 and VQA-CP datasets and can vary for different
samples. We propose four techniques: (1) hypernym and hyponym replacement, (2)
color inversion, (3) adversarial replacement, and (4) counterfactual samples.
An example of each technique is shown in Figure 2.

Hypernym and Hyponym Replacement. Following (Gokhale et al., 2020), we use
hypernym and hyponym replacement to introduce similar yet semantically
distinct mutations into the image descriptions. For a given word, an hypernym
covers a wider range of concepts that the original word implies, e.g. food is
an hypernym of fruit. On the other hand, an hyponym covers a narrower range of
meanings, e.g. apple is an hyponym of fruit.

To generate new samples, let $A_{\mathbf{d}}$ denote the set of ground truth
answers $a$ such that $a$ appears in $\mathbf{d}$. We replace $a\in
A_{\mathbf{d}}$ with its hypernym $h_{e}(a)$ (or hyponym $h_{o}(a)$) in both
$A$ and $\mathbf{d}$. The new training triplet for hypernym replacement
$s_{\text{h}}$ is:

(1) $s_{\text{h}}=(\mathbf{q},\mathbf{d}_{\text{h}},A_{\text{h}})$ (2)
$A_{\text{h}}=A\setminus A_{\mathbf{d}}\cup\\{h_{e}(a)\\}_{a\in
A_{\mathbf{d}}}$ (3) $\mathbf{d}_{\text{h}}=[d_{1},\cdots,d_{L}]\text{ with
$d_{i}=h_{e}(a)$ if $d_{i}=a$ for all $a\in A_{\mathbf{d}}$},$

and equivalently with $h_{o}(a)$ for hyponym replacement. To avoid duplicates,
we do not generate $s_{\text{h}}$ if $h_{e}(a)\in A$ (or $h_{o}(a)\in A$). To
identify hypernyms and hyponyms, we use WordNet (Miller, 1995).

Color Inversion. For color inversion (Gokhale et al., 2020), we substitute a
color word in a description with another color word. First, we manually create
a set $C=\\{c_{1},\cdots,c_{K}\\}$ of $K$ color words, and a set $T_{C}$ of
question types related to colors.111$T_{C}=\\{\text{``what
color''},\text{``what color are the''},\text{``what color is''},\text{``what
color is the''}\\}$. For a given training triplet $s$ and its question type
$t$, if $t\in T_{C}$ and a ground truth color answer $a\in A\cap C$ is found
in the input description (i.e. $d_{i}=a$ for some $i$), we replace the
description and the answer with a different random color word $c\neq a$. The
new training triplet $s_{\text{c}}$ is:

(4) $s_{\text{c}}=(\mathbf{q},\mathbf{d}_{\text{c}},A_{\text{c}})$ (5)
$A_{\text{c}}=A\setminus\\{a\\}\cup\\{c\\}$ (6)
$\mathbf{d}_{\text{c}}=[d_{1},\cdots,d_{L}]\text{ with $d_{i}=c$ if
$d_{i}=a$}$

Adversarial Replacement. For Yes/No samples, i.e.
$s=(\mathbf{q},\mathbf{d},A)$ such that $\\{\text{yes},\text{no}\\}\cup
A\notin\emptyset$, we replace object words $o\in O$ in description
$\mathbf{d}$ with adversarial words, where $O$ is the set of 80 object classes
in COCO dataset (Lin et al., 2014). Following (Gokhale et al., 2020), we
define adversarial word, $w_{\text{adv}}(o)$, as the word that is the most
similar yet with a different meaning to $o$. If $o$ (or its synonyms) are in
$\mathbf{q}$, we change the answer from yes to no; otherwise the answer is not
changed. Formally, the new training triplet $s_{\text{a}}$ is:

(7) $s_{\text{a}}=(\mathbf{q},\mathbf{d}_{\text{a}},A_{\text{a}})$ (8)
$A_{\text{a}}=\begin{cases}~{}\\{\text{no}\\}&\text{if $o$ is in
$\mathbf{q}$}\\\ ~{}A&\text{otherwise}\end{cases}$ (9)
$\mathbf{d}_{\text{a}}=[d_{1},\cdots,d_{L}]\text{ with
$d_{i}=w_{\text{adv}}(o)$ if $d_{i}=o$},$

where $w_{\text{adv}}(o)$ is selected as the closest word to $o\in O$
according to the Euclidean distance between their Glove embeddings (Pennington
et al., 2014).

Differently from (Gokhale et al., 2020), we do not generate adversarial
samples from questions, as changing or masking a word in a question leads to a
new answer, which may not be determined automatically, e.g., “How many bins?”
→“How many pens?”.

Counterfactual Samples. Counterfactual samples (Chen et al., 2020b) are
modifications of questions or (part of) images that make the original
question-answer pairs irrelevant. We generate counterfactual training samples
by adapting (Chen et al., 2020b) to language-only description-question pairs.
Given the training triplet $s=(\mathbf{q},\mathbf{d},A)$, we generate
counterfactual samples $s_{\text{css}_{q}}$ and $s_{\text{css}_{d}}$ from the
query and description:

(10) $\displaystyle
s_{\text{css}_{\text{q}}}=(\mathbf{q}_{\text{css}_{\text{q}}},\mathbf{d},A_{\text{css}_{\text{q}}})$
(11) $\displaystyle
s_{\text{css}_{\text{d}}}=(\mathbf{q},\mathbf{d}_{\text{css}_{\text{d}}},A_{\text{css}_{\text{d}}})$

To find $s_{\text{css}_{\text{q}}}$, we feed $\mathbf{q}$ and $\mathbf{d}$
into a trained Transformer VQA model on the target dataset, $\mathcal{M}$, and
obtain the contribution of each word in $\mathbf{q}$ to the answer set $A$
with Grad-CAM (Selvaraju et al., 2017). The top-D words with highest
contribution are selected as a set $\Omega_{\text{q}}$ of critical words. Two
new questions $\mathbf{q}_{\text{css}_{\text{q}}}^{+}$ and
$\mathbf{q}_{\text{css}_{\text{q}}}^{-}$ are generated by masking all the
words in $\mathbf{q}$ not contained in $\Omega_{\text{q}}$, and masking all
the words contained in $\Omega_{\text{q}}$, respectively:

(12)
$\displaystyle\mathbf{q}_{\text{css}_{\text{q}}}^{+}=[q_{1},\cdots,q_{L}]$
$\displaystyle\quad\text{ with }q_{i}=\text{<mask>}\quad\text{for all
}q_{i}\notin\Omega_{\text{q}}$ (13)
$\displaystyle\mathbf{q}_{\text{css}_{\text{q}}}^{-}=[q_{1},\cdots,q_{L}]$
$\displaystyle\quad\text{ with }q_{i}=\text{<mask>}\quad\text{for all
}q_{i}\in\Omega_{\text{q}},$

where $\mathbf{q}_{\text{css}_{q}}^{-}$ is used as this CSS sample’s question
$\mathbf{q}_{\text{css}_{q}}$ and in Eq. (12), the first few words that
correspond to the question type (e.g., what color is) are not masked, as in
(Chen et al., 2020b).

To find $A_{\text{css}_{\text{q}}}$, $\mathbf{q}_{\text{css}_{\text{q}}}^{+}$
and $\mathbf{d}$ are again fed into $\mathcal{M}$ to obtain the score for each
candidate answer. The top-J scoring answers are removed from the original set
of answers. Specifically, letting
$\mathcal{M}_{J}(\mathbf{q}_{\text{css}_{\text{q}}}^{+},\mathbf{d})$ be the
set of the top-J answers, the new ground truth answers is given by

(14)
$A_{\text{css}_{\text{q}}}=A\setminus\mathcal{M}_{J}(\mathbf{q}_{\text{css}_{\text{q}}}^{+},\mathbf{d}).$

For $s_{\text{css}_{\text{d}}}$, the values of $\Omega_{\text{d}}$,
$\mathbf{d}^{+}_{\text{css}_{\text{d}}}$,
$\mathbf{d}^{-}_{\text{css}_{\text{d}}}$, and
$M_{N}(\mathbf{q},\mathbf{d}^{+}_{\text{css}_{\text{d}}})$ are found in the
same way.

###  3.3. Data Augmentation for Language

Given that the input to our VQA model is solely based on the language
modality, we also explore NLP data augmentation techniques. Among all existing
techniques, we adopt three of the most popular and successful ones (Wei and
Zou, 2019; Kumar et al., 2020; Xie et al., 2020; Yu et al., 2018; Peters et
al., 2018b; Peters et al., 2018a): (1) EDA, (2) back translation, and (3) word
replacement/insertion via contextual word embedding. Each technique is applied
to either the description or the question of the input triplet $s$ to generate
new samples:

(15) $\displaystyle
s_{\text{nlp}_{\text{q}}}=(\mathbf{q}_{\text{nlp}},\mathbf{d},A)$ (16)
$\displaystyle
s_{\text{nlp}_{\text{d}}}=(\mathbf{q},\mathbf{d}_{\text{nlp}},A)$

where $\mathbf{q}_{\text{nlp}}$ and $\mathbf{d}_{\text{nlp}}$ are the question
and the description, respectively, after applying one of the transformations
below. Examples of synthetic samples are shown in Table 1.

Table 1. Examples of Data Augmentation for Language.
---
DAL Method | Synthetic Question
Original | What is the giraffe standing behind?
EDA (SR) | What is the camelopard standing behind?
EDA (RI) | What is the giraffe abide standing behind?
EDA (RS) | What is the standing giraffe behind?
EDA (RD) | What is the standing behind?
BT | What’s behind the giraffe?
CWR | What is the giraffe tree behind?
CWI | What is the giraffe standing silently behind?

EDA (Easy Data Augmentation) (Wei and Zou, 2019) is composed of 4 operations:

  * •

Synonym Replacement randomly chooses $n$ words from the input sentence and
replaces them with their synonyms.

  * •

Random Insertion finds a random synonym of a random word in the sentence and
inserts it in an arbitrary position.

  * •

Random Swap chooses two words and swaps them.

  * •

Random Deletion removes words with probability $p$.

While EDA consists of simple word replacement-based techniques, it has been
shown to improve text classification performance in low-resource tasks (Wei
and Zou, 2019). For each sample, we apply one of these four operations
randomly.

Back Translation (Sennrich et al., 2016; Yu et al., 2018) translates a
sentence into another language and then translates it back into the original
language. As shown in (Xie et al., 2020; Yu et al., 2018), back translation
can generate diverse paraphrases while preserving the original sentences’
semantics, resulting in significant performance improvements in various NLP
tasks (Rajpurkar et al., 2016; Maas et al., 2011b; Zhang et al., 2015). For
the implementation of back translation, we use the python nlpaug library (Ma,
2019) based on fairseq (Ott et al., 2019; Ng et al., 2019). The translator is
based on the big Transformer architecture (Vaswani et al., 2017) and
translates a given sentence to German and back into English. If the sentence
has not changed after applying the back translation, we discard it.

Contextual Word Replacement/Insertion replaces or inserts words by considering
the context. To obtain the words that match the context, it uses deep
bidirectional language models (Devlin et al., 2019; Peters et al., 2018a; Yang
et al., 2019). We again use the implementation of contextual word embedding
provided by python nlpaug library (Ma, 2019). We replace or insert the most
similar words of randomly selected words in the description or question to
generate a new one, where we use pre-trained XLNET (Yang et al., 2019) in the
library to obtain the contextual word embeddings.

###  3.4. Language-Only VQA Model

As opposed to most VQA models, which take image-question pairs as input, the
input of our language-only VQA model is description-question pairs. Given the
original VQA triplet $(\mathbf{q},I,A)$, where $I$ is the image associated
with the question, we first create the image description $\mathbf{d}$ by
concatenating the narrative of Localized Narratives and the captions of COCO
captions. The question and the image description are compiled into a single
sequence, $\mathbf{l}$, by inserting a classifier token <s> at the beginning
of the sequence, and end of sentence tokens </s> as:

(17)
$\mathbf{l}={\small\texttt{<s>}}+\mathbf{q}+{\small\texttt{</s>}}+{\small\texttt{</s>}}+\mathbf{d}+{\small\texttt{</s>}}$

where “$+$” indicates concatenation. The input sequence is fed into our
Transformer-based language model $\mathcal{T}$ to obtain sequence $\mathbf{f}$
of embeddings, i.e.,

(18) $\mathbf{f}=\mathcal{T}(\mathbf{l}).$

Then, the embedding corresponding to the classifier token <s> is fed into
classifier $\mathcal{C}$, to make a prediction:

(19) $\bm{\rho}=\mathcal{C}(\mathbf{f}_{\small\texttt{<s>}})$

Unless otherwise stated, we use large RoBERTa (Liu et al., 2019) as our
Transformer language model, which has been shown to be one of the best
performing models on NLP tasks (Saeedi et al., 2020; Minaee et al., 2020).

##  4\. Experiments

We conducted five main experiments to validate the effectiveness of textual
representations for VQA: (1) comparison between different image descriptions,
(2) comparison between data augmentation techniques, (3) comparing textual
representations with deep visual features, (4) evaluating the impact of data
augmentation for questions on deep visual feature-based models, and (5)
comparison between language-only Transformers.

For RoBERTa hyperparameters, we followed (Liu et al., 2019), only changing the
number of training epochs to $20$. As a classifier, we used a multi-layer
perceptron with two fully-connected layers with $2048$ hidden units, and Swish
activation function (Ramachandran et al., 2017) between them. We use softmax
cross entropy over the answer vocabulary for the loss function. The parameters
for the data augmentation techniques were as follows: for counterfactual
samples, we set top-D=$10$ to select critical words and top-J=$5$ for
assigning new answers; for EDA, we set the rate of the words to be changed to
$0.1$, following the recommended value in the original paper (Wei and Zou,
2019). Unless otherwise stated, the input of our model consists of the whole
sequence of question, narrative, and five captions.

Results are presented in terms of accuracy, following the standard protocol
(Antol et al., 2015). For the datasets split, the VQA-CP v2 training set
contains $438$K questions about $120$K images, whereas the test set has $220$K
questions about $98$K images. As for VQA 2.0, the training set includes $443$K
questions about $82$K images, while the val set contains $214$K questions
about $40$K images. As the VQA 2.0 test set requires access to external
server, we report results on the val set.

###  4.1. Describing an Image with Words

Table 2. Image Description Evaluation. Length denotes the mean number of
tokens in the image descriptions.
---
Image Description | Length | Accuracy
None (Question-Only) | - | 21.39
1 Caption | 10.5 | 35.31
2 Captions | 21.0 | 38.49
3 Captions | 31.5 | 40.09
4 Captions | 42.0 | 41.93
5 Captions | 52.5 | 42.34
Narrative | 42.9 | 36.45
Whole (Narrative + 5 Captions) | 95.3 | 43.64

Figure 3. Effect of truncating the input sequence on the VQA CP v2 test set.
Removing words from the input has a negative effect on the accuracy,
suggesting that every word in the sequence contributes to the overall
accuracy.

Answer overlap

Here, we investigated the relationship between the quality of the image
descriptions and the accuracy on the VQA task by conducting two experiments:
(1) comparing the performance of different types of image descriptions and (2)
studying the correlation between accuracy and the length of the input
descriptions.

Image descriptions. We first evaluate the performance of different language-
only inputs. Specifically, we consider the following inputs: only the
question, question and 1 to 5 random captions, question and narrative, and the
whole input (question, narrative, and 5 captions). Results on the VQA-CP v2
test set are reported in Table 2, along with the average sequence length.

The whole input, consisting on merging the narrative with the 5 captions with
an average of $95.3$ tokens per sample, performs the best, which indicates
that the two datasets contain complimentary useful information for VQA. When
comparing captions and narratives, we find that the former produce better
results with fewer words. Specifically, by just using two captions, the
performance is already better than with narratives, even though the average
number of words is about half. This confirms that the VQA dataset contains a
substantial amount of questions about the general content of the image, rather
than its specific details, as the captions in COCO dataset, in contrast to
narratives, focus mostly on the prominent areas in the scene. When using a
single caption, the accuracy improves by $13.92$% compared to the Question-
Only input, showing that a short description of about $10$ words already
contains enough information to answer correctly one in three questions.

Input length For studying the correlation between accuracy and length of the
input, we progressively truncate the image descriptions at test time and
evaluate their performance on a model trained on the full input without
truncation (question, narrative, and 5 captions). To truncate the input, we
rearrange the sentences randomly, truncate the words at the end of the
rearranged sequence, and put the sentences back in their original order to
avoid losing contextual information. Questions are not truncated in any case.
Results are shown in Figure 3, where it can be seen that the accuracy
decreases as the image descriptions are trimmed. In particular, the accuracy
decreases linearly up to $60$%. For more than $60$% of removal rate, the model
cannot deal with the information loss and the accuracy decreases rapidly. This
tendency suggests that most words on the input have a positive contribution to
the overall accuracy.

Table 3. Data augmentation results on the VQA-CP v2 test set. In DAL, D and Q
denote when applied to descriptions or questions, respectively. Gap is the
overall accuracy difference compared to the accuracy when not using synthetic
samples.
---
| Input Data | Num. Synthetic | Num. Total | Yes/No | Number | Other | Overall | Gap
| Narrative + 5 Captions | - | 438,183 | 45.13 | 20.06 | 49.33 | 43.64 | -
DAV | w/ Hyponym Replacement | 132,570 | 570,753 | 45.65 | 25.36 | 50.52 | 45.26 |  $+$1.62
w/ Hypernym Replacement | 23,869 | 462,052 | 47.28 | 17.69 | 49.10 | 43.70 |  $+$0.06
w/ Hyponym and Hypernym Replacement | 183,944 | 622,177 | 45.80 | 21.46 | 51.15 | 45.06 |  $+$1.42
w/ Color Inversion | 19,308 | 457,491 | 45.61 | 19.93 | 50.60 | 44.47 |  $+$1.06
w/ Adversarial Word Replacement | 169,929 | 608,112 | 44.71 | 19.84 | 50.03 | 43.93 |  $+$0.29
w/ Counterfactual Samples | 438,183 | 876,366 | 44.20 | 19.84 | 52.07 | 44.86 |  $+$1.22
DAL | w/ EDA (D) | 438,183 | 876,366 | 44.68 | 20.64 | 50.08 | 44.02 |  $+$0.38
w/ EDA (Q) | 438,183 | 876,366 | 46.86 | 23.50 | 50.62 | 45.39 |  $+$1.75
w/ Contextual Word Replacement (D) | 438,183 | 876,366 | 44.69 | 19.40 | 48.91 | 43.18 |  $-$0.46
w/ Contextual Word Replacement (Q) | 438,183 | 876,366 | 46.09 | 22.49 | 49.10 | 44.16 |  $+$0.52
w/ Contextual Word Insertion (D) | 438,183 | 876,366 | 45.15 | 19.31 | 48.86 | 43.27 |  $-$0.37
w/ Contextual Word Insertion (Q) | 438,183 | 876,366 | 45.86 | 21.44 | 51.10 | 45.05 |  $+$1.41
w/ Back Translation (D) | 438,183 | 876,366 | 45.28 | 21.01 | 50.89 | 44.70 |  $+$1.06
w/ Back Translation (Q) | 293,811 | 731,994 | 62.43 | 27.15 | 51.84 | 51.16 |  $+$7.52

###  4.2. Use of Synthetic Samples

We evaluate the performance when augmenting the VQA-CP v2 training set with
synthetic samples. The input consists on the whole description with the
narrative and the five captions.

Results for each of the proposed data augmentation techniques are shown in
Table 3. Whereas most techniques are able to increase the accuracy with
respect to the baseline (when no data augmentation is used), back translation
for questions has by far the best results, with a gain of $7.52$ points on
overall, and a boost of $17.30$ points on Yes/No questions. As back
translation is the only data augmentation technique in Table 3 that generates
new samples while maintaining the original semantics, its impressive
performance points us to the importance of 1) having diversity within the
training questions set, whereas at the same time, 2) a correct relationship
between the triplet question-description-answer semantics.

Most of the other DAL techniques, except contextual word replacement/insertion
for descriptions, also increase the accuracy with respect to the baseline. An
interesting observation is that when data augmentation is applied to the
questions, the performance consistently improves more than when it is applied
to descriptions. As for the DAV techniques, while all of them improve
accuracy, hyponym replacement turns out to be the most powerful one, with an
overall accuracy gain of $1.62$. Surprisingly, using hyponym and hypernym
replacement together does not perform better than hyponym replacement by its
own, showing that in some cases the combination of synthetic samples obtained
from different techniques may be harmful.

###  4.3. Comparison against Deep Visual Features

Table 4. Comparison of language-only representations with standard deep visual
features. ∗ indicates our re-implementations.
---
|  | VQA-CP v2 test |  | VQA 2.0 val
Model |  | Yes/No | Number | Other | Overall |  | Yes/No | Number | Other | Overall
HAN (Malinowski et al., 2018) |  | 52.25 | 13.79 | 20.33 | 28.65 |  | - | - | - | -
RAMEN (Shrestha et al., 2019) |  | - | - | - | 39.21 |  | - | - | - | -
MuRel (Cadène et al., 2019a) |  | 42.85 | 13.17 | 45.04 | 39.54 |  | - | - | - | 65.14
UpDn (Anderson et al., 2018) |  | 42.27 | 11.93 | 46.05 | 39.74 |  | 81.18 | 42.14 | 55.66 | 63.48
ReGAT (Li et al., 2019a) |  | - | - | - | 40.42 |  | - | - | - | 67.18
$\text{BAN}^{\ast}$ (Kim et al., 2018) |  | 43.14 | 13.63 | 46.92 | 40.74 |  | 83.19 | 48.13 | 57.52 | 65.93
$\text{LXMERT}^{\ast}$ (Tan and Bansal, 2019) |  | 42.01 | 14.16 | 48.34 | 41.28 |  | 83.30 | 46.15 | 56.91 | 65.31
NSM (Hudson and Manning, 2019) |  | - | - | - | 45.80 |  | - | - | - | -
Ours (Narrative + 5 Captions) |  | 45.13 | 20.06 | 49.33 | 43.64 |  | 87.91 | 56.47 | 59.43 | 69.74

We compare our whole language-only representations (question, narrative, and
five captions) with state-of-the-art VQA models based on deep visual features
on both the VQA-CP v2 and the VQA 2.0 datasets. For a fair comparison, we do
not include the models developed to mitigate language bias (Cadène et al.,
2019a, b; Clark et al., 2019), as these modules can be added as a plug-in
extension to any other method, including ours. For the same reason, we do not
use data augmentation.

Results are reported in Table 4. Our model outperforms most of the baselines
that use deep visual features on both VQA-CP v2 and VQA 2.0. For the accuracy
on VQA-CP v2, NSM (Hudson and Manning, 2019) performs slightly better than our
language-only model. This result verifies that the textual representations of
the images are effective for both cases where the distribution of answers per
question is different between the training and test set and when it is not.
This also shows the textual representations capture well the image content,
such that they are competitive with deep visual features.

###  4.4. Back Translation for Other Models

Table 5. Results of applying back translation to different VQA models in the
VQA-CP v2 test set. Gap represents the improvement by training with the
synthetic samples.
---
Model | Yes/No | Number | Other | Overall | Gap
BAN (Kim et al., 2018) | 43.14 | 13.63 | 46.92 | 40.74 | -
w/ back translation | 47.87 | 16.27 | 48.76 | 43.57 | +2.83
LXMERT (Tan and Bansal, 2019) | 42.01 | 14.16 | 48.34 | 41.28 | -
w/ back translation | 48.81 | 16.16 | 49.75 | 44.35 | +3.07
Ours | 45.13 | 20.06 | 49.33 | 43.64 | -
w/ back translation | 62.43 | 27.15 | 51.84 | 51.16 | +7.52

The results of data augmentation techniques in Section 4.2 showed that
applying back translation to questions is very effective to boost accuracy in
our language-only setting. As back translation can be easily applicable to any
other settings, we explore if its benefits are transferable to standard VQA
models. We conduct experiments on BAN (Kim et al., 2018) and LXMERT (Tan and
Bansal, 2019) by adding synthetic back translated samples (applied to
questions) to the training set.

Results are shown in Table 5. We can see that training BAN and LXMERT with
synthesized back translated samples improves the performance by a large
margin, with gaps of $2.83$ and $3.07$ points, respectively. All types of
questions are benefited from this technique, but especially Yes/No with an
improvement of about $6.80$. This tendency is consistent with the findings in
our model, strongly indicating that leveraging additional samples containing
questions with the same meaning but expressed in a different way is extremely
beneficial. In other words, adding diversity to the questions increases the
model’s ability to interpret the questions.

###  4.5. Language Transformers

Table 6. Evaluation of different language-only Transformer models on the VQA-
CP v2 test set.
---
Model | Yes/No | Number | Other | Overall
BERT base | 42.66 | 16.19 | 45.66 | 40.29
BERT large | 42.72 | 17.43 | 48.47 | 42.06
XLNET base | 43.49 | 17.61 | 48.45 | 42.30
XLNET large | 44.58 | 20.67 | 50.52 | 44.23
RoBERTa base | 44.39 | 17.46 | 48.74 | 42.70
RoBERTa large | 45.13 | 20.06 | 49.33 | 43.64

Finally, we compare the performance of prevailing language-only Transformer
models: BERT (Devlin et al., 2019), XLNET(Yang et al., 2019), and RoBERTa(Liu
et al., 2019), both in their base and large versions. All the models are
exposed to the same input, consisting of the question, the narrative, and the
five captions.

Results are shown in Table 6. All the models present a similar behavior. XLNET
large has the best performance, and its accuracy is higher $0.59$% compared to
RoBERTa large. However, while the improvement is minor, the computational time
of XLNET large is about $2.7$ times that RoBERTa. Considering this, it is
reasonable to use RoBERTa large to save the training time while obtaining
relatively similar results.

##  5\. What do Textual Representations Learn?

In this section, we analyze the features and advantages of text for
representing images compared with deep visual features. To this end, we
conducted the following two analysis: (1) exploring the overlap of the
predictions between our language-only model with the models using deep visual
features, (2) investigating some visual examples for qualitative analysis. The
input to our models is formed by combining the narrative with the five
captions.

Figure 4. Qualitative Comparison. The red boxes in the images denote the
result of detection by Faster R-CNN. Only bounding boxes with a confidence
score greater than $\bm{0.5}$ are shown. The shown descriptions are extraction
of the actual descriptions. The red highlighted words are the relevant words
to the answers.

Qualitative comparison

###  5.1. Error Analysis

Figure 5. Answer Overlap. The ratio of same or different predictions between
our language-only model and BAN or LXMERT. Bar graphs denote the proportion of
the total.

Answer overlap

We compare our model that only takes text as input with models that use deep
visual features and investigate whether our model makes the same or different
mistakes. We adopt BAN and LXMERT as representative models leveraging deep
visual features. Whereas BAN was one of the top-performing models before the
emergence of the Transformer-based vision-and-language models, LXMERT is one
of the state-of-the-art pre-trained models based on multi-modal Transformers
extracting cross-modal features. LXMERT can be seen as the middle model
between our language-only model and BAN, as LXMERT is pre-trained by taking a
pair of image and caption as input.

Figure 5 compares our model with BAN and LXMERT in terms of the consistency of
correct and incorrect answers. Note that the ratios of correct answers are
higher than the results in Table 4 because the original evaluation method uses
a non-binary accuracy metric for each prediction (Antol et al., 2015), while
for this comparison, we employed a binary one where a prediction is counted to
be correct if the predicted answer is in $A$ for simplicity. We can see that
our model shows the relatively high consistency, and the ratios of
consistent/inconsistent predictions are almost same for both BAN and LXMERT.
Our model and both baselines are consistent for $80$% of the questions (blue
part). The ratio that only our model answering correctly is $12$% for both
baselines (orange part), while the ratio reduces to $8$% (red part). This
provides further evidence that the textual representation of images can
compete with deep visual features.

###  5.2. Qualitative Analysis

We show some qualitative examples in Figure 4. We compare our model’s
predictions with the predictions of BAN and LXMERT. Example (1) asks what the
man is doing with his hand. The image description contains the expression of
what the man is doing ( “A man pointing to pots…”), result in our language-
only model answering correctly. On the other hand, BAN and LXMERT fail to make
the correct prediction, which they answer “cooking” nevertheless the man is
not cooking. The object detector can detect the cooking tools, so the models
may cause this mistake guess from the utensils, not from the man’s move. In
example (2), the image description also describes the critical information to
answer the question (“Six snowboards”), while the object detector cannot
detect all objects. On the contrary, examples (3) and (4) show the limitations
of our language-only model. Example (3) requires reading the letters on the
side of the plane. The image description contains the necessary word
(“AirFrance”) to answer the question but fails to make a correct prediction.
This result poses the lack of language-only Transformer models’ ability to
understand the contents and relations between the question and description.
Additionally, example (4) shows the importance of the image descriptions’
quality. Our model fails to correctly answer because there is no information
to answer the question. From these observations, utilizing well-described text
as image representation has the advantage against the deep visual features
when answering the questions that deep visual features do not work well. On
the other hand, we identify the limitation of our language-only model for
understanding the text input.

###  5.3. Limitations

Lastly, we analyze the limitation of our approach. Note that it is not fair to
directly compare the results by our language-only model and the deep visual
feature-based models, as our approach uses additional annotations that deep
visual features does not, i.e., annotated sentences from Localized Narratives
and COCO captions. This means that the sentences describe the content
prominent for human eyes. The questions in the dataset, also annotated by
humans, are based on the same perspective, and our model may take this
advantage. Meanwhile, deep visual features are trained in the end-to-end
manner, which can provide strong supervision on what to see. This on the other
hand benefits deep visual feature-based models. Yet, our results give
interesting insights on differences and analogies between deep visual features
and textual representation, providing a baseline for the VQA tasks with the
interpretable representation. Moreover, this is study brings us the
opportunity to introduce a new research direction for VQA in particular, and
image understanding in general: to automatically generate image description as
image representations instead of (or combined with) deep visual features.

##  6\. Conclusion

In this paper, we explored using textual representations of images instead of
deep visual features for VQA. Additionally, we explored data augmentation
methods for descriptions and questions to increase the training data size and
diversity. Through the experiments, including ablations, we validated our
language-only model competed with deep visual feature-based models. Also, most
of the data augmentation techniques enhanced the performance; especially the
boost by back translation for questions was outstanding. It was revealed that
machines do not need to take a thousand words to understand an image, but only
a hundred. In this study, we dealt with VQA to validate the effectiveness of
the textual representations, but these representations may be potentially
applied to other tasks, not only VQA. In future work, we will 1) delve into
why back translation for questions works so well; 2) delve into the textual
representations for various tasks; 3) explore the performance when using a
text generator to make the image descriptions.

## References

  * (1)
  * Agarwal et al. (2020) Vedika Agarwal, Rakshith Shetty, and Mario Fritz. 2020\.  Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing. In _CVPR_. IEEE, 9687–9695.
  * Agrawal et al. (2016) Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. 2016\.  Analyzing the Behavior of Visual Question Answering Models. In _EMNLP_. The Association for Computational Linguistics, 1955–1960.
  * Agrawal et al. (2018) Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018.  Don’t Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering. In _CVPR_. IEEE Computer Society, 4971–4980.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. In _CVPR_. IEEE Computer Society, 6077–6086.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual Question Answering. In _ICCV_. IEEE Computer Society, 2425–2433.
  * Cadène et al. (2019a) Rémi Cadène, Hedi Ben-younes, Matthieu Cord, and Nicolas Thome. 2019a.  MUREL: Multimodal Relational Reasoning for Visual Question Answering. In _CVPR_. Computer Vision Foundation / IEEE, 1989–1998.
  * Cadène et al. (2019b) Rémi Cadène, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. 2019b.  RUBi: Reducing Unimodal Biases for Visual Question Answering. In _NeurIPS_. 839–850.
  * Chen et al. (2020b) Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. 2020b.  Counterfactual Samples Synthesizing for Robust Visual Question Answering. In _CVPR_. IEEE, 10797–10806.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015.  Microsoft COCO Captions: Data Collection and Evaluation Server.  _CoRR_ abs/1504.00325 (2015).
  * Chen et al. (2020a) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020a.  UNITER: UNiversal Image-TExt Representation Learning. In _ECCV (30)_ _(Lecture Notes in Computer Science, Vol. 12375)_. Springer, 104–120.
  * Clark et al. (2019) Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019\.  Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases. In _EMNLP/IJCNLP (1)_. Association for Computational Linguistics, 4067–4080.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT (1)_. Association for Computational Linguistics, 4171–4186.
  * Ganapathibhotla and Liu (2008) Murthy Ganapathibhotla and Bing Liu. 2008.  Mining Opinions in Comparative Sentences. In _COLING_. 241–248.
  * Garcia and Nakashima (2020) Noa Garcia and Yuta Nakashima. 2020.  Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions. In _ECCV (18)_ _(Lecture Notes in Computer Science, Vol. 12363)_. Springer, 581–598.
  * Gokhale et al. (2020) Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020.  MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering. In _EMNLP (1)_. Association for Computational Linguistics, 878–892.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014\.  Generative Adversarial Nets. In _NIPS_. 2672–2680.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In _CVPR_. IEEE Computer Society, 6325–6334.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _CVPR_. IEEE Computer Society, 770–778.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews. In _KDD_. ACM, 168–177.
  * Huang et al. (2021) Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. 2021\.  Seeing Out of the Box: End-to-End Pre-Training for Vision-Language Representation Learning. In _CVPR_. 12976–12985.
  * Hudson and Manning (2019) Drew A. Hudson and Christopher D. Manning. 2019.  Learning by Abstraction: The Neural State Machine. In _NeurIPS_. 5901–5914.
  * Jiang et al. (2020) Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik G. Learned-Miller, and Xinlei Chen. 2020\.  In Defense of Grid Features for Visual Question Answering. In _CVPR_. IEEE, 10264–10273.
  * Jiang et al. (2018) Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. 2018\.  Pythia v0.1: the Winning Entry to the VQA Challenge 2018.  _CoRR_ abs/1807.09956 (2018).
  * Kim et al. (2018) Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018\.  Bilinear Attention Networks. In _NeurIPS_. 1571–1581.
  * Kumar et al. (2020) Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020\.  Data Augmentation using Pre-trained Transformer Models.  _CoRR_ abs/2003.02245 (2020).
  * Kuznetsova et al. (2020) Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. 2020.  The Open Images Dataset V4.  _Int. J. Comput. Vis._ 128, 7 (2020), 1956–1981.
  * Li et al. (2019a) Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019a.  Relation-Aware Graph Attention Network for Visual Question Answering. In _ICCV_. IEEE, 10312–10321.
  * Li et al. (2019b) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019b.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  _CoRR_ abs/1908.03557 (2019).
  * Li and Roth (2002) Xin Li and Dan Roth. 2002\.  Learning Question Classifiers. In _COLING_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context. In _ECCV (5)_ _(Lecture Notes in Computer Science, Vol. 8693)_. Springer, 740–755.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ abs/1907.11692 (2019).
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In _NeurIPS_. 13–23.
  * Ma (2019) Edward Ma. 2019\.  NLP Augmentation.  https://github.com/makcedward/nlpaug.
  * Maas et al. (2011a) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011a.  Learning Word Vectors for Sentiment Analysis. In _ACL_. The Association for Computer Linguistics, 142–150.
  * Maas et al. (2011b) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011b.  Learning Word Vectors for Sentiment Analysis. In _ACL_. The Association for Computer Linguistics, 142–150.
  * Malinowski et al. (2018) Mateusz Malinowski, Carl Doersch, Adam Santoro, and Peter W. Battaglia. 2018.  Learning Visual Question Answering by Bootstrapping Hard Attention. In _ECCV (6)_ _(Lecture Notes in Computer Science, Vol. 11210)_. Springer, 3–20.
  * Miller (1995) George A. Miller. 1995\.  WordNet: A Lexical Database for English.  _Commun. ACM_ 38, 11 (1995), 39–41.
  * Minaee et al. (2020) Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. 2020.  Deep Learning Based Text Classification: A Comprehensive Review.  _CoRR_ abs/2004.03705 (2020).
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook FAIR’s WMT19 News Translation Task Submission. In _WMT (2)_. Association for Computational Linguistics, 314–319.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019\.  fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In _NAACL-HLT (Demonstrations)_. Association for Computational Linguistics, 48–53.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts. In _ACL_. ACL, 271–278.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global Vectors for Word Representation. In _EMNLP_. ACL, 1532–1543.
  * Peters et al. (2018a) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a.  Deep Contextualized Word Representations. In _NAACL-HLT_. Association for Computational Linguistics, 2227–2237.
  * Peters et al. (2018b) Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b.  Dissecting Contextual Word Embeddings: Architecture and Representation. In _EMNLP_. Association for Computational Linguistics, 1499–1509.
  * Pont-Tuset et al. (2020) Jordi Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020\.  Connecting Vision and Language with Localized Narratives. In _ECCV (5)_ _(Lecture Notes in Computer Science, Vol. 12350)_. Springer, 647–664.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100, 000+ Questions for Machine Comprehension of Text. In _EMNLP_. The Association for Computational Linguistics, 2383–2392.
  * Ramachandran et al. (2017) Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017\.  Swish: a self-gated activation function.  _arXiv preprint arXiv:1710.05941_ 7 (2017), 1.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 91–99.  https://proceedings.neurips.cc/paper/2015/hash/14bfa6bb14875e45bba028a21ed38046-Abstract.html
  * Saeedi et al. (2020) Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis Cheuk Ming Fong. 2020.  CS-NLP Team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Task. In _SemEval@COLING_. International Committee for Computational Linguistics, 507–515.
  * Selvaraju et al. (2017) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In _ICCV_. IEEE Computer Society, 618–626.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016\.  Improving Neural Machine Translation Models with Monolingual Data. In _ACL (1)_. The Association for Computer Linguistics.
  * Shrestha et al. (2019) Robik Shrestha, Kushal Kafle, and Christopher Kanan. 2019\.  Answer Them All! Toward Universal Visual Question Answering Models. In _CVPR_. Computer Vision Foundation / IEEE, 10472–10481.
  * Socher et al. (2013) Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013.  Parsing with Compositional Vector Grammars. In _ACL (1)_. The Association for Computer Linguistics, 455–465.
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In _ICLR_. OpenReview.net.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In _EMNLP/IJCNLP (1)_. Association for Computational Linguistics, 5099–5110.
  * Teney et al. (2020) Damien Teney, Ehsan Abbasnejad, and Anton van den Hengel. 2020\.  Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision. In _ECCV (10)_ _(Lecture Notes in Computer Science, Vol. 12355)_. Springer, 580–599.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NIPS_. 5998–6008.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In _EMNLP/IJCNLP (1)_. Association for Computational Linguistics, 6381–6387.
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020.  Unsupervised Data Augmentation for Consistency Training. In _NeurIPS_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019\.  XLNet: Generalized Autoregressive Pretraining for Language Understanding. In _NeurIPS_. 5754–5764.
  * Yang et al. (2016) Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2016.  Stacked Attention Networks for Image Question Answering. In _CVPR_. IEEE Computer Society, 21–29.
  * Yao et al. (2017) Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. 2017.  Boosting Image Captioning with Attributes. In _ICCV_. IEEE Computer Society, 4904–4912.
  * You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016.  Image Captioning with Semantic Attention. In _CVPR_. IEEE Computer Society, 4651–4659.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014\.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Trans. Assoc. Comput. Linguistics_ 2 (2014), 67–78.
  * Yu et al. (2018) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018.  QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In _ICLR (Poster)_. OpenReview.net.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level Convolutional Networks for Text Classification. In _NIPS_. 649–657.
  * Zhou et al. (2019) Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2019.  Semantic Understanding of Scenes Through the ADE20K Dataset.  _Int. J. Comput. Vis._ 127, 3 (2019), 302–321.


# Investigating the Semantic Robustness of CLIP-based Zero-Shot Anomaly
Segmentation

Kevin Stangl
TTIC, USA
<EMAIL_ADDRESS>Work done while at Intel Labs, USA.
This publication includes work that was funded by the Government under the
Defense Advanced Research Projects Agency (DARPA) Guaranteeing AI Robustness
against Deception (GARD) Program, Agreement #HR00112030001. Marius Arvinte
Weilin Xu Cory Cornelius
Intel Labs, USA
<EMAIL_ADDRESS>
###### Abstract

Zero-shot anomaly segmentation using pre-trained foundation models is a
promising approach that enables effective algorithms without expensive,
domain-specific training or fine-tuning. Ensuring that these methods work
across various environmental conditions and are robust to distribution shifts
is an open problem. We investigate the performance of WinCLIP [14] zero-shot
anomaly segmentation algorithm by perturbing test data using three semantic
transformations: bounded angular rotations, bounded saturation shifts, and hue
shifts. We empirically measure a lower performance bound by aggregating across
per-sample worst-case perturbations and find that average performance drops by
up to $20\%$ in area under the ROC curve and $40\%$ in area under the per-
region overlap curve. We find that performance is consistently lowered on
three CLIP backbones, regardless of model architecture or learning objective,
demonstrating a need for careful performance evaluation.

##  1 Introduction

Visual anomaly segmentation is a challenging and important task, especially in
manufacturing line applications, where objects have to be inspected in a fast,
automated way, that is robust under a varied number of environmental
conditions [3, 32]. Anomalies manifest as fabrication defects (scratches,
chips) and localizing them using fine-grained segmentation allows for
interpretability of different types of defects [3]. Given the variety of items
that are inspected, collecting large, annotated training sets for each object
is prohibitive. Even if data would be available, training and deploying
separate models for each object is costly.

Figure 1: Effects of the three augmentations applied to the same anomalous (a
large crack in the shell of a hazelnut) MVTec sample. The third column
represents the original sample.

Recent work [14, 31] leverages the zero-shot generalization ability of
foundation models [21] to address anomaly segmentation across a varied number
of objects and environments. A modern line of work in computer vision deals
with test-time robustness of discriminative models against natural
distribution shifts [10, 7], where it is found that, generally, even large-
scale models are fragile to well-chosen, per-sample perturbations. This has
been recently demonstrated for CLIP-based models both for the case of noise-
like perturbations [23] and for natural perturbations (such as camera pose
changes) [26].

In this work, we investigate the robustness of downstream anomaly segmentation
using CLIP when faced with three perturbations (rotations, hue shift,
saturation shift) that simulate naturally occurring distribution shifts.
Importantly, we bound these shifts in a controlled way such that anomalies
remain clearly present when inspected by humans – the question we investigate
is if CLIP-based anomaly segmentation is invariant to these shifts?

Our results demonstrate that average segmentation performance is reduced
across most objects across CLIP backbones, with drops of up to $40\%$ in the
per-sample worst-case setting. This performance drop points to the need of
conducting lower bound performance evaluations of foundation models used in
zero-shot anomaly segmentation, and is in line with a recent line of work that
investigates the resilience of foundation models to natural distribution
shifts and adversarial examples [15, 24].

###  1.1 Related Work

A large number of recent works have introduced zero-shot anomaly
classification and segmentation algorithms that use foundation models [12, 9,
5, 6, 30]. Among these, we focus on WinCLIP [14] because of its good
performance, fast inference speed, and open-source implementation in the
anomalib library [1]. WinCLIP for anomaly segmentation has two components: a
template-based prompting strategy, and a multi-scale pooling strategy to
aggregate information across multiple spatial scales. Other recent methods
have used the properties of diffusion processes for anomaly detection [19],
pre-trained diffusion models [30], and vision-language models [9].

Recent work [24, 27] investigates the robustness of CLIP and how previously
reported results likely over-state the observed effective robustness due to
evaluating on a static test set from one distribution. The work in Idrissi et
al. [13] characterizes subsets of the ImageNet dataset and identifies color
shifts as a common subset and potential augmentation.

Our proposed work is similar in spirit to both previous idea, given that we
propose to evaluate anomaly segmentation on worst-case augmented samples from
the original test set, and our augmentations include color shifts [11, 4].
Another line of work [17, 26, 28] questions whether CLIP suffers from
inaccurate physical grounding, showing that semantic properties of images are
not understood by vision-language models such as LLaVA [18] or GPT-4V [20].

##  2 Methods

To investigate the performance of anomaly segmentation under worst-case
distribution shifts, we use three semantic preserving augmentations: rotation,
hue shift, and saturation shift. Their effects on the same sample are shown in
Figure 1, where it can be seen that the anomaly remains detectable across the
entire augmentation range.

The query sample $x$ is clockwise rotated using an angle $\theta$, followed by
an RGB to HSV conversion. Additive shifts $\delta_{h}$ and $\delta_{s}$ are
independently applied to the hue and saturation channels [11], respectively,
after which the sample is converted back to its RGB representation, yielding
the augmented sample $x_{\textrm{aug}}$. The pre-processing is summarized as:

$x_{\textrm{aug}}=f(x;\theta,\delta_{h},\delta_{s}).$ (1)

The modified sample $x_{\textrm{aug}}$ is input to zero-shot anomaly detection
using WinCLIP [14], yielding the estimated soft (continuous) anomaly maps
$\tilde{y}_{\textrm{aug}}$ as:

$\tilde{y}_{\textrm{aug}}=w(x_{\textrm{aug}}).$ (2)

Finally, the output sensitivity maps are additionally rotated counter-
clockwise using the angle $\theta$ (hue and saturation shifts do not spatially
change the sample) to yield:

$\tilde{y}=f(\tilde{y}_{\textrm{aug}};-\theta,0,0),$ (3)

where the choice of $\delta_{h}=\delta_{s}=0$ implies there is no HSV space
conversion performed. This allows for fair end-to-end comparison between
$\tilde{y}$ and the ground-truth segmentation maps $y$, without requiring any
intervention on $y$.

The entire pre-processing pipeline is differentiable, thus the parameters
$\theta,\delta_{h},\delta_{s}$ can be optimized using gradient-based methods.
However, doing so on a per-sample basis requires defining a differentiable
loss function that measures anomaly segmentation loss. We introduce the
following simplified version of the Dice loss [25] (using the $\ell_{1}$-norm)
from medical image segmentation:

$l(\tilde{y},y)=\frac{\sum_{i}\tilde{y}_{i}(1-y_{i})}{\sum_{i}(1-y_{i})+\epsilon}-\frac{\sum_{i}\tilde{y}_{i}y_{i}}{\sum_{i}y_{i}+\epsilon},$
(4)

where $\epsilon=1\mathrm{e}\textrm{-}8$ is a small constant added for
numerical stability.

Note that this is a valid segmentation loss: the first term encourages low
values for $\tilde{y}_{i}$, where $i$ satisfies $y_{i}=0$ (i.e., true negative
pixels). Similarly, the second term encourages higher values of
$\tilde{y}_{i}$ for the true positive pixels. Unlike the Dice loss, our
proposed loss does not contain the estimated output $\tilde{y}$ in the
denominator, which improves numerical stability when back-propagating
gradients to the inputs.

Unlike supervised learning (where the goal is loss minimization), the goal of
our work is to maximize this loss by optimizing the three parameters
$\theta,\delta_{h},\delta_{s}$ as:

$\displaystyle\theta^{*},\delta_{h}^{*},\delta_{s}^{*}\ =$
$\displaystyle\operatorname*{argmax}_{\theta,\delta_{h},\delta_{s}}\
l\left(\tilde{y}(\theta,\delta_{h},\delta_{s}),y\right),$ (5) s.t.
$\displaystyle\
\theta\in\left[-90^{\circ},90^{\circ}\right],\delta_{s}\in\left[-0.5,0.5\right].$

Figure 2: Zero-shot anomaly segmentation performance when the same rotation
angle $\theta$ is applied to the MVTec test set for rotation-invariant
objects. Figure 3: Zero-shot anomaly segmentation performance when the same
additive (modulo $2\pi$) hue shift $\delta_{h}$ is applied to the MVTec test
set for rotation-invariant objects.

To obtain the tightest lower performance bounds, this optimization is done
separately for each sample in the test set. We used the Adam optimizer [16]
with at most $200$ optimization steps per sample to approximate the solution
to (5). Details are provided in the supplementary material.

The above is an approximation of the worst-case setting encountered in
practice across an extended period of time: every individual test sample is
manipulated in its own worst way (by accident or ill-intended) with respect to
the downstream task, while still ensuring that the manipulation is realistic
and preserves the semantic information of the sample (e.g., using a bounded
saturation shift). Samples are aggregated and final performance is measured
using three metrics, following the protocol described in Jeong et al. [14]:
pixel-level area under the ROC curve (pAUROC), area under the per-region
overlap curve (AUPRO), and the $F_{1}$-max score.

##  3 Experimental Results and Discussion

Figure 4: Zero-shot anomaly segmentation performance for two datasets (MVTec and VisA, left and right, respectively) using three CLIP backbones (ViT-B/16+, ViT-L/14, and adversarially fine-tuned ViT-L/14-FARE2 for WinCLIP. The three test-time, worst-case semantic perturbations (angle, saturation, hue) are considered either separately, or simultaneously (3D). The bars show the difference between the original test sets and the considered lower bounds. Anomaly Segmentation | MVTec-AD | VisA
---|---|---
Eval. data | Backbone | pAUROC | AUPRO |  $F_{1}$-max | Loss | pAUROC | AUPRO |  $F_{1}$-max | Loss
Original | ViT-B/16+ | 80.1 | 64.2 | 24.1 | 0.075 | 76.4 | 55.4 | 10.9 | 0.165
ViT-L/14 | 59.1 | 21.9 | 9.9 | 0.108 | 59.3 | 26.3 | 2.89 | 0.199
ViT-L/14-$\text{FARE}^{2}$ | 37.7 | 10.6 | 8.3 | 0.108 | 30.6 | 4.81 | 2.10 | 0.203
Per-sample | ViT-B/16+ | 59.8 | 23.6 | 12.8 | 0.085 | 40.3 | 16.0 | 2.28 | 0.180
3D lower | ViT-L/14 | 26.4 | 4.25 | 7.68 | 0.115 | 19.9 | 3.59 | 1.37 | 0.210
bound | ViT-L/14-$\text{FARE}^{2}$ | 24.2 | 3.54 | 7.80 | 0.111 | 16.5 | 2.12 | 1.37 | 0.208
Table 1: Anomaly segmentation performance under the original test data and the
per-sample worst-case lower bound.

###  3.1 Uniform lower performance bounds

We first evaluate the performance of zero-shot anomaly segmentation when only
one of the three augmentations is applied using the same value to the entire
test set. This is a loose lower bound, but we find that even in this case,
conclusions are non-trivial.

Figure 2 shows zero-shot segmentation pAUROC for six objects in the MVTec
dataset when the same rotation angle $\theta$ is applied to all test samples
using the protocol in Section 2. Only objects that are placed on matte
backgrounds were chosen for evaluation, to ensure that rotation does not
produce artificial anomalies in the corners of $x_{\textrm{aug}}$. Figure 2
reveals that rotating samples has a non-trivial and non-smooth effect on
downstream task performance: small rotations at arbitrary angles can cause a
performance difference of up to $3\%$ in pAUROC to appear in most objects.
Performance can vary by up to $10\%$ between the lower and upper bounds of a
certain object (e.g., for capsule). Similar results for the VisA dataset are
in the supplementary material.

Figure 3 shows zero-shot segmentation pAUROC the same six objects in Figure 2
when the same hue shift $\delta_{h}$ is applied to the test samples. In this
case, variations are smoother, but effects are still non-trivial: certain
objects are less sensitive (e.g., all screw images are gray-scale, and
completely unaffected by hue), while others show a significant performance
variation (e.g., pill and metal nut) of up to $15\%$. Given that hue shifts
should not affect physical defects in objects, this indicates that the
backbone used here (ViT-B/16+) is not sufficiently robust to these shifts,
even when the shifts are not sample-specific.

###  3.2 Per-sample lower performance bounds

We report per-sample lower performance bounds where the worst-case
transformation is independently obtained for each test sample, followed by
aggregating all augmented samples and reporting performance.

As expected from Figure 2, the distribution of worst-case angles across
individual samples is approximately uniform: given that only rotation-
invariant objects were evaluated, there is no reason for specific angles to be
systematically worst-case for all samples. The distribution is shown in the
supplementary material.

Figure 4 shows the distribution of performance drops across objects for three
CLIP backbones tested on both the augmented MVTec and VisA datasets. Four per-
sample worst-case augmentations were used: three one-dimensional ones – in
angle $\theta$, hue shift $\delta_{h}$, saturation shift $\delta_{s}$ – and
the worst-case lower performance bound obtained by jointly optimizing all
three variables for each sample and aggregating results. In general, we notice
that the lower bound is looser (performance suffers a larger drop) for the
VisA dataset compared to MVTec in the case of non-adversarially fine-tuned
backbones. When considering that VisA is a more difficult test set that MVTec
(based on the lower original performance in Table 1), the even higher worst-
case performance drop indicates that a significant robustness gap exists in
harder segmentation problems.

In terms of specific one-dimensional perturbations, hue shifts affect VisA the
most on average (across all backbones), while the results for MVTec are less
conclusive, and more backbone-dependent. The adversarially fine-tuned FARE2
backbone [23] consistently shows lower performance drops, but the nominal
performance also suffers a significant drop, indicating a sharp trade-off
between robustness to distribution shifts and absolute performance.

Table 1 shows results for anomaly segmentation metrics, evaluated on both
datasets, using three models. We compare the original (un-augmented) test set
performance with that of the worst-case per-sample lower bound. Given the
imbalanced nature of anomaly segmentation (fewer positive pixels than
negatives), the AUPRO scores suffers a performance of up to $40\%$ using the
ViT-B/16+ architecture. Similarly, we reproduce the already-low F1-max score
on the original test sets, and this value drop to as low as $2\%$ for the VisA
dataset using the ViT-B/16+ architecture. Table 1 shows the numerical values
of the optimization objective defined in (4) used to find worst-case
augmentations: it can be seen that the values are inversely correlated with
the performance metrics, and that (4) is a valid segmentation loss.

##  4 Conclusion

Our work demonstrates the generalization challenges of zero-shot anomaly
segmentation algorithms when faced with relatively simple test-time
augmentations (rotation, hue, saturation). By formulating an optimization
problem that determines the per-sample worst-case augmentations, we show
performance drops for CLIP-based anomaly segmentation of up to $40\%$ in
pAUROC compared to the original test set.

Standard practice for CLIP pre-trained models is to not use heavy augmentation
pipelines [21] – some samples have text label information about their color,
which would make naive color augmentations not compatible with the learning
objective. Future research could consider how to generalize semantic
augmentations to multi-modal data and include this in the training of
foundation models. Finally, all our work targeted zero-shot anomaly
segmentation. Few-shot anomaly segmentation is common in practice and
significant improvements could be achieved if test-time augmentations were
optimized to maximize performance for each object type instead of minimizing
it, as we did in this work.

## References

  * Akcay et al. [2022] Samet Akcay, Dick Ameln, Ashwin Vaidya, Barath Lakshmanan, Nilesh Ahuja, and Utku Genc.  Anomalib: A deep learning library for anomaly detection.  In _2022 IEEE International Conference on Image Processing (ICIP)_ , pages 1706–1710. IEEE, 2022.
  * Bengio et al. [2013] Yoshua Bengio, Nicholas Léonard, and Aaron Courville.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_ , 2013.
  * Bergmann et al. [2019] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9592–9600, 2019.
  * Bhattad et al. [2020] Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and D. A. Forsyth.  Unrestricted adversarial examples via semantic manipulation, 2020.
  * Cao et al. [2023a] Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, and Weiming Shen.  Segment any anomaly without training via hybrid prompt regularization.  _arXiv preprint arXiv:2305.10724_ , 2023a.
  * Cao et al. [2023b] Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen.  Towards generic anomaly detection and understanding: Large-scale visual-linguistic model (gpt-4v) takes the lead.  _arXiv preprint arXiv:2311.02782_ , 2023b.
  * Chen et al. [2023] Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu.  Advdiffuser: Natural adversarial example synthesis with diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4562–4572, 2023.
  * Eboli [Accessed on April 24, 2024] Thomas Eboli.  High-quality pytorch image and volume rotation.  Accessed on April 24, 2024.
  * Gu et al. [2024] Zhaopeng Gu, Bingke Zhu, Guibo Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang.  Anomalygpt: Detecting industrial anomalies using large vision-language models.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1932–1940, 2024.
  * Hendrycks et al. [2021] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 15262–15271, 2021.
  * Hosseini and Poovendran [2018] Hossein Hosseini and Radha Poovendran.  Semantic adversarial examples.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1614–1619, 2018.
  * Huang et al. [2022] Chaoqin Huang, Haoyan Guan, Aofan Jiang, Ya Zhang, Michael Spratling, and Yan-Feng Wang.  Registration based few-shot anomaly detection, 2022.
  * Idrissi et al. [2022] Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim.  Imagenet-x: Understanding model mistakes with factor of variation annotations.  _arXiv preprint arXiv:2211.01866_ , 2022.
  * Jeong et al. [2023] Jongheon Jeong, Yang Zou, Taewan Kim, Dongqing Zhang, Avinash Ravichandran, and Onkar Dabeer.  Winclip: Zero-/few-shot anomaly classification and segmentation, 2023.
  * Kandpal et al. [2023] Nikhil Kandpal, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini.  Backdoor attacks for in-context learning with language models, 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Lewis et al. [2023] Martha Lewis, Nihal V. Nayak, Peilin Yu, Qinan Yu, Jack Merullo, Stephen H. Bach, and Ellie Pavlick.  Does clip bind concepts? probing compositionality in large image models, 2023.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Livernoche et al. [2023] Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh.  On diffusion modeling for anomaly detection.  _arXiv preprint arXiv:2305.18593_ , 2023.
  * OpenAI [2024] OpenAI.  Gpt-4 technical report, 2024.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Riba et al. [2020] E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski.  Kornia: an open source differentiable computer vision library for pytorch.  In _Winter Conference on Applications of Computer Vision_ , 2020.
  * Schlarmann et al. [2024] Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein.  Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models, 2024.
  * Shi et al. [2023] Zhouxing Shi, Nicholas Carlini, Ananth Balashankar, Ludwig Schmidt, Cho-Jui Hsieh, Alex Beutel, and Yao Qin.  Effective robustness against natural distribution shifts for models with different training data.  _arXiv preprint arXiv:2302.01381_ , 2023.
  * Sudre et al. [2017] Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M. Jorge Cardoso.  _Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations_ , page 240–248.  Springer International Publishing, 2017.
  * Tong et al. [2024] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie.  Eyes wide shut? exploring the visual shortcomings of multimodal llms.  _arXiv preprint arXiv:2401.06209_ , 2024.
  * Tu et al. [2024] Weijie Tu, Weijian Deng, and Tom Gedeon.  A closer look at the robustness of contrastive language-image pre-training (clip).  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Udandarao et al. [2023] Vishaal Udandarao, Max F. Burg, Samuel Albanie, and Matthias Bethge.  Visual data-type understanding does not emerge from scaling vision-language models, 2023.
  * Unser et al. [1995] Michael Unser, Philippe Thevenaz, and Leonid Yaroslavsky.  Convolution-based interpolation for fast, high-quality rotation of images.  _IEEE Transactions on image processing_ , 4(10):1371–1381, 1995.
  * Wyatt et al. [2022] Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks.  Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 650–656, 2022.
  * Zhou et al. [2023] Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen.  Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection.  _arXiv preprint arXiv:2310.18961_ , 2023.
  * Zou et al. [2022] Yang Zou, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer.  Spot-the-difference self-supervised pre-training for anomaly detection and segmentation, 2022.

Supplementary Material

##  A Implementation Details

###  A.1 Augmentations

To implement the rotation augmentation, we used the three-pass convolutional
approach from Unser et al. [29]. Compared to bilinear interpolation, this
method preserves the high-frequency content of the samples and ensures that
minimal distortions (at least when evaluated by humans) are introduced via the
rotation. To extrapolate pixels in the corners of the rotated sample, we used
the replicate padding mode: for each extrapolated pixel, the value of the
nearest pixel from the rotated sample itself is copied. We used the torch-
rotation library [8] to implement the differentiable three-pass rotation.

Hue and saturation shifts were implemented using the RGB-to-HSV conversion
function from the kornia library [22], with the shifts themselves being
otherwise straightforward to implement.

Given that saturation is clipped to $[0,1]$ after the shift is applied, we
used a straight-through estimator (STE) [2] to back-propagate through this
operation even outside its valid range. Similarly, we used the STE estimator
to clip and back-propagate the values of the rotation angle $\theta$ and
saturation shift $\delta_{s}$ whenever they exceed their permitted ranges.

###  A.2 Solving the optimization problem

We used the Adam optimizer [16] with learning rates of $5$ for $\theta$ and
$0.1$ for $\delta_{h}$ and $\delta_{s}$, respectively. The learning rates were
chosen based on the $\ell_{\infty}$-norm of the gradient with respect to each
variable at initialization, and were not extensively fine-tuned or searched
for.

We found that using random restarts and for the optimization problem is
crucial for successfully decreasing the per-sample lower performance bound. In
particular, the loss surface is highly non-convex for the angle $\theta$, and
optimization often gets trapped in local minima. Given the relatively low
budget of $200$ steps, we used $5$ restarts: the first optimization run starts
with the default values $\theta=\delta_{h}=\delta_{s}=0$, while the subsequent
four runs restart these values using uniform sampling from the valid interval
of each variable.

Additionally, we also monitor the per-sample segmentation pAUROC for samples
that have at least one anomalous labeled pixel in $y$ and select the specific
perturbations $(\theta^{*},\delta_{h}^{*},\delta_{s}^{*})$ that minimize this
value. Note that this is still sub-optimal when it comes to minimizing the
pAUROC on pixels from all samples of a specific object, which is generally
intractable. Finally, for samples that have $y_{i}=0$ for all $i$ (no
anomalous pixels), we monitor the loss function $L$ in (4) and select the
specific perturbations $(\theta^{*},\delta_{h}^{*},\delta_{s}^{*})$ that
maximize this value.

###  A.3 Selecting rotation-invariant objects

To select rotation-invariant objects, we used the per-pixel average mean
squared error between the extrapolated corner pixels using our rotation method
and the original corner pixels as:

$e(x)=\frac{\sum_{i}m_{i}\cdot(x_{i}-f(x_{i};\theta=45^{\circ}))^{2}}{\sum_{i}m_{i}},$
(6)

where $m_{i}$ is a binary mask that indicates whether a specific pixel was
extrapolated or not and is obtained by rotating a monochrome image with
maximum pixel intensity. We evaluate this error using the specific angle
$\theta=45^{\circ}$, with the reasoning that it is where the most
extrapolation occurs, and rotation-invariant objects should not be cut-off by
this rotation. If a cut-off would occur, the object texture would be reflected
in the extrapolated corner, thus producing a high error compared to the
original corner region.

The above is followed by a manual thresholding of the error, as well as final
careful visual inspection of the down-selected classes. Tables 2 and 3 show
the selection results for objects in the MVTec and VisA datasets,
respectively.

Object name |  Corner MSE |  Used
---|---|---
Metal nut |  0.000255 |  True
Pill |  0.000304 |  True
Hazelnut |  0.000391 |  True
Bottle |  0.002179 |  True
Screw |  0.005964 |  True
Leather |  0.013949 |  False
Wood |  0.021015 |  False
Capsule |  0.02621 |  True
Cable |  0.027923 |  False
Transistor |  0.077574 |  False
Tile |  0.09312 |  False
Carpet |  0.104486 |  False
Grid |  0.126188 |  False
Toothbrush |  0.17937 |  False
Zipper |  0.92926 |  False
Table 2: Average corner extrapolation MSE for objects in the MVTec dataset. Leather and wood were excluded from evaluation after manual inspection, given that the textures span the entire sample and simple extrapolation would lead to qualitative artifacts in the samples that could be mistaken for anomalies. Object name |  Corner MSE |  Used
---|---|---
Pipe fryum |  0.000668 |  True
Pcb3 |  0.001449 |  True
Macaroni1 |  0.004837 |  True
Pcb4 |  0.005264 |  True
Fryum |  0.00995 |  True
Pcb2 |  0.010521 |  True
Pcb1 |  0.013953 |  True
Chewing gum |  0.024117 |  True
Macaroni2 |  0.035739 |  True
Cashew |  0.049281 |  True
Candle |  0.051116 |  False
Capsules |  0.082838 |  False
Table 3: Average corner extrapolation MSE for objects in the VisA dataset.

###  A.4 Optimization outcomes

Figure 5: Empirical distribution of per-sample worst-case rotation angles for
all rotationally invariant object in MVTec. The slight peak around the origin
is caused by the optimization being sub-optimal due to the de-rotation of
$\tilde{y}_{\textrm{aug}}$ using zero value extrapolation in the corners.

Figure 5 shows the histogram of per-sample worst-case rotation angles
$\theta^{*}$ when optimizing only for rotation. The higher mode around the
origin is due to the intentional artifacts introduced when de-rotating the
estimated anomaly map $\tilde{y}_{\textrm{aug}}$: for difficult samples, this
may cause a slight performance boost at any rotation angle. This causes the
optimization in (5) to return a near-zero angle as the worst-case one.

##  B Additional uniform lower bound results

Figure 6 shows the results for applying the same saturation shift $\delta_{s}$
to all samples in the MVTec test set. Similar to the result for the hue shift
$\delta_{h}$ in Figure 3, there are significant performance variations for
most objects except metal nut, which already contains gray-scale samples that
are less sensitive to color shifts.

Figure 6: Zero-shot anomaly segmentation performance when the same saturation
shift $\delta_{s}$ is applied to the MVTec test set for rotation-invariant
objects.

Figures 7, 8, and 9 show the uniform lower bounds obtained from applying the
same rotation, hue shift, or saturation shift to all test samples in the VisA
dataset, respectively. While the same conclusions related to how smooth
performance varies with each augmentation type (noisy for rotations, smooth
for color shifts), the dynamic ranges of the per-object performance are much
larger for VisA, which is generally a more difficult dataset. For certain
objects, such as Pcb1 or Macaroni1, performance can vary by up to $0.4$ points
in pAUROC when rotations or hue shifts are applied, respectively.

Figure 7: Zero-shot anomaly segmentation performance when the same rotation
angle $\theta$ is applied to the VisA test set for rotation-invariant objects.
Figure 8: Zero-shot anomaly segmentation performance when the same hue shift
$\delta_{h}$ is applied to the VisA test set for rotation-invariant objects.
Figure 9: Zero-shot anomaly segmentation performance when the same saturation
shift $\delta_{s}$ is applied to the VisA test set for rotation-invariant
objects.

##  C Additional worst-case lower bound results

Figure 10 shows the results for worst-case lower bounds obtained by jointly
optimizing any combination of considered augmentations. In general, we note
that augmentations that only involve colours (hue and saturation) are more
effective (leads to a higher performance drop) than the ones involving the
rotation augmentation. We attribute this to the non-smooth optimization
surface for the rotation angle $\theta$, which holds even under our considered
segmentation loss.

Figure 10: Zero-shot anomaly segmentation performance for two datasets (MVTec
and VisA, left and right, respectively) using three CLIP backbones (ViT-B/16+,
ViT-L/14, and adversarially fine-tuned ViT-L/14-FARE2 for WinCLIP. Each
possible combination of the three test-time, worst-case semantic perturbations
(angle, saturation, hue) is considered, to yield a set of eight total test-
time augmentation strategies. The bars show the difference between the
original test sets and the considered lower bounds. This figure complements
Figure 4 in the main body.