# Spawrious: A Benchmark for Fine Control
of Spurious Correlation Biases

Aengus Lynch , Gbètondji J-S Dovonon11footnotemark: 1 Jean
Kaddour11footnotemark: 1 , Ricardo Silva
University College London
https://github.com/aengusl/spawrious Equal contribution, alphabetical order.

###### Abstract

The problem of spurious correlations (SCs) arises when a classifier relies on
non-predictive features that happen to be correlated with the labels in the
training data. For example, a classifier may misclassify dog breeds based on
the background of dog images. This happens when the backgrounds are correlated
with other breeds in the training data, leading to misclassifications during
test time. Previous SC benchmark datasets suffer from varying issues, e.g.,
over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many
(M2M) SCs arising between groups of spurious attributes and classes. In this
paper, we present Spawrious-{O2O, M2M}-{Easy, Medium, Hard}, an image
classification benchmark suite containing spurious correlations between
classes and backgrounds. To create this dataset, we employ a text-to-image
model to generate photo-realistic images and an image captioning model to
filter out unsuitable ones. The resulting dataset is of high quality and
contains approximately 152k images. Our experimental results demonstrate that
state-of-the-art group robustness methods struggle with Spawrious, most
notably on the Hard-splits with none of them getting over $70\%$ accuracy on
the hardest split using a ResNet50 pretrained on ImageNet. By examining model
misclassifications, we detect reliances on spurious backgrounds, demonstrating
that our dataset provides a significant challenge.

##  1 Introduction

One of the reasons we have not deployed self-driving cars and autonomous
kitchen robots everywhere is their catastrophic behavior in out-of-
distribution (OOD) settings that differ from the training distribution [13,
17]. To make models more robust to unseen test distributions, mitigating a
classifier’s reliance on spurious, non-causal features that are not essential
to the true label has attracted lots of research interest [61, 1, 33, 27]. For
example, classifiers trained on ImageNet [10] have been shown to rely on
backgrounds [73, 63, 53], which are spuriously correlated with class labels
but, by definition, not predictive of them.

Recent work has focused substantially on developing new methods for addressing
the spurious correlations (SCs) problem [33], yet, studying and addressing the
limitations of existing benchmarks remains underexplored. For example, the
_Waterbirds_ [61], and _CelebA hair color_ [45] benchmarks remain among the
most used benchmarks for the SC problem; yet, GroupDRO [61] achieves 90.5%
worst-group accuracy using group adjusted data with a ResNet50 pretrained on
ImageNet.

(a) One-To-One (O2O)

(b) Many-To-Many (M2M)

Figure 1: Spawrious Challenges: Letters on the images denote the background,
and the bottom bar in Figure 1(a) indicates each class’s proportion of the
spurious background. In the O2O challenge, each class associates with a
background during training, while the test data contains unseen combinations
of class-background pairs. In the M2M challenge, a group of classes correlates
with a group of backgrounds during training, but this correlation is reversed
in the test data.

Another limitation of existing benchmarks is their sole focus on overly
simplistic one-to-one (O2O) spurious correlations, where one spurious
attribute correlates with one label. However, in reality, we often face _many-
to-many_ (M2M) spurious correlations across groups of classes and backgrounds,
which we formally introduce in this work. Imagine that during summer, we
collect training data of two groups of two animal species (classes) from two
groups of locations, e.g., a tundra and a forest in eastern Russia and a lake
and mountain in western Russia. Each animal group correlates with a background
group. In the upcoming winter, while looking for food, each group migrates,
one going east and one going west, such that the animal groups have now
exchanged locations. The spurious correlation has now been reversed in a way
that cannot be matched from one class to one location.

While some benchmarks include multiple training environments with varying
correlations [36], they do not test classification performance on reversed
correlations during test time. Such M2M-SCs are _not_ an aggregation of
O2O-SCs and cannot be expressed or decomposed in the form of the latter; they
contain qualitatively different spurious structures, as shown in Figure 2. To
the best of our knowledge, this work is the first to conceptualize and
instantiate M2M-SCs in image classification problems.

#### Contributions

We introduce _Spawrious_ -{O2O, M2M}-{Easy, Medium, Hard}, a suite of image
classification datasets with O2O and M2M spurious correlations and three
difficulty levels each. Recent work [71, 47, 68] has demonstrated a proof-of-
concept to effectively discover spurious correlation failure cases in
classifiers by leveraging off-the-shelf, large-scale, image-to-text models
trained on vast amounts of data. Here, we take this view to the extreme and
generate a novel benchmark with $152,064$ images of resolution $224\times
224$, specifically targeted at the probing of classifiers’ reliance on
spurious correlations.

Our experimental results demonstrate that state-of-the-art methods struggle
with Spawrious, most notably on the _Hard_ -splits with $<70\%$ accuracy using
ResNet50 pretrained on ImageNet. We probe a model’s misclassifications and
find further evidence for its reliance on spurious features. We also
experiment with different model architectures, finding that while larger
architectures can sometimes improve performance, the gains are inconsistent
across methods, further raising the need for driving future research.

(a) O2O: training data

(b) M2M: training data

(c) O2O: test data

(d) M2M: test data

Figure 2: Data distributions for our challenges: ${\bm{x}}_{i}$ is a random
image sampled, each $s_{i}$ is a spurious attribute, and each $c_{i}$ is a
class label. The edges indicate the probability that the sample ${\bm{x}}_{i}$
has a given property, conditional on previous steps in the tree. The leaf
nodes indicate the possible attribute-class combinations in the distribution.
The colors emphasize the distribution shift in the test data.

##  2 Existing Benchmarks

We summarize the differences between Spawrious and related benchmarks in Table
1. DomainBed [20] is a benchmark suite consisting of seven previously
published datasets focused on domain generalization (DG), not on spurious
correlations (excluding CMNIST, which we discuss separately). After careful
hyper-parameter tuning, the authors find that ERM, not specifically designed
for DG settings, as well as DG-specific methods, perform all about the same on
average. They conjecture that these datasets may comprise an ill-posed
challenge. For example, they raise the question of whether DG from a photo-
realistic training environment to a cartoon test environment is even possible.
In contrast, we follow the same rigorous hyper-parameter tuning procedure by
[20] and observe stark differences among methods on Spawrious in Section 5.1,
with ERM being the worst and $7.30\%$ points worse than the best method on
average.

Dataset | DG | O2O-SC | M2M-SC
---|---|---|---
CelebA-Hair Color [45] | ✗ | ✓ | ✗
Waterbirds [61] | ✗ | ✓ | ✗
CMNIST [1] | ✓ | ✓ | ✗
DomainBed∗ [20] | ✓ | ✗ | ✗
WILDS [36] | ✓ | ✗ | ✗
NICO [78] | ✓ | ✗ | ✗
MetaShift [43] | ✓ | ✗ | ✗
Spawrious | ✓ | ✓ | ✓

Table 1: Differences between Spawrious and other benchmarks, according to
whether they pose a Domain Generalization (DG), One-To-One- and/or Many-To-
Many Spurious Correlations challenge.

WILDS [36], NICO [78] and MetaShift [43] collect in-the-wild data and group
data points with environment labels. However, these benchmarks do not induce
_explicit_ spurious correlations between environments and labels. For example,
WILDS-FMOW [36, 8] possesses a label shift between non-African and African
regions; yet, the test images pose a domain generalization (DG) challenge
(test images were taken several years later than training images) instead of
reverting the spurious correlations observed in the training data. Waterbirds
[61], and CelebA hair color [45, 27] are binary classification datsets
including spurious correlations but without unseen test domains (DG). Further,
Idrissi et al. [26] illustrates that a simple class-balancing strategy
alleviates most of their difficulty, while Spawrious is class-balanced from
the beginning. ColorMNIST [1] includes spurious correlations and poses a DG
problem. However, it is based on MNIST and, therefore, over-simplistic, i.e.,
it does not reflect real-world spurious correlations involving complex
background features, such as the ones found in ImageNet [63, 53].

None of the above benchmarks include explicit training and test environments
for M2M-SCs.

##  3 Benchmark Desiderata

Motivated by the shortcomings of previous benchmarks discussed in Section 2,
we want first to posit some general desiderata that an improved benchmark
dataset would satisfy. Next, we motivate and formalize the two types of
spurious correlations we aim to study.

###  3.1 Six Desiderata

1\. Photo-realism unlike datasets containing cartoon/sketch images [20] or
image corruptions [22], which are known to conflict with current backbone
network architectures [15, 16, 23], possibly confounding the evaluation of OOD
algorithms. 2\. Non-binary classification problem, to minimize accidentally
correct classifications achieved by chance. 3\. Inter-class homogeneity and
intra-class heterogeneity, i.e., low variability _between_ and high
variability _within_ classes, to minimize the margins of the decision
boundaries inside the data manifold [51]. This desideratum ensures that the
classification problem is non-trivial. 4\. High-fidelity backgrounds including
complex features to reflect realistic conditions typically faced in the wild
instead of monotone or entirely removed backgrounds [73]. 5\. Access to
multiple training environments, i.e., the conditions of the _Domain
Generalization_ problem [20], which allow us to learn domain invariances, such
that classifiers can perform well in novel test domains. 6\. Multiple
difficulty levels, so future work can study cost trade-offs. For example, we
may be willing to budget higher computational costs for methods succeeding on
difficult datasets than those that succeed only on easy ones.

###  3.2 Spurious Correlations (One-To-One)

Here, we provide some intuition and discuss the conditions for a (one-to-one)
spurious correlation (SC). We define a correlated, non-causal feature as a
feature that frequently occurs with a class but does not cause the appearance
of the class (nor vice versa). We abuse the term “correlated” as it is
commonly used by previous work, but we consider non-linear relationships
between two random variables too. Further, we call correlated features
_spurious_ if the classifier perceives them as a feature of the correlated
class.

Next, we want to define a _challenge_ that allows us to evaluate a
classifier’s harmful reliance on spurious features. Spurious features are not
always harmful; even humans use context information to make decisions [17].
However, a spurious feature becomes harmful if it alone is sufficient to
trigger the prediction of a particular class without the class object being
present in the image [53].

To evaluate a classifier w.r.t. such harmful predictions, we evaluate its
performance when the spurious correlations are reverted. The simplest setting
is when a positive/negative correlation exists between one background variable
and one label in the training/test environment. We formalize this setup as
follows.

ptO2O-SC Challenge Let $p({\mathbf{X}},S,C)$ be a distribution over images
${\mathbf{X}}\in\mathbb{R}^{D}$, spurious attributes
$S\in\mathcal{S}=\\{s_{1},\dots,s_{K}\\}$, and labels
$C\in\mathcal{C}=\\{c_{1},\dots,c_{P}\\}$. Given $p_{\rm{train}}\neq
p_{\rm{test}}$, and $K=P$ it holds that for $i\in[K]$,
$\displaystyle\text{corr}_{p_{\rm{train}}}\left(\mathbbm{1}(S=s_{i}),\mathbbm{1}(C=c_{i})\right)>0,\;\text{corr}_{p_{\rm{test}}}\left(\mathbbm{1}(S=s_{i}),\mathbbm{1}(C=c_{i})\right)<0.$
(1) where the indicator function $\mathbbm{1}(X=x)$ is non-zero when the
_variable_ $X$ equals the _value_ $x$.

By one-to-one (O2O) SC, we refer to a setting in which pair-wise SCs between
spurious features $S$ (also called _style_ variable [33]) and labels $C$ exist
within training environments, which then differ in the test environment, as
illustrated in Figure 1(a).

###  3.3 Many-To-Many Spurious Correlations

In this subsection, we conceptualize Many-To-Many (M2M) SCs, where the SCs
hold over disjoint groups of spurious attributes and classes. For instance, in
Figure 1(b), each class from the class group $\\{$_Bulldog_ , _Dachshund_
$\\}$ is observed with each background from the group $\\{$_Desert_ , _Jungle_
$\\}$ in equal proportion in the training data.

ptM2M-SC Challenge Consider $p({\mathbf{X}},S,C)$ defined in the O2O-SC
Challenge. We further assume the existence of partitions
$\mathcal{S}=\mathcal{S}_{1}\dot{\cup}\mathcal{S}_{2}$ and
$\mathcal{C}=\mathcal{C}_{1}\dot{\cup}\mathcal{C}_{2}$. Given
$p_{\rm{train}},p_{\rm{test}}$, it holds that for $j\in\\{1,2\\}$
$\displaystyle\text{corr}_{p_{\rm{train}}}\left(\mathbbm{1}(S\in\mathcal{S}_{j}),\mathbbm{1}(C\in\mathcal{C}_{j})\right)=1,\text{corr}_{p_{\rm{test}}}\left(\mathbbm{1}(S\in\mathcal{S}_{j}),\mathbbm{1}(C\in\mathcal{C}_{j})\right)=-1.$
(2)

Figure 2 shows an example of how to construct M2M-SCs, which contain richer
spurious structures, following an _hierarchy_ of the class groups correlating
with spurious attribute groups. As we will see later in Section 4.2, the data-
generating processes we instantiate for each challenge differ qualitatively.

##  4 The Spawrious Challenge

In this section, we instantiate the desiderata introduced in Section 3 by
presenting _Spawrious_ , a synthetic image classification dataset containing
images of four dog breeds (classes) in six background locations (spurious
attributes).

Figure 3: Spawrious Pipeline: We leverage text-to-image models for data
generation and image-to-text models for cleaning.

###  4.1 Dataset Construction

Figure 3 summarizes the dataset construction pipeline, which we now discuss in
more detail. The main idea is to leverage recently proposed text-to-image
models [59] for photo-realistic image generation and image-to-text models [55]
for filtering out low-quality images.

A prompt template allows us to define high-level factors of variation, akin to
a causal model [33] of the data-generating process. We then sample prompts by
filling in randomly sampled values for these high-level factors. The text-to-
image model generates images given a sampled prompt; we use _Stable Diffusion
v1.4_ [59]. We pass the raw, generated images to an image-to-text (I2T) model
to extract a concise description; here, we use the ViT-GPT2 image captioning
model [55]. We perform a form of substring matching by checking whether
important keywords are present in the caption, e.g., _“dog”_. This step avoids
including images without class objects, which we sometimes observed due to the
T2T model ignoring parts of the input prompt. We keep only “clean” images
whose captions include important keywords.

###  4.2 Satisfying Desiderata

To ensure photorealism, we generate images using _Stable Diffusion v1.4_ [59],
trained on a large-scale real-world image dataset [62], while carefully
filtering out images without detectable class objects. We construct a 4-way
classification problem to reduce the probability of accidentally-correct
classifications compared to a binary classification problem (e.g., CelebA hair
color prediction or Waterbirds). Next, we chose dog breeds to reduce inter-
class variance, inspired by the difference in classification difficulty
between Imagenette (easily classified objects) [24], and ImageWoof [25] (dog
breeds), two datasets based on subsets of ImageNet [10]. We increase intra-
class variance by adding animal poses to the prompt template.

We add “ _[location] [time of day]_ ” variables to the prompt template to
ensure diverse backgrounds, and select six combinations after careful
experimentation with dozens of possible combinations, abandoning over-
simplistic ones. Our final prompt template takes the form “ _one [fur]
[animal] [pose] [location], [time of day]. highly detailed, with cinematic
lighting, 4k resolution, beautiful composition, hyperrealistic, trending,
cinematic, masterpiece, close up_ ”, and there are 72 possible combinations.
The variables [location]/[animal] correspond to spurious backgrounds/labels
for a specific background-class combination. The other variables take the
following values: _fur:_ black, brown, white, [empty]; _pose:_ sitting,
running, [empty]; _time of day:_ pale sunrise, sunset, rainy day, foggy day,
bright sunny day, bright sunny day

To construct multiple training environments, we randomly sample from a set of
background-class combinations, which we further group by their difficulty
level into _easy_ , _medium_ , and _hard_. We construct two datasets for each
SC type with $3168$ images per background-class combination, thus $2\;\text{SC
types}\;\times 4\;\text{environments}\;\times 6\;\text{difficulties}\;\times
3168=152,064$ images in total.

#### O2O-SC Challenge

We select combinations such that each class is observed with two backgrounds,
spurious $b^{\text{sp}}$ and generic $b^{\text{ge}}$. For all images with
class label $c_{i}$ in the training data, $\mu$% of them have the spurious
background $b^{\text{sp}}_{i}$ and $(100-\mu)$% of them have the generic
background $b^{\text{ge}}$. Importantly, each spurious background is observed
with only one class ($p_{\rm{train}}(b^{\text{sp}}_{i}\mid c_{j})=1$ if $i=j$
and $0$ for $i\neq j$), while the generic background is observed for all
classes with equal proportion. We train on two separate environments (with
distinct data) that differ in their $\mu$ values. Thus, the change in this
proportion should serve as a signal to a robustness-motivated optimization
algorithm (e.g. IRM [1], GroupDRO [61] etc.) that the correlation is spurious.

For instance, in Figure 1(a), training environment 1, 97% of the _Bulldog_
images have spurious _Jungle_ backgrounds, while 3% have generic _Beach_
backgrounds. The spurious background changes depending on the class, but the
relative proportions between each trio $c_{i},b_{i}^{\text{sp}}$ and
$b_{i}^{\text{ge}}$ are the same. Notably, in training environment 2, the
proportions change to 87% and 13% split of spurious and generic backgrounds.

Class | Train Env 1 | Train Env 2 | Test | Train Env 1 | Train Env 2 | Test | Train Env 1 | Train Env 2 | Test
---|---|---|---|---|---|---|---|---|---
| O2O-Easy | O2O-Medium | O2O-Hard
Bulldog | 97% De 3% B | 87% De 13% B | 100% Di | 97% M 3% De | 87% M 13% De | 100% J | 97% J 3% B | 87% J 13% B | 100% M
Dachshund | 97% J 3% B | 87% J 13% B | 100% S | 97% B 3% De | 87% B 13% De | 100% Di | 97% M 3% B | 87% M 13% B | 100% S
Labrador | 97% Di 3% B | 87% Di 13% B | 100% De | 97% Di 3% De | 87% Di 13% De | 100% B | 97% S 3% B | 87% S 13% B | 100% De
Corgi | 97% S 3% B | 87% S 13% B | 100% J | 97% J 3% De | 87% J 13% De | 100% S | 97% De 3% B | 87% De 13% B | 100% J
| M2M-Easy | M2M-Medium | M2M-Hard
Bulldog | 100% Di | 100% J | 50% S 50% B | 100% De | 100% M | 50% Di 50% J | 100% B | 100% S | 50% De 50% M
Dachshund | 100% J | 100% Di | 50% S 50% B | 100% M | 100% De | 50% Di 50% J | 100% B | 100% S | 50% De 50% M
Labrador | 100% S | 100% B | 50% Di 50% J | 100% Di | 100% J | 50% De 50% M | 100% M | 100% De | 50% B 50% S
Corgi | 100% B | 100% S | 50% Di 50% J | 100% J | 100% Di | 50% De 50% M | 100% M | 100% De | 50% B 50% S

Table 2: Proportions of Spurious Backgrounds By Class and Environment.
Backgrounds include: Beach (B), Desert (De), Dirt (Di), Jungle (J), Mountain
(M), Snow (S).

#### M2M-SC Challenge

First, we construct disjoint background and class groups
${\mathcal{S}}_{1},{\mathcal{S}}_{2},{\mathcal{C}}_{1},{\mathcal{C}}_{2}$,
each with two elements. Then, we select background-class combinations for the
training data such that for each class $c\in{\mathcal{C}}_{i}$, we pick a
combination $(s,b)$ for each $s\in{\mathcal{S}}_{i}$. Second, we introduce two
environments as shown in Figure 1(b).

The difference between _Easy_ , _Medium_ , and _Hard_ is in the splits between
the available combinations of backgrounds and classes. We determined the final
splits based on the empirical performance of a standard ERM classifier Section
5). There are no theoretical reasons we present to explain the difference in
difficulty in the challenges.

#### Data combination selection

Each spurious correlation (O2O, M2M) comes with three data combinations (Easy,
Medium,Hard) and a test environment. We selected one data combination while
experimenting with ways to build spurious correlations. Those are
Spawrious-{O2O}-{Easy} and Spawrious-{M2M}-{Hard} which we used for early
experiments and fine-tuning. To find the remaining data combinations for
Spawrious-{O2O}-{Medium, Hard} and Spawrious-{M2M}-{Easy, Medium}, we
performed a random search within a space of possible data combinations for 20
random trials on O2O and M2M each. Full details on the final data combinations
are here: Table 2.

##  5 Experiments

Method | One-To-One SC | Many-To-Many SC | Average
---|---|---|---
Easy | Medium | Hard | Easy | Medium | Hard
ResNet18 pre-trained on ImageNet
ERM [67] | $72.15\%_{\pm 0.03}$ | $69.85\%_{\pm 0.01}$ | $65.65\%_{\pm 0.04}$ | $72.51\%_{\pm 0.05}$ | $51.36\%_{\pm 0.04}$ | $47.02\%_{\pm 0.01}$ | $63.09\%$
GroupDRO [61] | $68.72\%_{\pm 0.02}$ | $71.87\%_{\pm 0.01}$ | $60.90\%_{\pm 0.03}$ | $74.82\%_{\pm 0.04}$ | $52.06\%_{\pm 0.03}$ | $52.79\%_{\pm 0.03}$ | $63.53\%$
IRM [1] | $71.26\%_{\pm 0.02}$ | $68.18\%_{\pm 0.02}$ | $63.78\%_{\pm 0.03}$ | $73.28\%_{\pm 0.04}$ | $42.43\%_{\pm 0.07}$ | $44.51\%_{\pm 0.06}$ | $60.57\%$
CORAL [64] | $83.85\%_{\pm 0.01}$ | $73.96\%_{\pm 0.01}$ | $72.18\%_{\pm 0.03}$ | $79.91\%_{\pm 0.00}$ | $58.09\%_{\pm 0.01}$ | $56.51\%_{\pm 0.03}$ | $70.75\%$
CausIRL [7] | $\mathbf{84.21\%_{\pm 0.01}}$ | $73.45\%_{\pm 0.02}$ | $71.20\%_{\pm 0.02}$ | $81.21\%_{\pm 0.01}$ | $56.79\%_{\pm 0.01}$ | $56.31\%_{\pm 0.03}$ | $70.52\%$
MMD-AAE [42] | $82.92\%_{\pm 0.03}$ | $\mathbf{74.09\%_{\pm 0.03}}$ | $\mathbf{72.60\%_{\pm 0.05}}$ | $\mathbf{83.45\%_{\pm 0.01}}$ | $\mathbf{60.27\%_{\pm 0.03}}$ | $\mathbf{58.26\%_{\pm 0.00}}$ | $\mathbf{71.93\%}$
ResNet50 pre-trained on ImageNet
ERM [67] | $77.49\%_{\pm 0.05}$ | $76.60\%_{\pm 0.02}$ | $71.32\%_{\pm 0.09}$ | $83.80\%_{\pm 0.01}$ | $53.05\%_{\pm 0.03}$ | $58.70\%_{\pm 0.04}$ | $70.16\%$
GroupDRO [61] | $80.58\%_{\pm 0.01}$ | $75.96\%_{\pm 0.02}$ | $76.99\%_{\pm 0.03}$ | $79.96\%_{\pm 0.03}$ | $61.01\%_{\pm 0.05}$ | $60.86\%_{\pm 0.02}$ | $72.56\%$
IRM [1] | $75.45\%_{\pm 0.03}$ | $76.39\%_{\pm 0.02}$ | $74.90\%_{\pm 0.01}$ | $76.15\%_{\pm 0.03}$ | $\mathbf{67.82\%_{\pm 0.04}}$ | $60.93\%_{\pm 0.01}$ | $71.94\%$
CORAL [64] | $\mathbf{89.66\%_{\pm 0.01}}$ | $\mathbf{81.05\%_{\pm 0.01}}$ | $79.65\%_{\pm 0.02}$ | $81.26\%_{\pm 0.02}$ | $65.18\%_{\pm 0.05}$ | $67.97\%_{\pm 0.01}$ | $\mathbf{77.46\%}$
CausIRL [7] | $89.32\%_{\pm 0.01}$ | $78.64\%_{\pm 0.01}$ | $\mathbf{80.40\%_{\pm 0.01}}$ | $\mathbf{85.76\%_{\pm 0.01}}$ | $63.15\%_{\pm 0.03}$ | $\mathbf{68.93\%_{\pm 0.01}}$ | $77.20\%$
MMD-AAE [42] | $78.81\%_{\pm 0.02}$ | $75.33\%_{\pm 0.03}$ | $72.66\%_{\pm 0.01}$ | $80.55\%_{\pm 0.02}$ | $59.43\%_{\pm 0.04}$ | $54.39\%_{\pm 0.05}$ | $70.20\%$

Table 3: Results for Spawrious-{O2O,M2M}-{Easy, Medium, Hard}

In this section, we evaluate various group robustness methods on Spawrious and
discuss the results.

#### Methods

The field of worst-group-accuracy optimization is thriving with a plethora of
proposed methods, making it impractical to compare all available methods. We
choose the following six popular methods and their DomainBed implementation
[20]. ERM [66] refers to the canonical, average-accuracy-optimization
procedure, where we treat all groups identically and ignore group labels, not
targeting to improve the worst group performance. CORAL [64] penalizes
differences in the first and second moment of the feature distributions of
each group. IRM [1] is a causality-inspired [33] invariance-learning method,
which penalizes feature distributions that have different optimal linear
classifiers for each group. CausIRL [7] is another causally-motivated
algorithm for learning invariances, whose penalty considers only one distance
between mixtures of latent features coming from different domains. GroupDRO
[61] uses Group-Distributional Robust Optimization to explicitly minimize the
worst group loss instead of the average loss. MMD-AAE [42] penalizes distances
between feature distributions of groups via the maximum mean discrepancy (MMD)
and learning an adversarial auto-encoder (AAE).

#### Hyper-parameter tuning

We follow the hyper-parameter tuning process used in DomainBed [20] with a
minor modification. We keep the dropout rate ($0.1$) and the batch size fixed
(128 for ResNets and 64 for ViTs) because we found them to have only a very
marginal impact on the performance. We tune the learning rate and weight decay
on ERM with a random search of 20 random trials. For all other methods, we
further tune their method-specific hyper-parameters with a search of 10 random
trials. We perform model selection based on the training domain validation
accuracy of a subset of the training data. We reuse the hyper-parameters found
for Spawrious-{O2O}-{Easy} and Spawrious-{M2M}-{Hard} on
Spawrious-{O2O}-{Medium, Hard} and Spawrious-{M2M}-{Easy, Medium},
respectively. We also initially explored the ViT [11] architecture, with
results shown in Appendix A. Due to its poor performance, we chose to focus on
ResNet50 results.

#### Evaluation

We evaluate the classifiers on a test environment where the SCs present during
training change, as described in Table 2. For O2O, multiple ways exist to
choose a test data combination; we evaluate one of them as selected using a
random search process. In M2M, because there are only two class groups and two
background groups, we only need to swap them as seen in Figure 1(b).

###  5.1 Results

For our main results in Table 1, we fine-tune a ResNet18 [21] model pre-
trained on ImageNet, following previous work on domain generalization [12, 41,
20]. We also fine-tune larger models later in Appendix A.

There is a clear hierarchy among methods; the top three listed methods perform
significantly worse than the bottom three when using a ResNet18. The same
hierarchy is preserved when using a ResNet50 except for the MMD-AAE method
which gets worse after increasing the model size. This stimulating new result
contributes to the debate whether, for a fixed architecture, most robustness
methods perform about the same [20] or not [70]. The performances of most
methods get consistently worse as the challenge becomes harder. Most often,
the data splits of our newly formalized M2M-SC are significantly more
challenging than the O2O splits, most notably _M2M-{Hard, Medium}_. We
conjecture that there is a strong need for new methods targeting such. {ERM,
GroupDRO} and {CORAL, CausIRL} perform about the same, despite much different
robustness regularization. All methods consistently achieve 98-99% in-
distribution test performance (not shown in Table 1 to save space) despite
differences in OOD performance. IRM performs worst on average, with a
significant difference of 11.36% compared to the best method (MMD-AAE). This
result empirically confirms recent warnings about IRM’s effectiveness [60,
34].

###  5.2 Misclassification analysis

In Section 5.1, we learned that ERM performs particularly poorly on both hard
challenges. Now, we want to investigate why by examining some of the
misclassifications.

Figure 4: ERM misclassifications due to spurious correlations. The shown test
images correspond to the class _“Bulldog”_ with spurious backgrounds
_“Mountains”_ in the O2O-Hard (left) and _“Snow”_ in the M2M-Hard (right)
challenge.

For example, we observe in Figure 4 that on the test set, the class
_“Bulldog”_ is misclassified as the classes whose most common training set
background is the same as _“Bulldog”_ ’s test backgrounds.

Note that for all classes and in all data groups, both training and test
environments, the number of data points per class is always balanced;
rendering methods like _Subsampling large classes_ [26], which achieve state-
of-the-art performance on other SC benchmarks, inapplicable. Hence, we
conjecture that despite balanced classes, the model heavily relies on the
spurious features of the _“Mountains”_ and _“Snow”_ backgrounds.

(a) O2O-Hard

(b) M2M-Hard

Figure 5: Confusion matrices for ERM models. $X$-axis: predictions; $Y$-axis:
true labels.

We further corroborate that claim by examining the model’s confusion matrix in
Figure 5. For example, Figure 5(a) shows the highest non-diagonal value for
actual _“Dachshund”_ images being wrongly classified as _“Labrador”_. We
conjecture the reason being that in O2O-Hard, the background of _“Dachshund”_
in the test set is _“Snow”_ , which is the most common background of the
training images of _“Labrador”_ , as shown in Table 2.

##  6 Related Work

#### Failure Mode Analyses of Distribution Shifts

Nagarajan et al. [52] analyze distribution shift failure modes and find that
two different kinds of shifts can cause issues: _geometric_ and _statistical_
skew. Geometric skew occurs when there is an imbalance between groups of types
of data points (i.e., data points from different environments) which induces a
spurious correlation and leads to misclassification when the balance of groups
changes. This understanding has motivated simply removing data points from the
training data to balance between groups of data points [2]. In contrast, we
study two particular types of SCs, which persist in degenerating
generalization performance despite perfect balances of classes among groups.

#### Causality

The theory of causation provides another perspective on the sources and
possible mitigations of spurious correlations [56, 33]. Namely, we can
formalize environment-specific data as samples from different interventional
distributions, which keep the influence of variables not affected by the
corresponding interventions invariant. This perspective has motivated several
invariance-learning methods that make causal assumptions on the data-
generating process [1, 48, 49]. The field of treatment effect estimation also
deals with mitigating spurious correlations from observational data [6, 38,
31, 54].

##  7 Limitations and Future Work

Instantiating our desiderata with non-background spurious attributes. For
example, Neuhaus et al. [53] find that in the ImageNet [10] dataset, the class
_“Hard Disc”_ is spuriously correlated with _“label”_ , however, _“label”_ is
not a background feature but rather part of the classification object.
Instantiating our desiderata for other data modalities, e.g., text
classification, leveraging the text generation capabilities of large language
models [4]. Evaluating more generalization techniques on Spawrious, including
different robustness penalties [44, 3, 37, 5, 48, 27, 57], meta-learning [77,
9, 30, 69, 28], unsupervised domain adaptation [14, 46, 74], dropout [39],
flat minima [5, 32], weight averaging [58, 72, 29], (counterfactual) data
augmentation [33, 19, 75, 76], fine-tuning of only specific layers [35, 40],
diversity [65, 58], etc.

##  8 Conclusion

We present Spawrious, an image classification benchmark with two types of
spurious correlations, one-to-one (O2O) and many-to-many (M2M). We carefully
design six dataset desiderata and instantiate them by leveraging recent
advances in text-to-image and image captioning models. Next, we conduct
experiments, and our findings indicate that even state-of-the-art group
robustness techniques are insufficient in handling Spawrious, particularly in
scenarios with Hard-splits where accuracy is below $70\%$. Our analysis of
model errors revealed a dependence on irrelevant backgrounds, thus
underscoring the difficulty of our dataset and highlighting the need for
further investigations in this area.

## References

  * Arjovsky et al. [2019] Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Arjovsky et al. [2022] Arjovsky, M., Chaudhuri, K., and Lopez-Paz, D.  Throwing away data improves worst-class error in imbalanced classification.  _arXiv preprint arXiv:2205.11672_ , 2022.
  * Blumberg et al. [2019] Blumberg, S. B., Palombo, M., Khoo, C. S., Tax, C. M. W., Tanno, R., and Alexander, D. C.  Multi-stage prediction networks for data harmonization, 2019.  URL https://arxiv.org/abs/1907.11629.
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cha et al. [2021] Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S., Lee, Y., and Park, S.  Swad: Domain generalization by seeking flat minima.  _Advances in Neural Information Processing Systems_ , 34:22405–22418, 2021.
  * Chernozhukov et al. [2018] Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J.  Double/debiased machine learning for treatment and structural parameters, 2018.
  * Chevalley et al. [2022] Chevalley, M., Bunne, C., Krause, A., and Bauer, S.  Invariant causal mechanisms through distribution matching.  _arXiv preprint arXiv:2206.11646_ , 2022.
  * Christie et al. [2017] Christie, G., Fendley, N., Wilson, J., and Mukherjee, R.  Functional map of the world, 2017.  URL https://arxiv.org/abs/1711.07846.
  * Collins et al. [2020] Collins, L., Mokhtari, A., and Shakkottai, S.  Task-robust model-agnostic meta-learning.  _Advances in Neural Information Processing Systems_ , 33:18860–18871, 2020.
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dosovitskiy et al. [2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dou et al. [2019] Dou, Q., Coelho de Castro, D., Kamnitsas, K., and Glocker, B.  Domain generalization via model-agnostic learning of semantic features.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * D’Amour et al. [2020] D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al.  Underspecification presents challenges for credibility in modern machine learning.  _Journal of Machine Learning Research_ , 2020.
  * Ganin & Lempitsky [2015] Ganin, Y. and Lempitsky, V.  Unsupervised domain adaptation by backpropagation.  In _International conference on machine learning_ , pp. 1180–1189. PMLR, 2015.
  * Geirhos et al. [2018a] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  _arXiv preprint arXiv:1811.12231_ , 2018a.
  * Geirhos et al. [2018b] Geirhos, R., Temme, C. R., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A.  Generalisation in humans and deep neural networks.  _Advances in neural information processing systems_ , 31, 2018b.
  * Geirhos et al. [2020] Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, 2020.
  * Ghosal et al. [2022] Ghosal, S. S., Ming, Y., and Li, Y.  Are vision transformers robust to spurious correlations?, 2022.  URL https://arxiv.org/abs/2203.09125.
  * Gowal et al. [2021] Gowal, S., Rebuffi, S.-A., Wiles, O., Stimberg, F., Calian, D. A., and Mann, T. A.  Improving robustness using generated data.  _Advances in Neural Information Processing Systems_ , 34:4218–4233, 2021.
  * Gulrajani & Lopez-Paz [2021] Gulrajani, I. and Lopez-Paz, D.  In search of lost domain generalization.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=lQdXeXDoWtI.
  * He et al. [2016] He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks & Dietterich [2019] Hendrycks, D. and Dietterich, T.  Benchmarking neural network robustness to common corruptions and perturbations, 2019.  URL https://arxiv.org/abs/1903.12261.
  * Hermann et al. [2020] Hermann, K., Chen, T., and Kornblith, S.  The origins and prevalence of texture bias in convolutional neural networks.  _Advances in Neural Information Processing Systems_ , 33:19000–19015, 2020.
  * Howard [2019a] Howard, J.  Imagenette: A smaller subset of 10 easily classified classes from imagenet, March 2019a.  URL https://github.com/fastai/imagenette.
  * Howard [2019b] Howard, J.  Imagewoof: a subset of 10 classes from imagenet that aren’t so easy to classify, March 2019b.  URL https://github.com/fastai/imagenette#imagewoof.
  * Idrissi et al. [2022] Idrissi, B. Y., Arjovsky, M., Pezeshki, M., and Lopez-Paz, D.  Simple data balancing achieves competitive worst-group-accuracy.  In _Conference on Causal Learning and Reasoning_ , pp. 336–351. PMLR, 2022.
  * Izmailov et al. [2022] Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G.  On feature learning in the presence of spurious correlations.  _arXiv preprint arXiv:2210.11369_ , 2022.
  * Jiang et al. [2023] Jiang, P., Xin, K., Wang, Z., and Li, C.  Invariant meta learning for out-of-distribution generalization.  _arXiv preprint arXiv:2301.11779_ , 2023.
  * Kaddour [2022] Kaddour, J.  Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging.  In _Has it Trained Yet? NeurIPS 2022 Workshop_ , 2022.  URL https://openreview.net/forum?id=0OrABUHZuz.
  * Kaddour et al. [2020] Kaddour, J., Saemundsson, S., and Deisenroth (he/him), M.  Probabilistic Active Meta-Learning.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 20813–20822. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/ef0d17b3bdb4ee2aa741ba28c7255c53-Paper.pdf.
  * Kaddour et al. [2021] Kaddour, J., Zhu, Y., Liu, Q., Kusner, M. J., and Silva, R.  Causal effect inference for structured treatments.  _Advances in Neural Information Processing Systems_ , 34:24841–24854, 2021.
  * Kaddour et al. [2022a] Kaddour, J., Liu, L., Silva, R., and Kusner, M.  When do flat minima optimizers work?  In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), _Advances in Neural Information Processing Systems_ , 2022a.  URL https://openreview.net/forum?id=vDeh2yxTvuh.
  * Kaddour et al. [2022b] Kaddour, J., Lynch, A., Liu, Q., Kusner, M. J., and Silva, R.  Causal machine learning: A survey and open problems.  _arXiv preprint arXiv:2206.15475_ , 2022b.  URL https://arxiv.org/abs/2206.15475.
  * Kamath et al. [2021] Kamath, P., Tangella, A., Sutherland, D., and Srebro, N.  Does invariant risk minimization capture invariance?  In _International Conference on Artificial Intelligence and Statistics_ , pp. 4069–4077. PMLR, 2021.
  * Kirichenko et al. [2022] Kirichenko, P., Izmailov, P., and Wilson, A. G.  Last layer re-training is sufficient for robustness to spurious correlations.  _arXiv preprint arXiv:2204.02937_ , 2022.
  * Koh et al. [2021] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pp. 5637–5664. PMLR, 2021.
  * Krueger et al. [2021] Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Le Priol, R., and Courville, A.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pp. 5815–5826. PMLR, 2021.
  * Künzel et al. [2019] Künzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B.  Metalearners for estimating heterogeneous treatment effects using machine learning.  _Proceedings of the national academy of sciences_ , 116(10):4156–4165, 2019.
  * LaBonte et al. [2022] LaBonte, T., Muthukumar, V., and Kumar, A.  Dropout disagreement: A recipe for group robustness with fewer annotations.  In _NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2022.
  * Lee et al. [2023] Lee, Y., Chen, A. S., Tajwar, F., Kumar, A., Yao, H., Liang, P., and Finn, C.  Surgical Fine-Tuning Improves Adaptation to Distribution Shifts, March 2023.  URL http://arxiv.org/abs/2210.11466.  arXiv:2210.11466 [cs].
  * Li et al. [2019] Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y.-Z., and Hospedales, T. M.  Episodic training for domain generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1446–1455, 2019.
  * Li et al. [2018] Li, H., Pan, S. J., Wang, S., and Kot, A. C.  Domain generalization with adversarial feature learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 5400–5409, 2018.
  * Liang & Zou [2022] Liang, W. and Zou, J.  Metashift: A dataset of datasets for evaluating contextual distribution shifts and training conflicts.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=MTex8qKavoS.
  * Liu et al. [2021] Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C.  Just train twice: Improving group robustness without training group information.  In _International Conference on Machine Learning_ , pp. 6781–6792. PMLR, 2021.
  * Liu et al. [2015] Liu, Z., Luo, P., Wang, X., and Tang, X.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3730–3738, 2015.
  * Long et al. [2016] Long, M., Zhu, H., Wang, J., and Jordan, M. I.  Unsupervised domain adaptation with residual transfer networks.  _Advances in neural information processing systems_ , 29, 2016.
  * Lynch et al. [2022] Lynch, A., Kaddour, J., and Silva, R.  Evaluating the impact of geometric and statistical skews on out-of-distribution generalization performance.  In _NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications_ , 2022.  URL https://openreview.net/forum?id=wpT79coXAu.
  * Mahajan et al. [2021] Mahajan, D., Tople, S., and Sharma, A.  Domain generalization using causal matching.  In _International Conference on Machine Learning_ , pp. 7313–7324. PMLR, 2021.
  * Mao et al. [2022] Mao, C., Xia, K., Wang, J., Wang, H., Yang, J., Bareinboim, E., and Vondrick, C.  Causal transportability for visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7521–7531, 2022.
  * Mehta et al. [2022] Mehta, R., Albiero, V., Chen, L., Evtimov, I., Glaser, T., Li, Z., and Hassner, T.  You only need a good embeddings extractor to fix spurious correlations.  _arXiv preprint arXiv:2212.06254_ , 2022.
  * Murphy [2022] Murphy, K. P.  _Probabilistic Machine Learning: An introduction_.  MIT Press, 2022.  URL probml.ai.
  * Nagarajan et al. [2020] Nagarajan, V., Andreassen, A., and Neyshabur, B.  Understanding the failure modes of out-of-distribution generalization, 2020.  URL https://arxiv.org/abs/2010.15775.
  * Neuhaus et al. [2022] Neuhaus, Y., Augustin, M., Boreiko, V., and Hein, M.  Spurious features everywhere – large-scale detection of harmful spurious features in imagenet, 2022.  URL https://arxiv.org/abs/2212.04871.
  * Nie & Wager [2021] Nie, X. and Wager, S.  Quasi-oracle estimation of heterogeneous treatment effects.  _Biometrika_ , 108(2):299–319, 2021.
  * NLP Connect [2022] NLP Connect.  vit-gpt2-image-captioning (revision 0e334c7), 2022.  URL https://huggingface.co/nlpconnect/vit-gpt2-image-captioning.
  * Peters et al. [2017] Peters, J., Janzing, D., and Schölkopf, B.  _Elements of causal inference: foundations and learning algorithms_.  The MIT Press, 2017.
  * Rame et al. [2022a] Rame, A., Dancette, C., and Cord, M.  Fishr: Invariant gradient variances for out-of-distribution generalization.  In _International Conference on Machine Learning_ , pp. 18347–18377. PMLR, 2022a.
  * Rame et al. [2022b] Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M.  Diverse weight averaging for out-of-distribution generalization.  _arXiv preprint arXiv:2205.09739_ , 2022b.
  * Rombach et al. [2022] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10684–10695, 2022.
  * Rosenfeld et al. [2020] Rosenfeld, E., Ravikumar, P., and Risteski, A.  The risks of invariant risk minimization.  _arXiv preprint arXiv:2010.05761_ , 2020.
  * Sagawa et al. [2019] Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P.  Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2019.  URL https://arxiv.org/abs/1911.08731.
  * Schuhmann et al. [2022] Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S., Crowson, K., Schmidt, L., Kaczmarczyk, R., and Jitsev, J.  Laion-5b: An open large-scale dataset for training next generation image-text models, 2022.  URL https://arxiv.org/abs/2210.08402.
  * Singla & Feizi [2021] Singla, S. and Feizi, S.  Salient imagenet: How to discover spurious features in deep learning?  _arXiv preprint arXiv:2110.04301_ , 2021.
  * Sun & Saenko [2016] Sun, B. and Saenko, K.  Deep coral: Correlation alignment for deep domain adaptation.  In _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_ , pp. 443–450. Springer, 2016.
  * Teney et al. [2022] Teney, D., Abbasnejad, E., Lucey, S., and Van den Hengel, A.  Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 16761–16772, 2022.
  * Vapnik [1991a] Vapnik, V.  Principles of risk minimization for learning theory.  _Advances in neural information processing systems_ , 4, 1991a.
  * Vapnik [1991b] Vapnik, V.  Principles of risk minimization for learning theory.  _Advances in neural information processing systems_ , 4, 1991b.
  * Vendrow et al. [2023] Vendrow, J., Jain, S., Engstrom, L., and Madry, A.  Dataset interfaces: Diagnosing model failures using controllable counterfactual generation.  _arXiv preprint arXiv:2302.07865_ , 2023.
  * Wang et al. [2021] Wang, Z., Duan, T., Fang, L., Suo, Q., and Gao, M.  Meta learning on a sequence of imbalanced domains with difficulty awareness.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 8947–8957, 2021.
  * Wiles et al. [2021] Wiles, O., Gowal, S., Stimberg, F., Alvise-Rebuffi, S., Ktena, I., Dvijotham, K., and Cemgil, T.  A fine-grained analysis on distribution shift.  _arXiv preprint arXiv:2110.11328_ , 2021.
  * Wiles et al. [2022] Wiles, O., Albuquerque, I., and Gowal, S.  Discovering bugs in vision models using off-the-shelf image generation and captioning.  _arXiv preprint arXiv:2208.08831_ , 2022.
  * Wortsman et al. [2022] Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In _International Conference on Machine Learning_ , pp. 23965–23998. PMLR, 2022.
  * Xiao et al. [2020] Xiao, K., Engstrom, L., Ilyas, A., and Madry, A.  Noise or signal: The role of image backgrounds in object recognition.  _arXiv preprint arXiv:2006.09994_ , 2020.
  * Xu et al. [2021] Xu, T., Chen, W., Wang, P., Wang, F., Li, H., and Jin, R.  Cdtrans: Cross-domain transformer for unsupervised domain adaptation.  _arXiv preprint arXiv:2109.06165_ , 2021.
  * Yao et al. [2022] Yao, H., Wang, Y., Li, S., Zhang, L., Liang, W., Zou, J., and Finn, C.  Improving Out-of-Distribution Robustness via Selective Augmentation.  In _Proceedings of the 39th International Conference on Machine Learning_ , pp. 25407–25437. PMLR, June 2022.  URL https://proceedings.mlr.press/v162/yao22b.html.  ISSN: 2640-3498.
  * Yin et al. [2023] Yin, Y., Kaddour, J., Zhang, X., Nie, Y., Liu, Z., Kong, L., and Liu, Q.  Ttida: Controllable generative data augmentation via text-to-text and text-to-image models, 2023.
  * Zhang et al. [2020] Zhang, M., Marklund, H., Gupta, A., Levine, S., and Finn, C.  Adaptive risk minimization: A meta-learning approach for tackling group shift.  _arXiv preprint arXiv:2007.02931_ , 8:9, 2020.
  * Zhang et al. [2023] Zhang, X., He, Y., Wang, T., Qi, J., Yu, H., Wang, Z., Peng, J., Xu, R., Shen, Z., Niu, Y., et al.  Nico challenge: Out-of-distribution generalization for image recognition challenges.  In _European Conference on Computer Vision_ , pp. 433–450. Springer, 2023.

##  Appendix A Effect of Model Architectures

Method | One-To-One SC | Many-To-Many SC | Average
---|---|---|---
Easy | Medium | Hard | Easy | Medium | Hard
ViT-B/16 (ERM) | $69.06\%_{\pm 0.05}$ | $68.58\%_{\pm 0.01}$ | $59.38\%_{\pm 0.02}$ | $71.46\%_{\pm 0.02}$ | $43.16\%_{\pm 0.03}$ | $44.99\%_{\pm 0.02}$ | $59.44\%$
ViT-B/16 (MMD-AAE) | $71.29\%_{\pm 0.06}$ | $65.40\%_{\pm 0.01}$ | $60.83\%_{\pm 0.06}$ | $75.72\%_{\pm 0.01}$ | $46.45\%_{\pm 0.02}$ | $42.46\%_{\pm 0.03}$ | $60.36\%$

Table 4: Effect of Different Architectures on Spawrious-{O2O,M2M}-{Easy,
Medium, Difficult}

We experiment with the ViT-B/16 [11], following [27, 50]. Based on Table 4, we
make the following observations: The ViT backbone architecture worsens the
performance for both MMD-AAE and ERM, underperforming the ResNet18 and the
larger ResNet50. The best results were obtained by the CORAL algorithm applied
to a ResNet50 which performs 17.1% better than when than the best ViT. In the
debate on whether ViTs [11] are generally more robust to SCs [18] than CNNs or
not [27, 50], our results side with the latter. We observe that a ViT-B/16
pretrained on ImageNet22k had worse test accuracies than the ResNet
architectures.

##  Appendix B Appendix

Include extra information in the appendix. This section will often be part of
the supplemental material. Please see the call on the NeurIPS website for
links to additional guides on dataset publication.

  1. 1.

Submission introducing new datasets must include the following in the
supplementary materials:

    1. (a)

Dataset documentation and intended uses. Recommended documentation frameworks
include datasheets for datasets, dataset nutrition labels, data statements for
NLP, and accountability frameworks.

    2. (b)

URL to website/platform where the dataset/benchmark can be viewed and
downloaded by the reviewers.

    3. (c)

Author statement that they bear all responsibility in case of violation of
rights, etc., and confirmation of the data license.

    4. (d)

Hosting, licensing, and maintenance plan. The choice of hosting platform is
yours, as long as you ensure access to the data (possibly through a curated
interface) and will provide the necessary maintenance.

  2. 2.

To ensure accessibility, the supplementary materials for datasets must include
the following:

    1. (a)

Links to access the dataset and its metadata. This can be hidden upon
submission if the dataset is not yet publicly available but must be added in
the camera-ready version. In select cases, e.g when the data can only be
released at a later date, this can be added afterward. Simulation environments
should link to (open source) code repositories.

    2. (b)

The dataset itself should ideally use an open and widely used data format.
Provide a detailed explanation on how the dataset can be read. For simulation
environments, use existing frameworks or explain how they can be used.

    3. (c)

Long-term preservation: It must be clear that the dataset will be available
for a long time, either by uploading to a data repository or by explaining how
the authors themselves will ensure this.

    4. (d)

Explicit license: Authors must choose a license, ideally a CC license for
datasets, or an open source license for code (e.g. RL environments).

    5. (e)

Add structured metadata to a dataset’s meta-data page using Web standards
(like schema.org and DCAT): This allows it to be discovered and organized by
anyone. If you use an existing data repository, this is often done
automatically.

    6. (f)

Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted
by a data repository or a prefix on identifiers.org) for datasets, or a code
repository (e.g. GitHub, GitLab,…) for code. If this is not possible or
useful, please explain why.

  3. 3.

For benchmarks, the supplementary materials must ensure that all results are
easily reproducible. Where possible, use a reproducibility framework such as
the ML reproducibility checklist, or otherwise guarantee that all results can
be easily reproduced, i.e. all necessary datasets, code, and evaluation
procedures must be accessible and documented.

  4. 4.

For papers introducing best practices in creating or curating datasets and
benchmarks, the above supplementary materials are not required.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]  See Section 8

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See https://github.com/aengusl/spawrious

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]  see Section 5

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]  see Table 3

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [No]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]  We cite
the pretrained models and architectures we use

    2. (b)

Did you mention the license of the assets? [No]  License is in the github
repository https://github.com/aengusl/spawrious

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [No]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [N/A]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [N/A]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]


28

# Tilt-then-Blur or Blur-then-Tilt?
Clarifying the Atmospheric Turbulence Model

Stanley H. Chan The author is with the School of Electrical and Computer
Engineering, Purdue University, West Lafayette, IN 47907, USA. Email:
<EMAIL_ADDRESS>work is supported in part by the Intelligence Advanced
Research Projects Activity (IARPA) under Contract No. 2022-21102100004, and in
part by the National Science Foundation under the grants CCSS-2030570 and
IIS-2133032. The views and conclusions contained herein are those of the
authors and should not be interpreted as necessarily representing the official
policies, either expressed or implied, of IARPA, or the U.S. Government. The
U.S. Government is authorized to reproduce and distribute reprints for
governmental purposes notwithstanding any copyright annotation therein.

###### Abstract

Imaging at a long distance often requires advanced image restoration
algorithms to compensate for the distortions caused by atmospheric turbulence.
However, unlike many standard restoration problems such as deconvolution, the
forward image formation model of the atmospheric turbulence does not have a
simple expression. Thanks to the Zernike representation of the phase, one can
show that the forward model is a combination of tilt (pixel shifting due to
the linear phase terms) and blur (image smoothing due to the high order
aberrations).

Confusions then arise between the ordering of the two operators. Should the
model be tilt-then-blur, or blur-then-tilt? Some papers in the literature say
that the model is tilt-then-blur, whereas more papers say that it is blur-
then-tilt. This paper clarifies the differences between the two and discusses
why the tilt-then-blur is the correct model. Recommendations are given to the
research community.

###### Index Terms:

Atmospheric turbulence, image restoration, forward model, simulation,
convolution, signal processing

##  I Introduction

Over the past decade, there is a significant growth of image processing
research focusing on mitigating the atmospheric turbulence effects present in
images and videos taken by long-range cameras [1, 2, 3, 4, 5]. Since image
restoration is an inverse problem, knowing the forward image formation model
is necessary to formulate the restoration problem and derive the optimization
algorithm. However, unlike the many restoration problems such as
deconvolution, the atmospheric turbulence does not have a simple equation that
can describe how images are distorted. In fact, optics textbooks tell us that
the turbulent effect is caused by the changing index of refraction along the
optical path [6, 7, 8, 9]. The index of refraction is a stochastic process,
and the distortions are realized by how the phase of the wave is perturbed
[10, 11, 12].

From an image processing perspective, we all understand the importance of the
forward model but we also realize the difficulty of modeling the situation
using wave propagation theory. Thus, in the image processing literature, we
often see the so-called “tilt + blur” model [13, 14, 15]. The argument is that
the phase distortions will cause the pixels to _shift_ (thus the “tilt”) and
the high order aberrations will cause the image to look _smoothed_ (thus the
“blur”). Yet, when the two operators are present, the functional composition
requires us to specify the _order_. Shall we tilt the image first and then add
the blur, or shall we blur the image first and then add the tilt? Or, perhaps
it does not matter?

The confusion between the tilt-then-blur and the blur-then-tilt is not a light
one. To the best of the author’s knowledge, at least in the literature of
image processing, both models have been used. For the tilt-then-blur model,
one of the earlier papers is by Shimizu et al. published in CVPR 2008 [13].
When they solved the inverse problem, the optimization is implemented for the
tilt-then-blur model. Another highly referenced paper by Zhu and Milanfar,
published in T-PAMI in 2013, also used the tilt-then-blur model [16]. On the
other side of the spectrum, there is a consistent emphasis of the blur-then-
tilt model. One of the earlier work is by Mao and Gilles in 2012 [17], where
they modeled turbulence using blur-then-tilt. Mao and Gilles credited the
model to Frakes et al. [18] and Gepshtein et al. [19], although neither of
Frakes et al. nor Gepshtein et al. had any discussions of the forward model.
Subsequent work of Lou et al. [20] followed the blur-then-tilt model, and this
tradition continues until today where many of the latest deep learning based
approaches also use the blur-then-tilt model. For example, Lau et al. [21],
Nair and Patel [22], Yasarla and Patel [23], and Lau and Lui [24, 25] all
explicitly or implicitly use the blur-then-tilt model.

Which one is correct then? The goal of this paper is to clarify the difference
between the tilt-then-blur model and the blur-then-tilt model. There are three
main conclusions:

  * •

The blur-then-tilt model is unfortunately wrong.

  * •

The tilt-then-blur model is correct.

  * •

For natural images, the difference is not noticeable because many regions are
smooth. For point sources, the difference can be significant.

Figure 1: Comparing blur-then-tilt $\mathcal{T}\circ\mathcal{B}$ and tilt-
then-blur $\mathcal{B}\circ\mathcal{T}$. Given a grid of point sources, a
spatially invariant blur, and a dense tilt map, the result of
$\mathcal{T}\circ\mathcal{B}$ (the green case) shows a grid of destroyed blurs
whereas the result of $\mathcal{B}\circ\mathcal{T}$ (the yellow case) shows a
grid of shifted blurs. We remark that $\mathcal{T}\circ\mathcal{B}$ is
incorrect whereas $\mathcal{B}\circ\mathcal{T}$ is correct.

##  II Modeling Atmospheric Turbulence

As a wave propagates through the atmospheric turbulence, the phase is
distorted by the changing index of refraction. From the image formation point
of view, if the clean image is $J(\mathbf{x})$ where
$\mathbf{x}\in\mathbb{R}^{2}$ specifies the coordinate in the 2D space, the
distortion can be modeled by a spatially varying point spread function (PSF)
$h_{\mathbf{x}}(\mathbf{u})$. If the image $J(\mathbf{x})$ is discretized over
a grid of $N$ pixels $\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\\}$, the
observed image is

$I(\mathbf{x}_{i})=\sum_{j=1}^{N}h_{\mathbf{x}_{i}}(\mathbf{u}_{j})J(\mathbf{u}_{j}),\quad
i=1,\ldots,N.$ (1)

Here, $\mathbf{x}_{i}$ specifies the output coordinate at which the distorted
pixel should be located, and $\mathbf{u}_{j}$ is a running index for the
weighted average. (1) is the general form of a spatially _varying_
convolution. In the special case where the PSF is spatially invarying,
$h_{\mathbf{x}_{i}}(\mathbf{u}_{j})$ can be written as
$h(\mathbf{x}_{i}-\mathbf{u}_{j})$ which is the usual convolution kernel.

For atmospheric turbulence, if the source is incoherent (which is the typical
case for passive imaging systems without any laser transmitters), the PSF is
generated by taking the Fourier transform of the phase
$\phi_{\mathbf{x}}(\boldsymbol{\rho})$ via [6, 7]

$h_{\mathbf{x}}(\mathbf{u})=|\mathcal{F}(e^{-j2\pi\phi_{\mathbf{x}}(\boldsymbol{\rho})})|^{2}.$
(2)

Here, the phase function $\phi_{\mathbf{x}}(\boldsymbol{\rho})$ is defined
_per pixel_ at each coordinate $\mathbf{x}$. The variable
$\boldsymbol{\rho}\in\mathbb{R}^{2}$ denotes the phase coordinate, which is a
2D coordinate in the Fourier space.

The randomness of $h_{\mathbf{x}}(\mathbf{u})$ comes from the randomness of
the phase $\phi_{\mathbf{x}}(\boldsymbol{\rho})$ [10].
$\phi_{\mathbf{x}}(\boldsymbol{\rho})$ is constructed by cropping and
propagating the wave through a sequence of phase screens sampled from the
Kolmogorov power spectral density [9, 26, 8, 27]. This is a computationally
expensive process, but recent work has alleviated the difficulty by modeling
$\phi_{\mathbf{x}}(\boldsymbol{\rho})$ at the _aperture_ [28, 29]. Using the
Zernike polynomials111Zernike polynomials are orthogonal basis functions
defined on the unit disk. The unit disk is necessary here because most of the
apertures are circular in shape.
$\\{Z_{m}(\boldsymbol{\rho})\,|\,m=1,\ldots,M\\}$ as the basis representation,
the seminal work of Noll [30] states that we can write the phase
$\phi_{\mathbf{x}}(\boldsymbol{\rho})$ as [30]

$\displaystyle\phi_{\mathbf{x}}(\boldsymbol{\rho})=\sum_{m=1}^{M}a_{\mathbf{x},m}Z_{m}(\boldsymbol{\rho}),$
(3)

where $\\{a_{\mathbf{x},m}\,|\,m=1,\ldots,M\\}$ denote the $M$ Zernike
coefficients at coordinate $\mathbf{x}$. The Zernike coefficients are sampled
from a zero-mean Gaussian random process

$a_{\mathbf{x},m}\sim\mathcal{N}(0,\boldsymbol{\Sigma}),$ (4)

where the $(\mathbf{x},\mathbf{x}^{\prime},m,m^{\prime})$th component of
$\boldsymbol{\Sigma}$ is
$[\boldsymbol{\Sigma}]_{\mathbf{x},\mathbf{x}^{\prime},m,m^{\prime}}=\mathbb{E}[a_{\mathbf{x},m}a_{\mathbf{x}^{\prime},m^{\prime}}]$.
The joint expectation follows from Chanan [31], Takato and Yamaguchi [32], and
Chimitt and Chan [28].

The representation of $\phi_{\mathbf{x}}(\boldsymbol{\rho})$ in the Zernike
space gives us a way to decouple the _shift_ (ie, tilt) and the _smoothing_
(ie, blur). These two operations can be summarized as follows:

  * •

Tilt $\mathcal{T}$: The tilt is encoded by the first two Zernike bases via
$\phi_{\mathbf{x}}(\boldsymbol{\rho})=\sum_{m=1}^{2}a_{\mathbf{x},m}Z_{m}(\boldsymbol{\rho})$;

  * •

Blur $\mathcal{B}$: The blur is encoded by the remaining Zernike bases via
$\phi_{\mathbf{x}}(\boldsymbol{\rho})=\sum_{m=3}^{\infty}a_{\mathbf{x},m}Z_{m}(\boldsymbol{\rho})$.

However, how do we combine these two operations? There are a few options,
based on the ordering of the tilt and the blur:

$\displaystyle\text{Blur-then-tilt}:\;\;\;$ $\displaystyle
I(\mathbf{x})=[\mathcal{T}\circ\mathcal{B}](J(\mathbf{x}))=\mathcal{T}(\mathcal{B}(J(\mathbf{x}))),$
(5) $\displaystyle\text{Tilt-then-blur}:\;\;\;$ $\displaystyle
I(\mathbf{x})=[\mathcal{B}\circ\mathcal{T}](J(\mathbf{x}))=\mathcal{B}(\mathcal{T}(J(\mathbf{x}))),$
(6) $\displaystyle\text{Full model}:\;\;\;$ $\displaystyle
I(\mathbf{x})=\mathcal{H}(J(\mathbf{x})),$ (7)

where “$\circ$” denotes the functional composition. For reference, the full
turbulence model $I(\mathbf{x})=\mathcal{H}(J(\mathbf{x}))$ uses all the
Zernike coefficients _simultaneously_ without decoupling them into tilts and
blurs.

##  III Why are they different?

###  III-A Analysis from Matrices and Vectors

To understand the difference between $\mathcal{T}\circ\mathcal{B}$ and
$\mathcal{B}\circ\mathcal{T}$, consider a grid of point as shown in Figure 1.
The tilt map is synthesized using the first two Zernike coefficients whereas
the blur is synthesized using the remaining Zernike coefficients.

For digital images, the spatial coordinates are discretized and so the
operations by $\mathcal{T}$ and $\mathcal{B}$ can be written in terms of
matrices and vectors. Specifically, the operator $\mathcal{T}$ can be written
as a shifting matrix $\mathbf{T}\in\mathbb{R}^{N\times N}$ with the $(i,j)$th
entry being $[\mathbf{T}]_{ij}=t_{\mathbf{x}_{i}}(\mathbf{u}_{j})$, where
$t_{\mathbf{x}_{i}}(\mathbf{u}_{j})=1$ if a pixel located at $\mathbf{u}_{j}$
is relocated to coordinate $\mathbf{x}_{i}$, and
$t_{\mathbf{x}_{i}}(\mathbf{u}_{j})=0$ if otherwise. For example, if
$J(\mathbf{x})$ is a 1D signal and $\mathbf{T}$ shifts the signal by one
pixel, then $\mathbf{T}$ takes the form

$\mathbf{T}=\begin{bmatrix}0&1&0&0&\ldots&0\\\ 0&0&1&0&\ldots&0\\\
\vdots&\vdots&\vdots&\vdots&\ddots&\vdots\\\ 0&0&0&0&\ldots&0\end{bmatrix}.$
(8)

The operator $\mathcal{B}$ is a collection of tilt-free but spatially varying
blurs. In the matrix notation, we can define a matrix $\mathbf{B}$ where
$[\mathbf{B}]_{ij}=b_{\mathbf{x}_{i}}(\mathbf{u}_{j})$ where
$b_{\mathbf{x}_{i}}$ is the tilt-free blur located at $\mathbf{x}_{i}$. As
before, $b_{\mathbf{x}_{i}}$ is generated by the phase distortion at
$\mathbf{x}_{i}$ using high-order Zernike coefficients. The overall structure
of the matrix $\mathbf{B}$ is

$\mathbf{B}=\begin{bmatrix}b_{\mathbf{x}_{1}}(\mathbf{u}_{1})&b_{\mathbf{x}_{1}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{1}}(\mathbf{u}_{N})\\\
b_{\mathbf{x}_{2}}(\mathbf{u}_{1})&b_{\mathbf{x}_{2}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{2}}(\mathbf{u}_{N})\\\
\vdots&\vdots&\ddots&\vdots\\\
b_{\mathbf{x}_{N}}(\mathbf{u}_{1})&b_{\mathbf{x}_{N}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{N}}(\mathbf{u}_{N})\end{bmatrix}.$
(9)

At this point, it is easy to understand why $\mathcal{T}\circ\mathcal{B}$ and
$\mathcal{B}\circ\mathcal{T}$ cannot be the same. The reason is that matrices
do not commute, i.e., $\mathbf{T}\mathbf{B}\not=\mathbf{B}\mathbf{T}$. In
fact, if the tilt matrix is the one shown in (8), then $\mathbf{B}\mathbf{T}$
shifts $\mathbf{B}$ to the left

$\displaystyle\mathbf{B}\mathbf{T}=\begin{bmatrix}b_{\mathbf{x}_{1}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{1}}(\mathbf{u}_{N})&0\\\
b_{\mathbf{x}_{2}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{2}}(\mathbf{u}_{N})&0\\\
\vdots&\vdots&\ddots&\vdots\\\
b_{\mathbf{x}_{N}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{N}}(\mathbf{u}_{N})&0\end{bmatrix},$

whereas $\mathbf{T}\mathbf{B}$ shifts $\mathbf{B}$ upwards

$\displaystyle\mathbf{T}\mathbf{B}=\begin{bmatrix}b_{\mathbf{x}_{2}}(\mathbf{u}_{1})&b_{\mathbf{x}_{2}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{2}}(\mathbf{u}_{N})\\\
\vdots&\vdots&\ddots&\vdots\\\
b_{\mathbf{x}_{N}}(\mathbf{u}_{1})&b_{\mathbf{x}_{N}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{N}}(\mathbf{u}_{N})\\\
0&0&0&0\end{bmatrix}.$

Now, the question is: which one is correct? Let us go back to Figure 1.
Imagine that there is only one point source located at $\mathbf{x}$. This
point source is a delta function $J(\mathbf{u})=\delta(\mathbf{x}-\mathbf{u})$
which will be distorted by the PSF $h(\mathbf{u})$ via
$I(\mathbf{x})=\sum_{j}h(\mathbf{u}_{j})\delta(\mathbf{x}-\mathbf{u}_{j})=h(\mathbf{x})$.
Suppose that $h(\mathbf{x})$ is shifted by an amount $\mathbf{t}$. Physically,
the shift is caused by a phase offset in the exponential as can be seen in
(2). The tilt changes $I(\mathbf{x})=h(\mathbf{x})$ to
$I(\mathbf{x})=h(\mathbf{x}+\mathbf{t})$.

Using the tilt example shown in (8), the overall operator $\mathcal{H}$ is to
replace the blur $b_{\mathbf{x}_{i}}(\mathbf{u}_{j})$ by
$b_{\mathbf{x}_{i}}(\mathbf{u}_{j+1})$. Therefore, in terms of matrices and
vectors, $\mathcal{H}$ can be written as a matrix
$\mathbf{H}\in\mathbb{R}^{N\times N}$ where

$\mathbf{H}=\begin{bmatrix}b_{\mathbf{x}_{1}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{1}}(\mathbf{u}_{N})&0\\\
b_{\mathbf{x}_{2}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{2}}(\mathbf{u}_{N})&0\\\
\vdots&\vdots&\ddots&\vdots\\\
b_{\mathbf{x}_{N}}(\mathbf{u}_{2})&\ldots&b_{\mathbf{x}_{N}}(\mathbf{u}_{N})&0\end{bmatrix}=\mathbf{B}\mathbf{T}.$
(10)

Notice that the shift occurs _horizontally_ in the matrix because we are
shifting the blur. We are not relocating the output value to a different pixel
location.

###  III-B Analysis from Geometry

For readers who like a geometric explanation, here is the analysis. Assume
that the blur is spatially invariant so that each blur can be written as

$b_{\mathbf{x}_{i}}(\mathbf{u})=g(\mathbf{x}_{i}-\mathbf{u}).$ (11)

Also, for notation simplicity, let us rewrite the tilt operator $\mathcal{T}$
as a set of tilt vectors $\\{\mathbf{t}_{i}\;|\;i=1,\ldots,N\\}$. Each tilt
vector $\mathbf{t}_{i}$ is used to relocate pixel $J(\mathbf{x}_{i})$ to a new
coordinate $\mathbf{x}_{i}+\mathbf{t}_{i}$:

$I_{\mathcal{T}}(\mathbf{x}_{i}+\mathbf{t}_{i})=J(\mathbf{x}_{i}),\quad
i=1,\ldots,N.$

This new notation can avoid the complication arising from the matrix-vector
multiplication.

To clarify different combination of operations, it would be convenient to
consider the intermediate results $I_{\mathcal{B}}$ and $I_{\mathcal{T}}$ for
blur and tilt only, and the final results $I_{\mathcal{B}\circ\mathcal{T}}$
and $I_{\mathcal{T}\circ\mathcal{B}}$ for the two operators.

Blur-then-Tilt. When a blur is applied to the clean image $J(\mathbf{u})$, the
intermediate result $I_{\mathcal{B}}(\mathbf{x})$ is

$\displaystyle I_{\mathcal{B}}(\mathbf{x}_{i})$
$\displaystyle=\sum_{j=1}^{N}b_{\mathbf{x}_{i}}(\mathbf{u}_{j})J(\mathbf{u}_{j})=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{u}_{j})J(\mathbf{u}_{j}).$

This is simply a blurred point source. Now consider the tilt. The tilt
$\mathbf{t}_{i}$ assigns $I_{\mathcal{B}}(\mathbf{x}_{i})$ to a new pixel
location $\mathbf{x}_{i}+\mathbf{t}_{i}$ of the final image
$I_{\mathcal{T}\circ\mathcal{B}}$. That is,

$I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{x}_{i}+\mathbf{t}_{i})=I_{\mathcal{B}}(\mathbf{x}_{i}).$

Since $\mathbf{t}_{1},\ldots,\mathbf{t}_{N}$ is a dense field, $\mathcal{T}$
will move _every pixel_ individually to a new location. As a result, the blur
is not shifted uniformly to a new coordinate but each pixel is shifted to a
different coordinate. Thus, the blur is _destroyed_.

To prepare for a follow up discussion, we let
$\mathbf{v}_{i}=\mathbf{x}_{i}+\mathbf{t}_{i}$ and so
$I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{v}_{i})=I_{\mathcal{B}}(\mathbf{v}_{i}-\mathbf{t}_{i})$.
Since $\mathbf{v}_{i}$ is a dummy variable, $I_{\mathcal{T}\circ\mathcal{B}}$
becomes

$I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{x}_{i})=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{t}_{i}-\mathbf{u}_{j})J(\mathbf{u}_{j}).$
(12)

Tilt-then-Blur. When tilt is applied to the clean image $J(\mathbf{u})$, the
intermediate result $I_{\mathcal{T}}(\mathbf{x})$ is

$I_{\mathcal{T}}(\mathbf{u}_{j}+\mathbf{t}_{j})=J(\mathbf{u}_{j}).$ (13)

If now a blur is applied to $I_{\mathcal{T}}$, the blur will simply move to
the new location $\mathbf{u}_{j}+\mathbf{t}_{j}$. The shape of the blur is
_preserved_ , and this is what $\mathcal{H}$ is supposed to do.

To complete the discussion. we let
$\mathbf{v}_{j}=\mathbf{u}_{j}+\mathbf{t}_{j}$. Then
$I_{\mathcal{T}}(\mathbf{v}_{j})=J(\mathbf{v}_{j}-\mathbf{t}_{j})$, and

$\displaystyle I_{\mathcal{B}\circ\mathcal{T}}(\mathbf{x}_{i})$
$\displaystyle=\sum_{j=1}^{N}b_{\mathbf{x}_{i}}(\mathbf{v}_{j})I_{\mathcal{T}}(\mathbf{v}_{j})=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{v}_{j})J(\mathbf{v}_{j}-\mathbf{t}_{j}).$

Replacing $\mathbf{u}_{j}=\mathbf{v}_{j}-\mathbf{t}_{j}$, it follow that

$I_{\mathcal{B}\circ\mathcal{T}}(\mathbf{x}_{i})=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{t}_{j}-\mathbf{u}_{j})J(\mathbf{u}_{j}).$
(14)

Comparing (12) and (14), the difference between the two equations is that for
$\mathcal{T}\circ\mathcal{B}$, the tilt is $\mathbf{t}_{i}$ whereas for
$\mathcal{B}\circ\mathcal{T}$ the tilt is $\mathbf{t}_{j}$. Their impacts to
the image are shown in Figure 2.

Figure 2: $\mathcal{T}\circ\mathcal{B}$ assigns one tilt $\mathbf{t}_{i}$ to
one output, whereas $\mathcal{B}\circ\mathcal{T}$ assigns the same set of
tilts $\mathbf{t}_{j}$ to every output (hence preserving the shape of the
blur).

In the case of $\mathcal{T}\circ\mathcal{B}$, the tilt $\mathbf{t}_{i}$ is
applied to the _output_. That is, we blur the image (drawn as a circle in
Figure 2) and move the output by $\mathbf{t}_{i}$. Since each output pixel
experiences a different $\mathbf{t}_{i}$, the blur is destroyed. For
$\mathcal{B}\circ\mathcal{T}$, although the situation is more complicated
because there are $N$ tilts $\\{\mathbf{t}_{j}\;|\;j=1\ldots,N\\}$, they
perturb the location of the _input_. Therefore, while each $\mathbf{x}_{i}$
sees $N$ tilts, the $N$ tilts are common for every $\mathbf{x}_{i}$. The shape
of the blur is thus preserved.

##  IV Impact to Natural Images

Although the two operators $\mathcal{T}\circ\mathcal{B}$ and
$\mathcal{B}\circ\mathcal{T}$ are theoretically different, their impacts to
real images are less so. This is particularly true when the underlying image
is a natural image.

To see why this is the case, rewrite (12) and (14) as

$\displaystyle I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{x}_{i})$
$\displaystyle=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{u}_{j})J(\mathbf{u}_{j}-\mathbf{t}_{i}),$
$\displaystyle I_{\mathcal{B}\circ\mathcal{T}}(\mathbf{x}_{i})$
$\displaystyle=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{u}_{j})J(\mathbf{u}_{j}-\mathbf{t}_{j}).$

Then, by approximating $J(\mathbf{u}_{j}-\mathbf{t}_{i})$ and
$J(\mathbf{u}_{j}-\mathbf{t}_{j})$ to the first order, the pointwise
difference between $I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{x}_{i})$ and
$I_{\mathcal{B}\circ\mathcal{T}}(\mathbf{x}_{i})$ can be evaluated as

$\displaystyle
I_{\mathcal{T}\circ\mathcal{B}}(\mathbf{x}_{i})-I_{\mathcal{B}\circ\mathcal{T}}(\mathbf{x}_{i})$
$\displaystyle\qquad=\sum_{j=1}^{N}g(\mathbf{x}_{i}-\mathbf{u}_{j})\Big{[}J(\mathbf{u}_{j}-\mathbf{t}_{i})-J(\mathbf{u}_{j}-\mathbf{t}_{j})\Big{]}$
$\displaystyle\qquad\approx\sum_{j=1}^{N}\underset{\text{convolution}}{\underbrace{g(\mathbf{x}_{i}-\mathbf{u}_{j})}}\;\;\;\underset{\text{distorted
image gradient}}{\underbrace{\underset{\text{image
gradient}}{\underbrace{\nabla
J(\mathbf{u}_{j})^{T}}}\;\;\;\underset{\text{random
tilt}}{\underbrace{(\mathbf{t}_{i}-\mathbf{t}_{j})}}}}.$ (15)

An intuitive argument here is that $\mathbf{t}_{i}-\mathbf{t}_{j}$ is the
difference between two tilt vectors. Since each tilt is a zero-mean Gaussian
random vector, the difference remains a zero-mean Gaussian random vector.
Although they are not white Gaussian, they are nevertheless _noise_. If there
is a large ensemble average of these noise vectors, the result will be close
to zero.

So, where does the average comes from? There is a convolution by
$g(\mathbf{x}-\mathbf{u})$. If the support of this blur kernel is large, then
many of the noise vectors will be added and this will result in a small value.
However, for an image with a large field of view, the relative size of the
blur kernel $g$ is usually not big (at most $30\times 30$ for a $256\times
256$ image). So, there must be another source that makes the error small.

The main reason why natural images tend to show a less difference between
$\mathcal{T}\circ\mathcal{B}$ and $\mathcal{B}\circ\mathcal{T}$ is that the
image gradient $\nabla J(\mathbf{u}_{j})$ is typically _sparse_. For most
parts, the gradient is zero except for edges. (Textures are less of a problem
because they will be smoothed by the blur.) When $\nabla J(\mathbf{u}_{j})$ is
multiplied with the noise vector $\mathbf{t}_{i}-\mathbf{t}_{j}$, the result
is an edge map with noise multiplied to every pixel. Convolving $\nabla
J(\mathbf{u}_{j})^{T}(\mathbf{t}_{i}-\mathbf{t}_{j})$ with a blur kernel $g$
will further smooth out the variations. Figure 3 shows a typical example with
some standard optical configurations. The difference is not noticeable.

|
---|---
(a) $\mathcal{T}\circ\mathcal{B}$ | (b) $\mathcal{B}\circ\mathcal{T}$
Figure 3: Simulated turbulence using $\mathcal{T}\circ\mathcal{B}$ and
$\mathcal{B}\circ\mathcal{T}$. The optical parameters are as follows: Aperture
diameter = 0.2034m, wavelength = 0.525$\mu$m, optical path = 7km,
$C_{n}^{2}=5\times 10^{-6}$m-2/3, focal length = 1.2m.

##  V Recommendations

This paper illustrated the validity of the tilt-then-blur model and issues of
the blur-then-tilt model. The recommendation here is that when describing the
imaging through turbulence problem, statements such as

“ _… The image formation of atmospheric turbulence follows the equation
$I(\mathbf{x})=\mathcal{T}(\mathcal{B}(J(\mathbf{x})))$ … _”

should be avoided because the statement is technically wrong. Instead, it is
more appropriate to state that the image formation model is
$I(\mathbf{x})=\mathcal{B}(\mathcal{T}(J(\mathbf{x})))$ and comment that for
natural images it can be approximated by
$I(\mathbf{x})=\mathcal{T}(\mathcal{B}(J(\mathbf{x})))$.

From a simulation point of view, the computational complexity of implementing
$\mathcal{T}\circ\mathcal{B}$ and $\mathcal{B}\circ\mathcal{T}$ are identical
if one chooses the Zernike-based multi-aperture model [28]. Since
$\mathcal{B}\circ\mathcal{T}$ is the correct model, the recommendation is that
it should always be used instead of $\mathcal{T}\circ\mathcal{B}$.

As far as solving the inverse problem is concerned, however, the freedom is a
lot bigger. One can choose to recover $\mathcal{T}$ first, or one can choose
to recover $\mathcal{B}$ first, or simultaneously. Speaking of the author’s
own experience, recovering $\mathcal{B}$ is less recommended because it is
spatially varying. Therefore, even though the correct forward model is
$I(\mathbf{x})=\mathcal{B}(\mathcal{T}(J(\mathbf{x})))$ so that the proper
inversion is
$J(\mathbf{x})=\mathcal{T}^{-1}(\mathcal{B}^{-1}(I(\mathbf{x})))$, it might be
easier to estimate $\mathcal{T}$ first. In fact, many classical methods such
as the lucky imaging techniques are based on compensating for $\mathcal{T}$.
In any case, the author hopes that the confusion between blur-then-tilt and
tilt-then-blur is settled.

## References

  * [1] N. Anantrasirichai, A. Achim, N. G. Kingsbury, and D. R. Bull,  “Atmospheric turbulence mitigation using complex wavelet-based fusion,”  IEEE Transactions on Image Processing, vol. 22, no. 6, pp. 2398–2408, Jun. 2013.
  * [2] Z. Mao, N. Chimitt, and S. H. Chan,  “Image reconstruction of static and dynamic scenes through anisoplanatic turbulence,”  IEEE Transactions on Computational Imaging, vol. 6, pp. 1415–1428, Oct. 2020.
  * [3] N. Li, S. Thapa, C. Whyte, A. W. Reed, S. Jayasuriya, and J. Ye,  “Unsupervised non-rigid image distortion removal via grid deformation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 2522–2532.
  * [4] D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai,  “Neutralizing the impact of atmospheric turbulence on complex scene imaging via deep learning,”  Nature Machine Intelligence, vol. 3, pp. 876–884, 2021.
  * [5] B. Y. Feng, M. Xie, and C. A. Metzler,  “TurbuGAN: An adversarial learning approach to spatially-varying multiframe blind deconvolution with applications to imaging through turbulence,” 2022,  Available online: https://arxiv.org/pdf/2203.06764.pdf. Accessed 6/30/2022.
  * [6] J. W. Goodman,  Introduction to Fourier Optics,  Roberts and Company, Englewood, Colorado, 3 edition, 2005.
  * [7] J. W. Goodman,  Statistical Optics,  John Wiley and Sons Inc., Hoboken, New Jersey, 2 edition, 2015.
  * [8] M. C. Roggemann and B. M. Welsh,  Imaging through Atmospheric Turbulence,  Laser & Optical Science & Technology. Taylor & Francis, 1996.
  * [9] J. D. Schmidt,  Numerical simulation of optical wave propagation: With examples in MATLAB,  SPIE Press, Jan. 2010.
  * [10] V. I. Tatarski,  Wave Propagation in a Turbulent Medium,  New York: Dover Publications, 1961.
  * [11] D. L. Fried,  “Statistics of a geometric representation of wavefront distortion,”  Journal of the Optical Society of America, vol. 55, no. 11, pp. 1427–1435, Nov. 1965.
  * [12] D. L. Fried,  “Optical resolution through a randomly inhomogeneous medium for very long and very short exposures,”  Journal of Optical Society of America, vol. 56, no. 10, pp. 1372–1379, 1966.
  * [13] M. Shimizu, S. Yoshimura, M. Tanaka, and M. Okutomi,  “Super-resolution from image sequence under influence of hot-air optical turbulence,”  in Proc. IEEE Intl’ Conf. Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1–8.
  * [14] K. R. Leonard, J. Howe, and D. E. Oxford,  “Simulation of atmospheric turbulence effects and mitigation algorithms on stand-off automatic facial recognition,”  in Proc. SPIE 8546, Optics and Photonics for Counterterrorism, Crime Fighting, and Defence VIII, Oct. 2012, pp. 1–18.
  * [15] A. Schwartzman, M. Alterman, R. Zamir, and Y. Y. Schechner,  “Turbulence-indueced 2D correlated image distortion,”  in Proc. International Conference on Computational Photography, 2017, pp. 1–12.
  * [16] X. Zhu and P. Milanfar,  “Removing atmospheric turbulence via space-invariant deconvolution,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 157–170, Jan. 2013.
  * [17] Y. Mao and J. Gilles,  “Non rigid geometric distortions correction - application to atmospheric turbulence stabilization,”  Inverse Problems and Imaging, vol. 3, pp. 531–546, 2012.
  * [18] D. H. Frakes, J. W. Monaco, and M. J. T. Smith,  “Suppression of atmospheric turbulence in video using an adaptive control grid interpolation approach,”  in Proc. IEEE Intl’ Conf. Acoustics, Speech, and Signal Processing (ICASSP), 2001, pp. 1881–1884.
  * [19] A. Shteinman S. Gepshtein and B. Fishbain,  “Restoration of atmospheric turbulent video containing real motion using rank filtering and elastic image registration,”  in Proc. European Signal Processing Conference, Sep. 2004.
  * [20] Y. Lou, S. Ha Kang, S. Soatto, and A. Bertozzi,  “Video stabilization of atmospheric turbulence distortion,”  Inverse Problems and Imaging, vol. 7, no. 3, pp. 839–861, Aug. 2013\.
  * [21] C. P. Lau, H. Souri, and R. Chellappa,  “Atfacegan: Single face semantic aware image restoration and recognition from atmospheric turbulence,”  IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 3, no. 2, pp. 240–251, Feb. 2021.
  * [22] N. G. Nair and V.M. Patel,  “Confidence guided network for atmospheric turbulence mitigation,”  in Proc. IEEE Intl. Conf. Image Processing (ICIP), 2021, pp. 1359–1363.
  * [23] R. Yasarla and V. M. Patel,  “CNN-Based restoration of a single face image degraded by atmospheric turbulence,”  IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 4, no. 2, pp. 222–233, 2022.
  * [24] C. P. Lau, Y. H. Lai, and L. M. Lui,  “Restoration of atmospheric turbulence-distorted images via RPCA and quasiconformal maps,”  Inverse Problems, Mar. 2019.
  * [25] C. P. Lau and L. M. Lui,  “Subsampled turbulence removal network,”  Mathematics, Computation and Geometry of Data, vol. 1, no. 1, pp. 1–33, 2021.
  * [26] J. P. Bos and M. C. Roggemann,  “Technique for simulating anisoplanatic image formation over long horizontal paths,”  Optical Engineering, vol. 51, no. 10, pp. 1 – 9, 2012.
  * [27] R. C. Hardie, J. D. Power, D. A. LeMaster, D. R. Droege, S. Gladysz, and S. Bose-Pillai,  “Simulation of anisoplanatic imaging through optical turbulence using numerical wave propagation with new validation analysis,”  Optical Engineering, vol. 56, no. 7, pp. 1 – 16, 2017.
  * [28] N. Chimitt and S. H. Chan,  “Simulating anisoplanatic turbulence by sampling intermodal and spatially correlated Zernike coefficients,”  Optical Engineering, vol. 59, no. 8, pp. 1 – 26, 2020.
  * [29] Z. Mao, N. Chimitt, and S. H. Chan,  “Accelerating atmospheric turbulence simulation via learned phase-to-space transform,”  in Proc. IEEE/CVF International Conference on Computer Vision (ICCV), October 2021, pp. 14759–14768.
  * [30] R. J. Noll,  “Zernike polynomials and atmospheric turbulence,”  Journal of Optical Society of America, vol. 66, no. 3, pp. 207–211, Mar. 1976.
  * [31] G. A. Chanan,  “Calculation of wave-front tilt correlations associated with atmospheric turbulence,”  Journal of Optical Society of America A, vol. 9, no. 2, pp. 298–301, Feb. 1992.
  * [32] N. Takato and I. Yamaguchi,  “Spatial correlation of Zernike phase-expansion coefficients for atmospheric turbulence with finite outer scale,”  Journal of Optical Society of America A, vol. 12, no. 5, pp. 958–963, May 1995.