1.4pt

# Non-adversarial Robustness of Deep Learning Methods for Computer Vision

Gorana Gojić, Vladimir Vincan, Ognjen Kundačina, Dragiša Mišković and Dinu
Dragan Gorana Gojić is with the The Institute of Artificial Intelligence
Research and Development of Serbia, 1 Fruškogorska, 21000 Novi Sad, Serbia
(e-mail: gorana.gojic@ivi.ac.rs), ORCID ID
(https://orcid.org/0000-0002-6100-5871) Vladimir Vincan is with the The
Institute of Artificial Intelligence Research and Development of Serbia, 1
Fruškogorska, 21000 Novi Sad, Serbia (e-mail: vladimir.vincan@ivi.ac.rs),
ORCID ID (https://orcid.org/0000-0001-8356-8133) Ognjen Kundačina is with the
The Institute of Artificial Intelligence Research and Development of Serbia, 1
Fruškogorska, 21000 Novi Sad, Serbia (e-mail: ognjen.kundacina@ivi.ac.rs),
ORCID ID (https://orcid.org/0000-0003-0198-3363) Dragiša Mišković is with the
The Institute of Artificial Intelligence Research and Development of Serbia, 1
Fruškogorska, 21000 Novi Sad, Serbia (e-mail: dragisa.miskovic@ivi.ac.rs),
ORCID ID (https://orcid.org/0000-0002-0455-9552) Dinu Dragan is with the The
Faculty of Technical Sciences, University of Novi Sad, 6 Trg Dositeja
Obradovića, 21000 Novi Sad, Serbia (e-mail<EMAIL_ADDRESS>
###### Abstract

Non-adversarial robustness, also known as natural robustness, is a property of
deep learning models that enables them to maintain performance even when faced
with distribution shifts caused by natural variations in data. However,
achieving this property is challenging because it is difficult to predict in
advance the types of distribution shifts that may occur. To address this
challenge, researchers have proposed various approaches, some of which
anticipate potential distribution shifts, while others utilize knowledge about
the shifts that have already occurred to enhance model generalizability. In
this paper, we present a brief overview of the most recent techniques for
improving the robustness of computer vision methods, as well as a summary of
commonly used robustness benchmark datasets for evaluating the model’s
performance under data distribution shifts. Finally, we examine the strengths
and limitations of the approaches reviewed and identify general trends in deep
learning robustness improvement for computer vision.

###### Index Terms:

non-adversarial, natural, out-of-distribution, robustness, domain adaptation,
domain generalization, deep learning, computer vision

##  I Introduction

Robustness is an important property of deep learning models. It refers to the
model’s ability to produce expected outputs in cases when input data differs
from the data the model has been trained on [1]. Literature distinguishes
between adversarial and non-adversarial robustness also known as natural
robustness. Adversarial robustness refers to the model’s ability to accurately
classify input data that has been intentionally corrupted in a way that aims
to fool the model into making an incorrect prediction with high confidence.
This type of robustness relies on exploiting model and data properties [2].
While adversarial robustness prevents the model malfunction under intentional
attacks [2], non-adversarial robustness preserves the model performance under
naturally-induced data transformations such as environmental and sensory
transformations [1]. Natural robustness is a desirable model property, as
deployed models often encounter test data that differs from training data. For
example, a model trained to recognize traffic signs on images collected in an
area where rarely snows might generate an unpredictable output for a test
image displaying the same sign under severe snowing conditions [2].
Differences in training and test datasets stem from distribution shifts that
are common in practice due to the changing nature of data distributions, which
can be both temporal or non-temporal (i.e., changes across locations, camera
choice, image artifacts) [3]. In a standardized machine learning pipeline, a
training dataset is collected by sampling from theoretically infinite, high-
dimensional data space with a priori unknown distribution $\mathcal{D}$. A set
of drawn samples dictates a source distribution of a training dataset, denoted
with an orange curve in Fig. 1, that the model learns through training.
Trained model inferences on test data having a target distribution, denoted by
a blue curve in Fig. 1, shifted relative to the source distribution.
Identifying the causes of distribution shift can be straightforward in some
cases, such as in [4], where a model trained on sharp images was tested on
blurry images that were not present in the training set. However, in many
cases, the causes for distribution shift are not obvious and cannot be easily
identified [5]. In [5] the authors acquire new test datasets for CIFAR-10 [6]
and ImageNet [7] following dataset acquisition guidelines and observe
significantly decreased accuracy on a broad range of models when compared to
original datasets. Assessing the causes of distribution shifts accurately,
especially in high-dimensional data, poses a challenge to training models that
can generalize well to an arbitrary sample from distribution $\mathcal{D}$
[8].

$\mathcal{D}_{t}$$\mathcal{D}_{s}$$\mathcal{D}$training sampletest sample
Figure 1: Distribution shift illustration. A priori unknown real-world data
distribution $\mathcal{D}$ (denoted by dashed line) with source
$\mathcal{D}_{s}$ and target distribution $\mathcal{D}_{t}$ (denoted with
orange and blue solid lines) being shifted relative to $\mathcal{D}_{s}$.

When a model performs well on instances from a different distribution compared
the the one it was trained on, it demonstrates out-of-distribution (OOD)
robustness. If the model performs consistently well on instances from the same
distribution, it demonstrates in-distribution (ID) robustness, ensuring stable
performance on unseen data from the same distribution. While ID robustness
does not explore the model’s generalization under distribution shifts, it is
used as a standard method to detect overfitting and underfitting during the
training. It is also used as a control metric for methods boosting OOD
generalization, since it has been shown that some of the methods can increase
OOD while simultaneously hurt model performance ID [9].

In this paper, we overview recent advancements in non-adversarial robustness,
or shorter robustness, as we will refer to in this paper. While there are many
papers addressing well-explored adversarial robustness [10, 11, 12], non-
adversarial robustness has received significantly less attention. Robustness
improvement techniques have been proposed and discussed in multiple papers in
the context of proposing a novel robustness benchmark dataset [4], robustness
method [13, 14, 15], or both [16]. However, we were unable to find a
systematic overview of these methods that could serve as an entry point to the
non-adversarial robustness research field. Thus, this paper gives a broad
overview of robustness improvement methods based on domain adaptation and
domain generalization approaches. We discuss the advantages and disadvantages
of the approaches and complement the study with an overview of publicly
available robustness benchmark datasets in computer vision.

The paper is organized as follows. Section II gives a brief overview of the
main deep learning principles that might help in following the content of
Sections III and IV where we discuss major robustness improvement
methodologies in domain generalization and domain adaptation fields. In
Section V we list major publicly available robustness benchmarking datasets,
discuss the findings in Section VI, and conclude in Section VII.

##  II Fundamentals of Deep Learning

The development of deep learning was motivated by the failure of traditional
machine algorithms to adequately solve complex tasks such as speech or object
recognition. These tasks rely on processing high-dimensional data with an
immense input data space. With just a subset of the data points from the
entire space, traditional machine learning algorithms could not provide good
generalization on unseen data points, since they rely on the similarity
between the new data points and those they have been previously learned on
[8]. To address this issue, deep learning algorithms are designed as complex
functions that use sparsely filled input data space to learn how to generalize
by incorporating knowledge about the underlying data distribution. Here we
introduce the basic concepts of feed-forward deep neural networks (DNNs) since
the majority of literature studying robustness relies on DNNs. We explain the
fully-connected neural network (FCNN) that serves as a base for other DNN
specializations, such as convolutional neural networks (CNNs). Since CNNs are
widely recognized as the primary architecture for computer vision tasks, we
also provide a brief explanation of their structure.

###  II-A Fully-connected neural networks

The goal of DNN is to learn a function approximation $y^{\prime}=f(x;\theta)$
that maps the input $x$ to the prediction $y^{\prime}$ ideally equal to
expected output $y$ by fitting the function parameters $\theta$ on some input
dataset $D=\\{(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{k},y_{k})\\}$ with $k$
data points. The learning is performed by computational units called neurons
with the general architecture shown in Fig. 2. Each neuron can receive
multiple inputs $x_{i}$ weighted by a set of learnable parameters $w_{i}$ and
learnable bias $b$ to produce a single scalar value representing the neuron’s
activation. Since neurons are inherently linear, a nonlinear activation
function is applied to the scalar to support DNNs in learning nonlinear
transformations. The activation of one neuron is propagated to succeeding
neurons to facilitate the learning of complex feature representations from the
data [8].

$x_{1.0}$$x_{2.0}$$x_{0.0}$$\sum$white$w_{1}.0$white$w_{2}.0$white$w_{0}.0$+bout$\vdots$$\begin{aligned}
&activation(\sum_{i=1}^{n}x_{i}w_{i}+b)\end{aligned}$ Figure 2: The structure
of a neuron located in the hidden layer of a FCNN.

Neurons are organized into layers as shown in Fig. 3. The number of layers
determines the depth of the network, and it is characteristic of DNNs to have
many layers [17, 18, 19, 20]. The first layer in the network is the input
layer, which serves as data entry point. It is followed by hidden layers that
usually contain many neurons simultaneously calculating their activations
based on the output of the preceding layer. The final, output layer is
responsible for generating DNN prediction for a given task.

$\mathcal{L}_{in}$$\mathcal{L}_{hidden}$$\mathcal{L}_{out}$∈$n_{1}$$n_{0}$
Figure 3: A FCNN consisting of the input layer $\mathcal{L}_{in}$, the hidden
layer $\mathcal{L}_{hidden}$, and the output layer $\mathcal{L}_{out}$.

The process of fitting DNN parameters on the training dataset $D_{train}$ is
called training. The scope of the training pertains to the adjustable weights
$\mathbf{W}$ and biases $\mathbf{b}$ attributed to all trainable neurons in
the network, where the weight vector $w_{i}^{j}$ and bias $b_{i}^{j}$
correspond to a neuron $i\in[1,n]$ located in layer $j\in[1,m]$. The training
is formulated as an optimization problem over a parameter set $\theta$
consisting of $\mathbf{W}$ and $\mathbf{b}$ that minimizes a loss function $L$
serving as an optimization objective function. It is typically performed
following a two-pass back-propagation algorithm [21]. In the first, feed-
forward pass, the loss value is calculated to estimate the error between DNN
predictions and desired outputs for training samples from $D_{train}$. The
gradients over $\theta$ are then calculated to minimize the loss using some
variation of gradient-based algorithms, i.e., standard stochastic gradient
descent (SGD), SGD with momentum, Adagrad [22], RMSProp [23], or Adam [24].
Next, gradients are used to update $\theta$. The process is iteratively
repeated until the stop condition is met, i.e., exceeded number of iterations
or gradient descent learning rate being lower than the predefined threshold
value. Hyperparameters are the values used to control the training process and
define DNN architecture. They can be adjusted manually or automatically [25,
26] to determine the optimal configuration for a certain task. In traditional
DNN they can be immutable (e.g., a choice of a loss function, number of DNN
layers) or mutable (e.g., learning rate in gradient descent). In meta-learning
paradigm, some immutable hyperparameters can be treated as mutable and learned
by another algorithm [26].

In cases when DNN is not trained on extremely large datasets, it is advisable
to introduce regularization to prevent the network from overfitting [8]. The
overfitted model has a poor ID and OOD robustness since it has not learned
patterns in data, but has memorized training data instead. Regularization can
be imposed on the training data, neurons, and training algorithm. Data
augmentation regularizes the network by enlarging the training dataset and
diversifying source distribution (see Section IV for details). Dropout [27]
prevents the network from favoring certain neurons in decision-making by
randomly eliminating their activations during the training. Regularization
strategies, such as L1 and L2 regularization [8], constrain loss function by
imposing restrictions on weight values. Another standard regularization
strategy is to stop training when a training error is consistently decreasing
while the validation error calculated on a control group of unseen data
samples is consistently increasing. The output of a training stage is a DNN
model evaluating function $y^{\prime}$ with fixed $\theta$. The model is used
to generate predictions for unseen data samples that come from the test
dataset $D_{test}$ in a process called inference.

###  II-B Convolutional neural networks

Convolutional neural networks (CNNs) [28] are a type of DNNs designed to
handle data with a topological grid structure, such as images. While the input
of the FCNN neuron is densely connected to the outputs of all neurons from the
previous layer, the CNN neuron is sparsely connected to activations in the
previous layer. Thus CNNs exhibit locality inductive bias, meaning that by
design CNN neurons extract features from spatially close grid elements in
input data. The bias is motivated by the assumption that meaningful features
in the grid are assembled of spatially close pixels [8]. Each CNN layer learns
sets of trainable kernels which are weights shared among the neurons in the
layers. Weight sharing is another inductive bias in CNNs that enables them to
detect similar features in a translation-invariant manner. Learning features
in subsequent layers enables CNNs to learn low-level features from input data,
which are then used to gradually assemble more complex features.

CNNs have been successfully utilized in many tasks such as object detection
[17, 29], classification [17, 18], [17], semantic segmentation [30, 29],
instance segmentation [31], and representation learning [32]. Nowadays modern
CNN architectures have many stacked convolutional layers that may be followed
by fully connected layers [17, 18, 29] or can act as feature extractors [32].
It is important to note that vision transformers [33] have emerged recently as
a successful alternative to CNNs in image processing. However, due to review
limitations, we omit them from the study.

##  III Domain Generalization Techniques

Domain generalization is a machine learning field that aims to train the model
on source distribution data so that it can generalize well to arbitrary target
distribution. The unexpected nature of target distribution makes data
generalization a challenging task. One approach to address the task is to
learn feature representations that are distribution shift invariant. Further
in this section, we discuss some implementations of data generalization
approach.

###  III-A Data augmentation

One approach to anticipate distribution shifts is to increase source
distribution diversity and simultaneously enlarge the dataset by applying
transformations on data sample copies [8]. The idea behind data augmentation
is to support the model generalization by learning feature representations
invariant to data augmentations. Simple geometric image augmentations, such as
random rotation, translation, and mirroring, do not change data distribution
significantly, but can still have a regularization effect. Although simple to
implement, some of these augmentations can lead to information loss (i.e.,
rotating the image for 45$\degree$) [34]. In [15, 35] authors propose
augmentations based on erasing image patches. The underlying idea is to
increase robustness against occlusions by erasing image patches. In this way,
the network learns not to heavily rely on some specific data feature when
generating predictions. A cutout augmentation [15] implements this
conceptually similar to dropout, by setting to zero activations of input
neurons associated with randomly selected input image patch. While dropout
targets individual neurons in hidden layers, cutout zeroes out weights of
spatially connected groups of input layer neurons. The authors of [35] propose
conceptually similar approach to that in [15], but with extensive experiments
on patch size and erased pixel color. Data augmentation by erasing is intended
to improve generalization under occlusions. Although this technique may have
limited benefits when occlusions are not present in the target dataset, it can
still provide additional advantages when combined with other data augmentation
methods [35]. An approach in [36] argues that replacing image patches with
semantically unmeaningful values introduces information loss and proposes an
approach where the patch that has been cut out is replaced with a patch from
another image in the training dataset. The augmentation demonstrates OOD
robustness by extending source distribution with new training samples
generated as a combination of two existing ones, combining both raw images and
corresponding labels as initially proposed in [37]. The basic concern in
synthetic augmentation techniques is that creating semantically non-meaningful
combinations will map poorly to real-world data [38], e.g., it is improbable
that a real-world image will display a horse with a frog’s head patch.
Approaches in [13] and [16] address this issue by proposing complex, yet OOD
robust augmentations that do not significantly change augmented image
statistics. In [13] authors propose chaining many simple augmentations to
create a complex augmentation. The augmentation pipeline starts with randomly
sampling augmentations from a predefined set into $n$ disjunct subsets and
sequentially applying each augmentation subset in parallel starting from the
raw image. The pipeline outputs multiple augmented versions of the raw image
linearly combined with the raw image to obtain the final augmentation. In
[16], authors propose an image-to-image network that applies random
augmentations on network weights, producing uniquely augmented images. A style
transfer variant of generative adversarial networks (GAN) [39] proposed in
[38] has been shown to facilitate OOD robustness when used as an augmentation
technique [16].

While multiple studies emphasize the necessity of combining multiple
augmentations to achieve good generalization [15, 34, 16], it is a non-trivial
task to choose the optimal combination. Theoretical foundations of data
augmentation are yet to be explored, and the majority of discoveries in the
field are experimental [34]. At the moment, studies suggest that optimal data
augmentation choice is specific for a training dataset and task combination
[34], with inappropriate choice leading to reduced model generalization such
as in [16] for DeepFashion Remixed benchmark dataset and training instability
[4] even when applied augmentations seem to be visually almost unnoticeable.

One proposed solution to this issue is to use algorithms for automated
augmentation selection. These algorithms search the augmentation space to find
the optimal augmentation policy [40, 41, 42, 43]. The algorithms are built on
the assumption that different data features require different augmentations to
properly regularize the trained model. However, automatic augmentation may not
be fully utilized in domain generalization since target distribution is not
known in training time. Still, the approach might have benefits in supervised
domain adaptation (refer to Section IV) where model is adapted to target
distribution.

###  III-B DNN architecture-based techniques

Although data augmentation tends to be the major robustness improvement
technique, changes in DNN architecture can also be used to increase OOD
natural robustness.

Network depth. In [4] it is demonstrated that deeper versions of DenseNet [44]
and ResNeXt [45] achieve higher robustness on ImageNet-C and ImageNet-P
synthetic benchmarks when comparing to their shallower counterparts. Similar
conclusions are reached in [46] but for adversarial robustness. Although we do
not intend to review adversarial robustness techniques, we include the study
to denote that OOD robustness increases with network depth in both adversarial
and non-adversarial scenarios to support findings in [4] in absence of other
supporting non-adversarial OOD studies. The results from [38] suggest that
natural robustness to synthetic distribution shifts does not apply to real-
world distribution shifts, unless the model is trained on large and varied
datasets. As a result, the conclusions drawn in [4] only apply to synthetic
distribution shifts. However, succeeding work in [16] contradicts [38] by
achieving comparable robustness both on synthetic and real-world benchmarks.

Network architecture. It has been demonstrated that some architectures exhibit
increased natural robustness [44, 45, 47, 48] when compared to base ResNet
architecture. DenseNet and ResNeXt are shown to be more robust both on
perturbations and corruptions [4]. The increased robustness results from
improved data aggregation capabilities introduced through enriched connections
between layers. For example, in a DenseNet every convolutional layer inside a
dense block uses feature maps from all preceding convolution layers alongside
the standard feature map produced by sequential processing. According to [4],
multi-scale networks [47, 48] demonstrate corruption robustness, with no clear
benefits on perturbation robustness. Allegedly, this is due to network design
where data in different scales is affected differently by corruptions to
produce the final prediction.

Although increasing DNN depth and adapting architecture toward feature
aggregation positively influence the model’s overall robustness, using these
two techniques jointly has demonstrated the best performance.

##  IV Domain Adaptation Techniques

When deploying a DL model in real-world scenarios, data distribution shifts
are expected to occur at some point in time, causing the model’s performance
to deteriorate. Domain adaptation is a field of machine learning aiming to
modify a model trained on a source distribution to adapt to data from a target
distribution [49]. In this section we will discuss domain adaptation
implementations using transfer [50], meta [26], and few-shot learning [51].

###  IV-A Transfer learning

Transfer learning is a machine learning field researching how to utilize
knowledge learned in one task to improve another, related task. The basic idea
of transfer learning is that DNN learns feature representations gradually from
simple, task-agnostic features (e.g., lines) to complex, task-specific
features (e.g., nose or ears for face recognition task) that can be fully or
partially transferred to another problem. A common way to implement transfer
learning is to pretrain a DNN on a large and diverse dataset and then use the
first $n$ pretrained layers as an initialization for a new DNN that is then
trained on a new dataset. There are two primary transfer learning approaches:
fine-tuning and linear probing. With linear probing, the convolutional layer
weights of a DNN that act as feature extractors remain fixed, while only the
downstream task prediction layers are trained. Fine-tuning, on the other hand,
involves updating both the feature extractor and prediction layers.

Using transfer learning in domain adaptation settings is based on the
assumption that distribution invariant feature representations can be learned
from source and target distribution data [8]. Although pretraining might not
necessarily improve model performance [52] as it was previously believed [53],
it has been demonstrated that in certain settings it can increase the models
OOD robustness, both in adversarial and non-adversarial settings [54].
However, it is shown that the transfer learning efficiency on domain
adaptation is dependent on a choice of transfer learning method [9] and
similarities in source and target datasets [53]. In [9], authors use fine-
tuning and linear probing approaches to transfer learning, demonstrating that
fine-tuning leads to better ID and worse OOD performance compared to linear
probing, resulting in higher OOD and lower ID performance. Observing that
fine-tuning can distort pretrained features, the authors in [9] propose a
novel transfer learning approach that combines fine-tuning and linear probing.

The recent breakthrough in self-supervised learning (SSL) for representation
learning has inspired a community to combine transfer learning with SSL [55,
56]. Since SSL does not require explicit supervision to learn representations,
this approach can be used for both labeled and unlabeled target data. In SSL a
model learns to solve an auxiliary, pretext task to learn rich feature
representations, e.g., learning to predict image rotation [57], or distinguish
between images [58, 59, 60]. SSL approach in [55] uses unlabeled data to learn
shared representations on multiple tasks and multiple datasets. The authors
train a feature extractor using multitask learning [61] with one task being a
downstream task and other tasks being auxiliary SSL tasks (e.g., predicting
rotation and localization). All tasks are trained simultaneously using both
source and target distribution data. The approach utilizes learning on
multiple tasks and datasets to increase knowledge transfer by reducing the
distance between learned tasks [53] and training simultaneously on source and
target distribution datasets. Work in [56] extends on [55] to support test-
time training in online settings. While [55] adapts the model on target data
and uses adapted model weights further in inference, [56] starts from the
model trained solely on source distribution and adapts its weights online as
target distribution data arrives.

###  IV-B Other techniques

Here we mention other approaches to domain adaptation that have not yet
received as much attention as the simplest, transfer learning approach. Our
motivation here is not to present a comprehensive overview of existing
methods, but to provide the reader with a starting point for further research.

Meta-learning [26], or learning to learn, is an approach where the meta-
learner algorithm learns parts of the base-learner algorithm that performs a
downstream task. Some examples include learning an optimization algorithm [62]
and base-learner parameter initialization [63]. A broad idea behind meta-
learning is to support quick and efficient adaptation of the base-learner
through meta-training across different tasks and domains. In [64] the authors
propose a meta-learning framework to learn the initial conditions of the
existing domain adaptation algorithms relying on the gradient descent method.
The suggested framework improves the performance of various domain adaptation
techniques in multi-source and semi-supervised scenarios for the target
distribution.

Few-Shot Learning. In few-shot learning (FSL)[51] the task is to train a
network to recognize new classes having just a few labeled instances per
class. Having a dataset with few instances per class is insufficient to train
DNN from scratch or adapt to target distribution using transfer learning.
Therefore, domain adaptation in FSL is often formulated as meta-learning
problem, where instances from target distribution are used to speed up
convergence of the base-learner. The challenge in few-shot learning domain
adaptation is to adapt to new classes that can partially overlap with base
classes [65] or can be completely disjoint [66, 67].

##  V Benchmarking Datasets

In this section, we cover benchmarking datasets used to assess model
performance under domain generalization and adaptation approaches.

Domain generalization benchmarks. There are multiple publicly available
robustness benchmarks intended to test for the natural robustness of computer
vision DNNs. Both synthetic and real-world benchmarks exist. While synthetic
benchmarks provide testing in controlled conditions under known distribution
shifts, recent work shows that demonstrating robustness on synthetic
benchmarks does not necessarily guarantee robustness on real-world
distribution shifts [38]. To alleviate the issue, real-world benchmarks have
been introduces.

ImageNet-C[4] and ImageNet-P [4] are synthetic benchmarks that decouple
robustness benchmarking to corruption and perturbation robustness
respectively. To produce both benchmarks, image transformations are applied to
original images from the ImageNet dataset. Corruptions are designed to
significantly change image statistics and enforce testing in OOD settings.
There are 15 types of corruption transformations selected from noise, blur,
weather, and digital categories, that are applied to original images with five
severity levels to control the degree of image distribution shift.
Perturbations are subtle transformations of original images sampled from the
same categories as corruptions, but harder to perceive visually when compared
to corruptions. The motivation behind the perturbation benchmark is to test
subtle data distribution shifts.

Collecting a comprehensive real-world robustness benchmark with systematic
distribution shifts can be challenging compared to synthetic benchmarks. The
nature of the distribution shifts is difficult to determine due to multiple
dimensions of variation that can occur in real-world data simultaneously. For
example, an image displaying the same object can be captured with different
cameras, from different viewpoints, in different locations, showing the object
in various weather conditions. However, the issue raised in [38] emphasizes
the importance of real-world benchmarking datasets. Here we mention ImageNetV2
[5], ImageNet-Renditions [16], StreetView StoreFronts [16], DeepFashionRemixed
[16], and Real Blurry Images [16].

ImageNetV2 [5] is created by replicating the original ImageNet data collection
procces. For this dataset, distribution shifts cannot be clearly identified,
but study [5] has shown degraded performance for a range of classifiers
trained on ImageNet and tested on ImageNetV2, indicating clear existence of
natural distribution shifts in the ImageNetV2 dataset. Similar is true for
ImageNet-Renditions (ImageNet-R) [16] which contains image renditions such as
paintings, sculptures, and embroidery for the ImageNet classes. The dataset
contains images of significantly different textures and local image
statistics, shifting ImageNet-R distribution relative to ImageNet. StreetView
StoreFronts (SVSF) [16] dataset contains business storefront images. The
dataset varies in location, the year the image is captured, and camera
properties introducing sensor-induced distribution shifts. Close to the idea
of quantifying distribution shifts as introduced in synthetic benchmarks,
real-world benchmark DeepFashion Remixed (DFR) [16] uses metadata from
DeepFashion2 [68] to make systematic changes in object occlusion, zoom,
orientation, and scale. Both SVSF and DFR benchmarks provide a way to control
distribution shifts by altering dataset parameters (e.g., fixing location for
SVSF or varying zoom for DFR and fixing other parameters) and testing DNN OOD
generalization with more granularity. Real Blurry Images [16] benchmark
contains naturally blurry images. The dataset is proposed as a real-world
counterpart of ImageNet-C and ImageNet-P datasets that test model robustness
on synthetically generated blurry images.

Domain adaptation benchmarks. These benchmarks are intended to test how well a
given model adapts to changes in data distribution. Thus, domain adaptation
benchmarks can be assembled from any two datasets suitable for a particular
downstream task. The first in a pair, or a training dataset is of the source
distribution, and the second dataset is of the target distribution. While in
this setup source and target distributions must be different, literature shows
they also should retain some similarities for data distribution shift not to
be too extreme. Often, this means having a shared label space [9, 55, 69].
While domain adaptation benchmarks can be assembled in this manner for any
downstream task, in this paper we further discuss domain adaptation benchmarks
for object recognition task, as it is a well-established field in deep
learning for computer vision.

Some of the datasets used to train object recognition DNNs include MNIST [28],
MNIST-M [70], ImageNet-1K [71], SVHN [72], USPS [73], CIFAR-10 [74], and
STL-10 [75]. Some properties of these datasets depicting possible distribution
shifts are given in Table I. These datasets have been used as domain
adaptation benchmarks in [55] by assembling pairs MNIST+MNIST-M, MNIST+SVHN,
SVHN+MNIST, USPS+MNIST, CIFAR-10+STL-10, STL-10+CIFAR-10, while in [9]
ImageNet-1K is paired with ImageNet-R [16], ImageNet-A [76], and ImageNetV2
[5] for OOD evaluation. In cases where datasets do not share all labels,
differing labels are usually discarded from both datasets [69]. There are also
specialized benchmarks containing images from multiple domains for all classes
in a dataset, with DomainNet [77] being a standard benchmark in the field
representing each class in six different domains (e.g., clipart, sketch).
Another approach to benchmark domain adaptation is to use domain
generalization datasets [56]. In few-shot learning (FSL) setup, where label
spaces of source and target domains differ, FSL leverages knowledge from the
source domain to learn a downstream task on the target domain containing new
labels. In [66] authors propose a domain adaptation benchmark for non-
overlapping label spaces in FSL.

TABLE I: Properties of object recognition datasets Dataset | Classes | Description
---|---|---
MNIST [28] | 10 (one class for each digit) | Counts 70000 grayscale images of handwritten digits of size $28\times 28$ pixels.
MNIST-M [70] | 10 (one class for each digit) |  MNIST dataset extension for robustness evaluation with various transformations applied to MNIST images, e.g., color jittering, rotation, and scaling. Contains 70000 color images of size $28\times 28$ pixels.
ImageNet-1K [71] | 1000 (e.g., bird, cat, car). |  Contains 1.2 million color images of varying resolution showing objects from 1000 common object categories.
SVHN [72] | 10 (one class for each digit) | Contains 60000 color images of street view house numbers. Image size and quality vary.
USPS [73] | 10 (one class for each digit) |  Count 9298 grayscale images with handwritten digits of size $16\times 16$ pixels. Image quality and orientations vary.
CIFAR-10 [75] | 10 (e.g., airplane, bird, cat) |  Contains 60000 color images of size $32\times 32$ of varying quality showing vehicles, animals, and other objects.
STL-10 [74] | 10 (e.g., deer, fog, airplane) |  Contains 150000 color images of $96\times 96$ of varying quality showing images with complex backgrounds, with 50000 images having labels.

##  VI Discussion

Domain generalization. We distilled two common approaches for domain
generalization, namely data augmentation, and DNN architectural modifications,
including DNN depth and building blocks. We observe that there is no universal
method to be applied in all scenarios, but that different methods are suitable
for different situations. The literature on data augmentation agrees that the
right choice of data augmentation can boost natural OOD robustness, but that
multiple augmentations can boost robustness even more [15, 34, 16]. However,
the challenge remains in identifying the optimal augmentation set, which
largely depends on the downstream task and dataset properties [34]. On the
other hand, increasing network depth [4] and incorporating architectural
building blocks that promote feature aggregation have been shown to enhance
model robustness [44, 45]. Increasing network depth depends on large and
diverse training datasets to regularize the model, so this strategy is often
combined with data augmentation.

Domain adaptation is often achieved through transfer learning, though other
methods such as FSL and meta-learning have also been utilized for this
purpose. While transfer learning can enhance robustness on the target
distribution, the method of knowledge transfer should be carefully selected
based on the properties of source and target distributions. Both experimental
and theoretical studies have shown that an inappropriate method can distort
pretrained features learned from source distribution, exacerbating both ID and
OOD generalization. Among supervised and SSL approaches to domain adaptation,
SSL has demonstrated its superiority [9]. As SSL continues to evolve toward
new approaches for efficient representation learning, it can be expected that
more domain adaptation methods will incorporate SSL. In particular,
contrastive learning has demonstrated superior performance in representation
learning field [58, 60]. Learning better feature extractors can improve the
linear probing approach to transfer learning or any hybrid method that
includes linear probing. While a limited review of few-shot and meta-learning
approaches suggests that they can improve robustness in domain adaptation,
they may still be outperformed by simple transductive fine-tuning [78].

##  VII Conclusions

Since DL models are increasingly used in practice, a growing body of research
focuses on minimizing the effect of data distribution shifts. In this review
we have covered recent progress in domain generalization and domain adaptation
techniques where the first approach aims to prepare the model for unknown data
distribution shifts in training time and the second to adapt the model for
known distribution shift in inference time. We also provide information on
commonly used benchmarking datasets to provide the reader with better
understanding of the reviewed methods.

## Acknowledgment

This paper has received funding from the European Union’s Horizon 2020
research and innovation programme under Grant Agreement number 85696. The
paper has also been supported by the Ministry of Science, Technological
Development, and Innovation of Republic of Serbia through project no.
451-03-47/2023-01/200156 “Innovative scientific and artistic research from the
FTS domain”.

## References

  * [1] N. Drenkow, N. Sani, I. Shpitser, and M. Unberath, “Robustness in Deep Learning for Computer Vision: Mind the gap?” Dec. 2021, arXiv:2112.00639 [cs]. [Online]. Available: http://arxiv.org/abs/2112.00639
  * [2] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [3] M. Sugiyama and M. Kawanabe, _Machine learning in non-stationary environments: Introduction to covariate shift adaptation_. MIT press, 2012.
  * [4] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” in _Proc. ICLR_ , 2019.
  * [5] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?” in _ICML_. PMLR, 2019, pp. 5389–5400.
  * [6] A. Krizhevsky, “Learning Multiple Layers of Features from Tiny Images,” Master’s thesis, University of Toronto, Canada, 2009.
  * [7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_ , 2009, pp. 248–255.
  * [8] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016, http://www.deeplearningbook.org.
  * [9] A. Kumar, A. Raghunathan, R. M. Jones, T. Ma, and P. Liang, “Fine-tuning can distort pretrained features and underperform out-of-distribution,” in _ICLR_ , 2022.
  * [10] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _ICML_. PMLR, 2020, pp. 2206–2216.
  * [11] J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in _ICML_. PMLR, 2019, pp. 1310–1320.
  * [12] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _ICLR_ , 2018. [Online]. Available: https://openreview.net/forum?id=rJzIBfZAb
  * [13] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” _arXiv preprint arXiv:1912.02781_ , 2019.
  * [14] S. Yun, D. Han, S. Chun, S. J. Oh, Y. Yoo, and J. Choe, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proc. ICCV_ , 2019, pp. 6022–6031.
  * [15] T. Devries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _CoRR_ , vol. abs/1708.04552, 2017. [Online]. Available: http://arxiv.org/abs/1708.04552
  * [16] D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer, “The many faces of robustness: A critical analysis of out-of-distribution generalization,” in _ICCV_. Los Alamitos, CA, USA: IEEE Computer Society, oct 2021, pp. 8320–8329. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.00823
  * [17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. CVPR_ , 2016, pp. 770–778.
  * [18] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proc. CVPR_ , 2017, pp. 4700–4708.
  * [19] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _ICML_. PMLR, 2019, pp. 6105–6114.
  * [20] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [21] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” _Nature_ , vol. 323, no. 6088, pp. 533–536, Oct. 1986, number: 6088 Publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/323533a0
  * [22] J. Duchi, E. Hazan, and Y. Singer, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,” _JMLR_ , vol. 12, no. 61, pp. 2121–2159, 2011. [Online]. Available: http://jmlr.org/papers/v12/duchi11a.html
  * [23] T. Tieleman and G. Hinton, “Lecture 6.5 - rmsprop: Divide the gradient by a running average of its recent magnitude,” _Coursera: Neural Networks for Machine Learning_ , 2012, https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
  * [24] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , Y. Bengio and Y. LeCun, Eds., 2015.
  * [25] J. Bergstra, D. Yamins, and D. D. Cox, “Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures,” in _Proc. ICML_ , 2013.
  * [26] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, “Meta-learning in neural networks: A survey,” _PAMI_ , vol. 44, no. 09, pp. 5149–5169, sep 2022.
  * [27] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _JMLR_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [28] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [29] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in _Proc. CVPR_ , 2014, pp. 580–587.
  * [30] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_. Springer, 2015, pp. 234–241.
  * [31] B. Cheng, A. Schwing, and A. Kirillov, “Per-pixel classification is not all you need for semantic segmentation,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 864–17 875, 2021.
  * [32] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [33] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy
  * [34] S. Yang, W. Xiao, M. Zhang, S. Guo, J. Zhao, and F. Shen, “Image data augmentation for deep learning: A survey,” _arXiv preprint arXiv:2204.08610_ , 2022.
  * [35] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” _CoRR_ , vol. abs/1708.04896, 2017. [Online]. Available: http://arxiv.org/abs/1708.04896
  * [36] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proc. ICCV_ , October 2019.
  * [37] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in _ICLR_ , 2018.
  * [38] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt, “Measuring Robustness to Natural Distribution Shifts in Image Classification,” in _NeurIPS_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 18 583–18 599.
  * [39] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in _NIPS_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014.
  * [40] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in _Proc. CVPR_ , 2019, pp. 113–123.
  * [41] S. Lim, I. Kim, T. Kim, C. Kim, and S. Kim, “Fast autoaugment,” _Adv Neural Inf Process Syst_ , vol. 32, 2019.
  * [42] D. Ho, E. Liang, X. Chen, I. Stoica, and P. Abbeel, “Population based augmentation: Efficient learning of augmentation policy schedules,” in _ICML_. PMLR, 2019, pp. 2731–2741.
  * [43] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, “Randaugment: Practical automated data augmentation with a reduced search space,” in _Proc. CVPRW_ , 2020, pp. 702–703.
  * [44] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely Connected Convolutional Networks,” Jan. 2018. [Online]. Available: http://arxiv.org/abs/1608.06993
  * [45] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proc. ICPR_ , 2017, pp. 1492–1500.
  * [46] C. Xie and A. Yuille, “Intriguing properties of adversarial training at scale,” _arXiv preprint arXiv:1906.03787_ , 2019.
  * [47] T.-W. Ke, M. Maire, and S. X. Yu, “Multigrid neural architectures,” in _Proc. CVPR_ , 2017, pp. 6665–6673.
  * [48] G. Huang, D. Chen, T. Li, F. Wu, L. van der Maaten, and K. Weinberger, “Multi-scale dense networks for resource efficient image classification,” in _ICLR_ , 2018.
  * [49] G. Csurka _et al._ , _Domain adaptation in computer vision applications_. Springer, 2017.
  * [50] S. J. Pan and Q. Yang, “A survey on transfer learning,” _IEEE Trans. Knowl. Data Eng._ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [51] Y. Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” _ACM computing surveys (csur)_ , vol. 53, no. 3, pp. 1–34, 2020.
  * [52] K. He, R. Girshick, and P. Dollár, “Rethinking imagenet pre-training,” in _Proc. ICCV_ , 2019, pp. 4918–4927.
  * [53] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?” _NIPS_ , vol. 27, 2014.
  * [54] D. Hendrycks, K. Lee, and M. Mazeika, “Using pre-training can improve model robustness and uncertainty,” in _ICML_. PMLR, 2019, pp. 2712–2721.
  * [55] Y. Sun, E. Tzeng, T. Darrell, and A. A. Efros, “Unsupervised domain adaptation through self-supervision,” _arXiv preprint arXiv:1909.11825_ , 2019.
  * [56] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt, “Test-time training with self-supervision for generalization under distribution shifts,” in _Proc. ICML_ , ser. PMLR, H. D. III and A. Singh, Eds., vol. 119\. PMLR, 13–18 Jul 2020, pp. 9229–9248.
  * [57] Z. Feng, C. Xu, and D. Tao, “Self-supervised representation learning by rotation feature decoupling,” in _Proc. CVPR_ , 2019, pp. 10 364–10 374.
  * [58] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _ICML_. PMLR, 2020, pp. 1597–1607.
  * [59] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proc. CVPR_ , 2020, pp. 9729–9738.
  * [60] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision transformers,” in _Proc. CVPR_ , 2021, pp. 9650–9660.
  * [61] R. Caruana, _Multitask learning_. Springer, 1998.
  * [62] S. Ravi and H. Larochelle, “Optimization as a model for few-shot learning,” in _ICLR_ , 2017.
  * [63] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_. PMLR, 2017, pp. 1126–1135.
  * [64] D. Li and T. Hospedales, “Online meta-learning for multi-source and semi-supervised domain adaptation,” in _ECCV_ , A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp. 382–403.
  * [65] K. Saito, D. Kim, S. Sclaroff, and K. Saenko, “Universal domain adaptation through self-supervision,” _NIPS_ , vol. 33, pp. 16 282–16 292, 2020.
  * [66] H.-Y. Tseng, H.-Y. Lee, J.-B. Huang, and M.-H. Yang, “Cross-domain few-shot classification via learned feature-wise transformation,” _arXiv preprint arXiv:2001.08735_ , 2020.
  * [67] Y. Guo, N. C. Codella, L. Karlinsky, J. V. Codella, J. R. Smith, K. Saenko, T. Rosing, and R. Feris, “A broader study of cross-domain few-shot learning,” in _ECCV_. Springer, 2020, pp. 124–141.
  * [68] Y. Ge, R. Zhang, X. Wang, X. Tang, and P. Luo, “Deepfashion2: A versatile benchmark for detection, pose estimation, segmentation and re-identification of clothing images,” in _Proc. CVPR_ , 2019, pp. 5337–5345.
  * [69] R. Shu, H. H. Bui, H. Narui, and S. Ermon, “A dirt-t approach to unsupervised domain adaptation,” _arXiv preprint arXiv:1802.08735_ , 2018.
  * [70] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” _J Mach Learn Res_ , vol. 17, no. 1, pp. 2096–2030, 2016.
  * [71] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _Int J Comput Vis_ , vol. 115, pp. 211–252, 2015.
  * [72] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  * [73] J. Hull, “A database for handwritten text recognition research,” _PAMI_ , vol. 16, no. 5, pp. 550–554, 1994.
  * [74] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [75] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in _Proc. AISTATS_ , ser. PMLR, G. Gordon, D. Dunson, and M. Dudík, Eds., vol. 15. Fort Lauderdale, FL, USA: PMLR, 11–13 Apr 2011, pp. 215–223. [Online]. Available: https://proceedings.mlr.press/v15/coates11a.html
  * [76] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” in _Proc. CVPR_ , 2021, pp. 15 262–15 271.
  * [77] X. Peng, Q. Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang, “Moment matching for multi-source domain adaptation,” in _Proc. ICCV_ , 2019, pp. 1406–1415.
  * [78] G. S. Dhillon, P. Chaudhari, A. Ravichandran, and S. Soatto, “A baseline for few-shot image classification,” in _ICLR_ , 2020. [Online]. Available: https://openreview.net/forum?id=rylXBkrYDS