# Automated Domain Discovery from Multiple Sources to Improve Zero-Shot
Generalization

Kowshik Thopalli<EMAIL_ADDRESS>
Geometric Media Lab,
Department of ECEE,
Arizona State University,
Tempe, AZ, USA
Sameeksha Katoch<EMAIL_ADDRESS>
Qualcomm
San Diego, CA, USA
Pavan Turaga<EMAIL_ADDRESS>
Geometric Media Lab,
Department of ECEE,
Arizona State University,
Tempe, AZ, USA
Jayaraman J. Thiagarajan<EMAIL_ADDRESS>
Lawrence Livermore National Laboratory
Livermore, CA, USA

###### Abstract

Domain generalization (DG) methods aim to develop models that generalize to
settings where the test distribution is different from the training data. In
this paper, we focus on the challenging problem of multi-source zero shot DG
(MDG), where labeled training data from multiple source domains is available
but with no access to data from the target domain. A wide range of solutions
have been proposed for this problem, including the state-of-the-art multi-
domain ensembling approaches. Despite these advances, the naïve ERM solution
of pooling all source data together and training a single classifier is
surprisingly effective on standard benchmarks. In this paper, we hypothesize
that, it is important to elucidate the link between pre-specified domain
labels and MDG performance, in order to explain this behavior. More
specifically, we consider two popular classes of MDG algorithms –
distributional robust optimization (DRO) and multi-domain ensembles, in order
to demonstrate how inferring custom domain groups can lead to consistent
improvements over the original domain labels that come with the dataset. To
this end, we propose (i) GroupDRO++, which incorporates an explicit clustering
step to identify custom domains in an existing DRO technique; and (ii) DReaME
, which produces effective multi-domain ensembles through implicit domain re-
labeling with a novel meta-optimization algorithm. Using empirical studies on
multiple standard benchmarks, we show that our variants consistently
outperform ERM by significant margins ($1.5\%-9\%$), and produce state-of-the-
art MDG performance. Our code can be found at
https://github.com/kowshikthopalli/DREAME

Keywords: Multi-domain generalization, distribution shifts, ERM, domain re-
labeling, ensembles, distributional robust optimization, meta learning

##  1 Introduction

Supervised machine learning models commonly rely on the assumption that the
training and testing data are independent and identically distributed
(i.i.d.). As a result, such models can fail drastically when tested on data
that is not i.i.d. (Torralba and Efros, 2011). Such drops in performance
indicate poor generalization capabilities of the models. Addressing this
fundamental challenge has become an important topic of research (Hoffman et
al., 2018; Ganin et al., 2016; Deng et al., 2018) over the last few years.
Several classes of solutions, ranging from unsupervised adaptation approaches
(Wang and Deng, 2018), new formulations for domain-invariant learning (Zhang
et al., 2020b; Arjovsky et al., 2019), data augmentation techniques (Xu et
al., 2021) and novel regularization strategies (Zhou et al., 2021), have been
proposed to improve generalization under covariate shifts. Furthermore, given
the inherent limitation of using data from a single source domain to
generalize under real-world shifts, methods that leverage multiple domains
have also emerged (Blanchard et al., 2011). Commonly referred to as zero-shot,
multi-domain generalization (ZS-MDG), this formulation assumes that labeled
data from multiple source domains is available but with no access to the
target domain.

The simplest solution to this problem is the vanilla empirical risk
minimization (ERM) (Vapnik, 1999), which minimizes an average loss computed on
data pooled together from all available source domains. The inability of this
approach to exploit statistical discrepancies between domains has motivated
the design of multi-domain learning techniques (Wang et al., 2021). However,
recently Gulrajani et al. (Gulrajani and Lopez-Paz, 2021) reported that a
powerful feature extractor coupled with effective model selection can make
ERMs highly competitive on standard benchmarks. Since then there is renewed
interest in better understanding and improving the performance of vanilla ERM.
In this context, approaches that enforce ERM-based models to be consistent
under appropriate data augmentations have become popular (Xu et al., 2021;
Robey et al., 2021). Despite their effectiveness, choosing the most
appropriate augmentation for a given dataset is challenging, and in practice,
even advanced strategies, e.g., random convolutions, can provide varying
degrees of performance gains across datasets.

In this paper, we take a different perspective for improving ZS-MDG by exploit
the important link between domain-aware MDG solutions and the domain groups in
a dataset. To this end, we first consider a recent distributional robust
optimization algorithm (GroupDRO (Sagawa* et al., 2020)), wherein we develop a
new variant GroupDRO++ that jointly infers optimal “domain” groups (through a
clustering step) and performs multi-source model training with GroupDRO. Our
analysis shows that, GroupDRO++ outperforms both the vanilla ERM and the
standard GroupDRO implementation on all benchmarks.

Next, we consider another popular class of MDG methods, multi-domain
ensembles. Existing ensembling methods (Chattopadhyay et al., 2020; Seo et
al., 2020) typically combine multiple models (that can optionally share
parameters) that are trained for each of the observed domains. However, it is
non-trivial to use an approach like GroupDRO++ in this case, since GroupDRO++
requires integration of an auxiliary clustering step (e.g., K-means) and needs
significant modification to the training process for joint optimization.
Hence, We introduce DReaME (Domain Re-labeling for Multi-Domain Ensembles),
that advances existing multi-domain ensembling techniques by implicitly
reorganizing the data samples into custom domain groupings during ensemble
construction, through a novel meta optimization algorithm. Using extensive
empirical studies on a large suite of multi-domain benchmarks, we argue that,
one can learn invariances that are the most beneficial for out-of-domain
generalization through a more meaningful re-grouping of data samples. Our
results show that, the proposed approaches lead to significant performance
gains over the ERM baseline, and more importantly, produce state-of-the-art
performance on challenging datasets.

###  1.1 Contributions and Findings

We make the following contributions and findings in regard to ZS-MDG: (i) We
hypothesize that domain groups play a critical role in determining the
performance of MDG algorithms and validate the hypothesis with two popular MDG
approaches; (ii) We introduce GroupDRO++ to infer “optimal” domain labels in a
distributional robust optimization setting; (iii) Next, we propose DReaME, a
new multi-domain ensembling method with implicit domain re-labeling. DReaME
uses a gradient-matching based re-labeling strategy that is found to be
empirically superior to other design choices; (iv) We investigate two model
selection protocols for DReaME using only the source domain validation data;
and (v) We perform extensive empirical studies with DomainBed (Gulrajani and
Lopez-Paz, 2021) and a large suite of benchmarks (OfficeHome,
Camelyon$17$-WILDS, PACS and Terra-Incognita), to demonstrate the
effectiveness of our proposed approaches.

###  1.2 Paper Organization

The rest of this paper is organized as follows. Section 2 discusses the
background on multi-domain generalization and known theoretical bounds on
generalization error in comparison to standard ERM. Next, in Section 3, we
present our hypothesis on the need for domain re-labeling. Sections 4 and 5
present our proposed algorithms that integrate automated domain discovery into
two popular MDG algorithms, GroupDRO and multi-domain ensembles, respectively.
Section 6 presents the empirical findings and a discussion on related work is
provided in Section 7. Finally, Section 8 provides concluding remarks.

##  2 Background

##### Problem Setup

Given access to $K$ labeled source domains
$\\{\mathcal{D}_{1},\ldots,\mathcal{D}_{K}\\}$ where
$\mathcal{D}_{k}=\\{({\mathbf{x}}_{i}^{k},y_{i}^{k})\\}_{i=1}^{N_{k}}\sim
P^{(k)}_{XY}$ is the $k^{th}$ domain comprising $N_{k}$ image-label pairs, the
goal is to generalize to any novel test domain $\mathcal{D}^{\dagger}$,
without requiring labeled or unlabeled examples. We consider the homogeneous
MDG setting i.e., observed and unobserved domains share the same label space.

##### Vanilla ERM

Here, the goal is to find a function $f:{\mathbf{x}}\rightarrow y$ that maps
samples ${\mathbf{x}}$ to labels $y$ by minimizing the empirical risk on the
pooled data:

$\vspace{-2pt}\frac{1}{|\mathcal{D}|}\sum_{({\mathbf{x}}_{i},y_{i})\in\mathcal{D}}\ell\left(f\left({\mathbf{x}}_{i}\right),y_{i}\right);\quad\mathcal{D}\coloneqq\bigcup_{k=1}^{K}\mathcal{D}_{k},$
(1)

where, $\ell$ is the loss function that measures predictive error, e.g.,
cross-entropy, and $|\mathcal{D}|$ denotes the size of the pooled dataset.

Since this simple baseline method does not leverage the inherent discrepancies
between the sources domains, one might expect this to be ineffective in
practice. Surprisingly, this naïve solution has been found to produce
competitive performance on standard domain generalization benchmarks
(Gulrajani and Lopez-Paz, 2021). Consequently, there is a critical need to
understand why MDG methods that leverage the knowledge about the domain groups
do not provide non-trivial gains over ERM.

##### Improving ERM by Maximizing Worst-case Performance

A major drawback of the formulation in (1) is that it treats all samples from
all domains/groups equally and thus decreases the loss in an average sense. An
insight from distributional robust optimization is that decreasing the error
on worst-group can lead to better generalization. Hence, an important class of
MDG algorithms attempts to decrease a weighted mean of the group-level (pre-
specified in the dataset) losses with an adaptive domain-specific weighting,
such that large weights are assigned to groups with higher error. Formally,
let $g_{i}=k\in\\{1,\cdots,K\\}$ denote the group to which a sample
$({\mathbf{x}}_{i},y_{i})$ belongs to, where $K$ is the total number of
groups, and $q_{k}$ is the weight assigned to a particular group. The risk now
becomes

$\frac{1}{|\mathcal{D}|}\sum_{(x_{i},y_{i},g_{i})\in\mathcal{D}}q_{g_{i}}\ell\bigg{(}f(x_{i}),y_{i}\bigg{)};\quad\mathcal{D}\coloneqq\bigcup_{k=1}^{K}\mathcal{D}_{k}.$
(2)

For example, GroupDRO (Sagawa* et al., 2020) uses an update rule for $q_{g}$
across iterations, such that a group with larger error is assigned a higher
weight.

##### Error Bounds for MDG

Before introducing our hypothesis on the crucial role that domain labels play,
we briefly discuss existing theoretical bounds for MDG in a binary
classification setting (Blanchard et al., 2011). Let us begin by defining a
hyper-distribution $\bar{P}$ on $({\mathbf{x}},y)$ from which all source
domains, $P_{XY}^{(k)},k=1,\cdots,K$, and the target domain,
$P_{XY}^{\dagger}$, are drawn from. In order to train a classifier $f$ that
generalizes to any possible target domain, one can leverage the knowledge of
domain groupings during training. In such a case, the average risk can be
estimated as follows:

$\mathcal{E}(f)\coloneqq\mathbb{E}_{P_{XY}\sim\bar{P}}\text{
}\mathbb{E}_{({\mathbf{x}},y)\sim
P_{XY}}\ell\bigg{(}f(P_{X},{\mathbf{x}}),y\bigg{)},$ (3)

where $P_{X}$ is the marginal distribution for a domain drawn from $\bar{P}$.
In practice, we use finite approximations for these expectations:

$\hat{\mathcal{E}}(f)\coloneqq\frac{1}{K}\sum_{k=1}^{K}\frac{1}{N_{k}}\sum_{i=1}^{N_{k}}\ell\bigg{(}f(\mathcal{X}_{k},{\mathbf{x}}_{i}^{k}),y_{i}^{k}\bigg{)},\mathcal{X}_{k}=\\{{\mathbf{x}}_{i}^{k}|({\mathbf{x}}_{i}^{k},y_{i}^{k})\in\mathcal{D}_{k}\\}.$
(4)

The discrepancy between $\mathcal{E}(f)$ and the estimate
$\hat{\mathcal{E}}(f)$ can be measured using an appropriate divergence in some
hypothesis space for $f$. For example, Blanchard et al. (Blanchard et al.,
2011) considered the space of $f$ to be a reproducing kernel Hilbert space
(RKHS), where the inducing kernel is of the form
$\mathrm{k}((P_{X}^{(m)},{\mathbf{x}}_{i}^{m}),(P_{X}^{(n)},{\mathbf{x}}_{j}^{n}))$.
The following theorem provides an upper bound on the discrepancy
$|\hat{\mathcal{E}}(f)-{\mathcal{E}}(f)|$:

######  Theorem 1 (Average risk estimation error bound for binary
classification (Blanchard et al., 2011; Wang et al., 2021)).

Assume that the loss function $\ell$ is $L_{\ell}$-Lipschitz in its first
argument and is bounded by $B_{\ell}$. Assume also that the kernels
$\mathrm{k}_{X},\mathrm{k}^{\prime}_{X}$ and $\kappa$ are bounded by
$B_{\mathrm{k}}^{2},B_{\mathrm{k}^{\prime}}^{2}\geq 1$ and $B_{\kappa}^{2}$,
respectively, and the canonical feature map
$\Phi_{\kappa}:v\in\mathcal{H}_{\mathrm{k}^{\prime}_{X}}\mapsto\kappa(v,\cdot)\in\mathcal{H}_{\kappa}$
of $\kappa$ is $L_{\kappa}$-Hölder of order $\alpha\in(0,1]$ on the closed
ball
$\mathcal{B}_{\mathcal{H}_{\mathrm{k}^{\prime}_{X}}}(B_{\mathrm{k}^{\prime}})$
111 This means that for any
$u,v\in\mathcal{B}_{\mathcal{H}_{\mathrm{k}^{\prime}_{X}}}(\mathcal{B}_{\mathcal{H}_{\mathrm{k}^{\prime}}})$,
it holds that $\|\Phi_{\kappa}(u)-\Phi_{\kappa}(v)\|\leq
L_{\kappa}\|u-v\|^{\alpha}$, where the norms are of the respective RKHSs..
Then for any $r>0$ and $\delta\in(0,1)$, with probability at least $1-\delta$,
it holds that:

$\displaystyle\sup_{f\in\mathcal{B}_{\mathcal{H}_{\bar{k}}}(r)}\left|{\hat{\mathcal{E}}}(f)-\mathcal{E}(f)\right|\leq{}$
$\displaystyle C\bigg{(}B_{\ell}\sqrt{-K^{-1}\log\delta}$ (5)
$\displaystyle+rB_{\mathrm{k}}L_{\ell}\Big{(}B_{\mathrm{k}^{\prime}}L_{\kappa}\big{(}$
$\displaystyle
n^{-1}\log(K/\delta)\big{)}^{\alpha/2}+B_{\kappa}/\sqrt{K}\Big{)}\bigg{)},$
(6)

where $C$ is a constant.

For simplicity, assuming that $\forall N_{k}=N$, this upper bound becomes
larger when $(K,N)$ is replaced with $(1,KN)$ thus indicating that using
domain groupings leads to lower error than pooling all of them as done by ERM
(Wang et al., 2021). Under the light of this result, the success of ERM over
methods that use domain labels is very surprising and warrants attention.

##  3 Need for Domain Re-labeling

An overarching assumption made by most existing MDG methods is that exploiting
the domain groups (e.g., photos, art, cartoon, sketch in PACS Li et al. (2017)
dataset) will lead to improved generalization. However, in practice, we find
that these methods do not consistently provide non-trivial gains over ERM,
particularly when fair model selection strategies are adopted Gulrajani and
Lopez-Paz (2021). Hence, in this paper, we hypothesize that the domain labels
play a crucial role in determining the trade-off between transferability
(_i.e_. domain invariance features) and domain-level performance (_i.e_. worst
group error). Formally, we state our hypothesis as follows:

Hypothesis: Existing MDG approaches that exclusively optimize for minimizing
the discrepancy between pre-specified domain groups (i.e., transferability)
present the risk of compromising the performance in one or more of the source
domains. Hence, we hypothesize that by constructing custom domain groups one
can effectively trade-off transferability and domain-specific performance.

Consequently, we propose to improve existing domain-label aware learning
techniques (GroupDRO and multi-domain ensembles in this paper) by including a
custom domain discovery step into the training process. To this end, we
consider two popular MDG algorithms – GroupDRO Sagawa* et al. (2020), a
distributional robust optimization technique, and multi-domain ensembles. At a
high-level, we ignore the domain-labels that come with the dataset and attempt
to re-categorize the data into custom domain groups during training. We
describe in detail our proposed methods, GroupDRO++ and DReaME in Sections 4
and 5 respectively.

##  4 Improving GroupDRO with Custom Domain Discovery

Input: Set of training domains
$\mathcal{D}\coloneqq\\{\mathcal{D}_{1}\dots\mathcal{D}_{K}\\}$, hyper-
parameters $\lambda,\gamma$, $\eta_{q}$, $\alpha$

Output: Trained model $f({\theta})=h\circ c$ with feature extractor $h$ and
classifier $c$

Initialization: Model parameters $\theta^{(0)}$, group label for each sample
$G\coloneqq\\{g_{i}\\}$ and weights $\\{q_{k}\\}$ ;

for  _iter in $n_{iter}$_ do

for  _t in $1\cdots T$_ do

//run GroupDRO

for  _$k$ in $1\cdots K$_ do

//update group weight $\bar{q}_{k}\leftarrow
q_{k}\exp(\eta_{q}\ell(\theta^{(t-1)};({\mathbf{x}},y)))$

end for

//Renormalize q

$q_{k}\leftarrow\frac{\bar{q}_{k}}{\sum_{l}\bar{q}_{l}}$;

$\displaystyle\mathcal{L}=\sum_{k}q_{k}\sum_{\\{({\mathbf{x}}_{i},y_{i})|g_{i}=k\\}}\ell\bigg{(}f({\mathbf{x}}_{i}),y_{i}\bigg{)}$;

$\displaystyle\mathcal{R}=\sum_{i}q^{\gamma}_{g_{i}}\ell\bigg{(}f({\mathbf{x}}_{i}),y_{i}\bigg{)}$;

//update $\theta$

$\theta^{(t)}\leftarrow\theta^{(t-1)}-\alpha\nabla_{\theta}(\mathcal{L}+\lambda\mathcal{R})$

end for

//Cluster the samples//

$\mathcal{Z}\coloneqq\cup_{i}h({\mathbf{x}}_{i})$;

$G\leftarrow$ $K$-means$(\mathcal{Z})$

end for

Algorithm 1 GroupDRO++

While one can design a variety of approaches to group samples, we adopt a deep
clustering-based solution (Caron et al., 2018) that iteratively clusters
latent representations via $K$-means and uses the cluster-labels as domain-
labels to perform GroupDRO training. We begin by first extracting the latent
representations of data from all source domains via a pre-trained feature
extractor $h$. We represent the features using the set
$\mathcal{Z}\colon\\{\mathrm{z}_{1},\mathrm{z}_{2}....\mathrm{z}_{N}\\}$ with
$\mathrm{z}_{i}=h({\mathbf{x}}_{i})$, where $N$ denotes total number of
samples pooled from all source domains. Subsequently, $\mathcal{Z}$ is
clustered using $K$-Means to form $M$ groups (can be equal to or different
than $K$) and each sample can now be represented as a tuple
$({\mathbf{x}}_{i},y_{i},g_{i})$, where $g_{i}$ denotes the cluster label. The
feature extractor $h$ and the classifier $c$ are trained with a GroupDRO style
optimization _i.e_., with adaptive weights for each group for a pre-specified
number of iterations $T$, following which, we re-compute the latent
representations for data using the updated model and perform clustering to
refine the group labels.

Figure 1: GroupDRO++ Overview. An illustration of our approach for generating
custom domain groups to improve the GroupDRO technique.

Since the clustering algorithm is disconnected from the model training except
when utilizing the updated features, it is important to regularize the
training so that meaningful domain groups can be created. Note, ERM aggregates
losses at the sample-level, while GroupDRO operates at the group level, and
there is a need to enable a finer control. To illustrate this intuitively,
consider these two non-desirable cases: (i) An individual sample can have a
high loss while its group has been assigned a smaller weight. In this case,
the update to the model $f=h\circ c$ via GroupDRO will not have the desired
effect as the weight is low; (ii) On the other hand, a sample having a low
loss value when its group has a larger weight. Due to this wrong group
association, that sample would still contribute to the SGD update. To address
these issues, we introduce a regularization term that balances both the group-
level weighting and sample-level weighting:

$\mathcal{R}=\ell(f(\mathrm{z}_{i}),y_{i})q^{\gamma}_{g_{i}},\quad
0<\gamma<1.$

Here $\ell(f(\mathrm{z}_{i}),y_{i})$ is the sample level mis-classification
error, $q_{g_{i}}$ is the weight assigned to the group that the sample belongs
to and $\gamma$ controls the sharpness of the regularizer. Thus, the final
objective is given by $\ell(f(\mathrm{z}_{i}),y_{i})+\lambda\mathcal{R}$,
where $\lambda$ is the weight associated with the regularization term when
attempting to maximize worst-case performance. Our algorithm is summarized in
Algorithm 1.

Figure 2: DReaME overview. In the meta-train stage, we compute ERM-style
gradients for all models $M_{i}$ using the data batch
$\mathcal{B}\coloneqq\\{\mathcal{B}_{1},\mathcal{B}_{2},\mathcal{B}_{3}\\}$
pooled from all source domains. Next, we use a novel model relevance score
computation to assign each meta-validation batch to the most relevant model in
the ensemble. Finally, we use a gradient-through-gradient update to learn each
of the models $f_{\theta_{i}}$.

##  5 Domain Re-Labeling for Multi-Domain Ensembles

In this section, we describe our approach for incorporating domain discovery
in multi-domain ensembles.

Formally, we represent a multi-domain ensemble using a set of models
$\left\\{f_{\theta_{m}}\right\\}_{m=1}^{M}$, where $M$ is the ensemble size
(typically set to the number of source domains $K$) and all models are
initialized randomly. Upon training, the predictions for samples from a novel
target domain $\mathcal{D^{\dagger}}$ can be obtained as an unweighted average
of predictions, _i.e_.,
$\frac{1}{M}\sum_{m=1}^{M}f_{\theta_{m}}({\mathbf{x}}_{i}),\forall{\mathbf{x}}_{i}\in\mathcal{D^{\dagger}}$.
Labeled data from each observed domain $\mathcal{D}_{k}$ is divided into three
disjoint sets - train $\mathcal{D}_{k}^{t}$, meta-validation
$\mathcal{D}_{k}^{v}$ and held-out validation $\mathcal{D}_{k}^{hv}$. Note, in
contrast to existing approaches, $M$ can be different from $K$ in DReaME.

Our approach is comprised of two stages, both operating at the mini-batch
level - (i) meta-train stage: Obtain ERM-style gradients for each constituent
member of the ensemble using the collection $\bigcup_{k}\mathcal{D}_{k}^{t}$
from all $K$ source domains; (ii) meta-test stage: Utilize the proposed model
relevance score (MRS) to determine the most appropriate model $f_{\theta_{m}}$
from the ensemble, to apply for each of the meta-validation sets
$\mathcal{D}_{k}^{v}$ and subsequently $f_{\theta_{m}}$ is updated only using
meta-gradients from the subset of $\mathcal{D}_{k}^{v}$’s assigned to this
model. This step enables the implicit re-organization of the validation mini-
batches into different domain groups. In addition, we also explore the use of
synthetic augmentations to create additional meta-validation batches for
exposing the models to more diverse variations of data from the observed
domains. Finally, the held-out validation sets $\\{\mathcal{D}_{k}^{hv}\\}$
are used for model selection (following standard practice). Algorithm 2 lists
the steps involved in our algorithm.

###  5.1 Meta-train Stage:

In every iteration, $K$ mini-batches $\\{\mathcal{B}_{k}^{t}\\}$ are randomly
sampled from the $K$ training sets $\\{\mathcal{D}_{k}^{t}\\}$, which are then
pooled to form the data batch
$\mathcal{B}\coloneqq\bigcup_{k}\mathcal{B}_{k}^{t}\subset\mathcal{D}_{k}^{t}$
and passed as input to all $M$ models (initialized randomly). The empirical
risk

$\mathcal{L}_{\theta_{m}}=\frac{1}{|\mathcal{B}|}\sum_{({\mathbf{x}}_{i},y_{i})\in\mathcal{B}}\ell(f_{\theta_{m}}({\mathbf{x}}_{i}),y_{i}),\forall
m\in(1,\cdots,M),$ (7)

and the corresponding gradients for each of the models
$\nabla_{\theta_{m}}(\mathcal{L}_{\theta_{m}})$ are computed independently
w.r.t. $\mathcal{B}$. Akin to any MAML (Finn et al., 2017b) style algorithm,
DReaME takes one gradient step for each of the models $f_{\theta_{m}}$ to
obtain $f_{\theta^{\prime}_{m}}$, _i.e_.,

$\theta^{\prime}_{m}=\theta_{m}-\alpha\nabla_{\theta_{m}}\mathcal{L}_{\theta_{m}}(\theta_{m}),\forall
m\in(1,\cdots,M)$ (8)

with a pre-specified learning rate $\alpha$.

Input: Set of training domains
$\mathcal{D}\coloneqq\\{\mathcal{D}_{1}\dots\mathcal{D}_{K}\\}$

Output: Ensemble $\\{f_{\theta_{1}},\dots,f_{\theta_{M}}\\}$

Initialization: Parameters $\\{\theta_{1},\dots,\theta_{M}\\}$, meta-train
sets $\\{\mathcal{D}^{t}_{1}\dots\mathcal{D}^{t}_{K}\\}$, meta-validation sets
$\\{\mathcal{D}^{v}_{1}\dots\mathcal{D}^{v}_{K}\\}$, hyper-parameters
$\alpha,\lambda,\eta$ ;

for  _iter in $n_{iter}$_ do

//meta-train //

for  _$f_{\theta_{m}}$ in $f_{\theta_{1}}\cdots f_{\theta_{M}}$_ do

$\mathcal{B}=[\mathcal{B}_{1},\cdots\mathcal{B}_{K}]$ // pool the minibatches
from $\\{\mathcal{D}_{k}^{t}\\}$ //

Compute empirical risk $\mathcal{L}_{\theta_{m}}$ w.r.t. $\mathcal{B}$, using
eq. (7);

//inner gradient update//

Update
$\theta^{\prime}_{m}=\theta_{m}-\alpha\nabla_{\theta_{m}}\mathcal{L}_{\theta_{m}}(\theta_{m})$

end for

//model relevance score//

for  _$k$ in $1\cdots K$ _ do

for  _$f_{\theta_{m}}$ in $f_{\theta_{1}}\cdots f_{\theta_{M}}$_ do

sample a mini-batch $\mathcal{V}_{k}$ from $\mathcal{D}_{k}^{v}$;

compute $\beta_{km}$ using eq. (12);

end for

end for

//meta-update//

for  _$f_{\theta_{m}}$ in $f_{\theta_{1}}\cdots f_{\theta_{M}}$_ do

Identify $\gamma_{m}$, the set of indices of meta-validation batches, assigned
to $f_{\theta_{m}}$;

Compute meta-test loss $\mathcal{G}_{\theta^{\prime}_{m}}$ using eq. (10);

Perform meta-update using eq. (11)

end for

end for

Algorithm 2 DReaME training

###  5.2 Meta-test Stage

While the train stage is similar to conventional ERM, our goal is to build
ensembles that implicitly identify optimal domain groups for improved
generalization. To achieve this goal, we systematically regulate the gradient
flow from the meta-validation batches to each of the constituent models based
on a model relevance score. We denote a generic, model relevance scoring
function by $\mathrm{S}\colon\mathcal{V}_{k}\times
f_{\theta_{m}}\rightarrow\mathbb{R}^{+}[0,1]$ which scores the model
$f_{\theta_{m}}$ for a mini-batch $\mathcal{V}_{k}\subset\mathcal{D}_{k}^{v}$
from the meta-validation dataset. We denote by $\beta_{km}$ the resulting
score _i.e_.,

$\beta_{km}=\mathrm{S}(\mathcal{V}_{k},f_{\theta_{m}})$ (9)

We now discuss the optimization process and subsequently describe our proposed
gradient-matching based MRS below.

Optimization: Intuitively, when the relevance score $S$ is high, one expects
that taking a gradient step for $\theta_{m}$ based on $\mathcal{B}$ is highly
likely to improve the performance on $\mathcal{V}_{k}$. Hence, we compute this
relevance score between every pair of meta-validation mini-batches
$\\{\mathcal{V}_{1},\ldots,\mathcal{V}_{K}\\}$ and the models
$\\{f_{\theta_{1}},\ldots,f_{\theta_{M}}\\}$ to obtain the matrix
$\boldsymbol{\beta}\in\mathbb{R}^{K\times M}$. By identifying the model with
the largest $\beta_{km}$ value, one can assign the most relevant model for
each validation mini-batch $\mathcal{V}_{k}$. The final step is to compute the
meta-gradients for $\theta_{m},\forall m$ w.r.t. to their “relevant” domains
and perform a gradient-through-gradient update. We denote the indices of meta-
validation batches that are assigned to $f_{\theta_{m}}$ by
$\gamma_{m}=\\{j\in(1,\cdots,K)\\}$, such that for each $j$, model
$f_{\theta_{m}}$ provides the largest MRS. The meta-validation loss
$\mathcal{G}_{\theta^{\prime}_{m}}$ of $f_{\theta^{\prime}_{m}}$ using the
relevant validation batches, $\gamma_{m}$, can be written as

$\mathcal{G}_{\theta^{\prime}_{m}}=\sum_{\forall({\mathbf{x}}_{i},y_{i})\in\\{\mathcal{V}_{j}\\},j\in\gamma_{m}}\ell(f_{\theta^{\prime}_{m}}({\mathbf{x}}_{i}),y_{i}),$
(10)

where the definition of $\theta^{\prime}_{m}$ comes from eq. 8. The final
meta-update of $f_{\theta_{m}}$ using a gradient-through-gradient optimization
can be written as follows:

$\hat{\theta}_{m}=\theta_{m}-\lambda\frac{\partial(\mathcal{L}_{\theta_{m}}+\eta\mathcal{G}_{\theta^{\prime}_{m}})}{\partial\theta_{m}}.$
(11)

As a consequence of computing this assignment for meta-update in every
iteration: (i) different parts of data from one source domain could get
assigned to different models, thus producing a multi-domain ensemble that is
guided by the inferred domain groups; (ii) in cases where none of the
validation batches are assigned to a model, it converges to a standard ERM-
based solution from eq. (8), which is still a strong baseline model.

MRS Design: As outlined above, through the MRS scoring function $\mathrm{S}$,
we determine assignment of a model for a given meta-validation batch. We
considered the following choices for implementing $\mathrm{S}$:

  1. 1.

Random Assignment: In this case, $\mathrm{S}$ assigns randomly assigns each
mini-batch to one of the models;

  2. 2.

All-to-All assignment: Here, every validation mini-batch $\mathcal{V}_{k}$ is
assigned to all members of members;

  3. 3.

Loss-based assignment: In this case, the empirical loss is used to determine
the member assignment _i.e_.,
$\beta_{km}=1-\frac{1}{|\mathcal{V}_{k}|}\sum_{({\mathbf{x}}_{i},y_{i})\in\mathcal{V}_{k}}\ell(f_{\theta_{m}}({\mathbf{x}}_{i}),y_{i})$.

  4. 4.

Gradient-matching: While empirical loss-based assignment is a reasonable
choice, we propose to implement MRS through gradient-matching.

######  Definition 1 (Gradient-matching for model assignment).

$\beta_{km}=\sum\nabla_{\theta_{m}}(\mathcal{L}_{\theta_{m}}).\nabla_{\theta_{m}}(\mathcal{G}_{\theta_{m}}^{k}),$
(12)

where $\mathcal{L}_{\theta_{m}}$ and $\mathcal{G}_{\theta_{m}}^{k}$ are the
empirical risks (eq. 1) computed using the model $f_{\theta_{m}}$ on the meta-
train ($\mathcal{B}$) and meta-validation ($\mathcal{V}_{k}$) batches
respectively.

The summation is over all parameters in $\theta_{m}$, and this score computes
the dot product between parameter sensitivities of $\theta_{m}$ w.r.t. the
train and validation batches. Though gradient-matching has been used in
different contexts - for example, model alignment in MAML (Finn et al.,
2017b), task affinity in multi-task learning (Standley et al., 2020),
promoting diversity in active sample selection (Ash et al., 2020) etc., we
make a surprising finding that gradient-based domain grouping is superior to
loss-based grouping in domain re-labeling, though the latter metric is
routinely used for model selection in MDG.

###  5.3 Model Selection Strategies

A crucial component of any ZS-MDG algorithm is the specification of a model
selection criterion. Here, model selection mainly refers to the selection of
appropriate training checkpoints to evaluate on the unobserved domain
$\mathcal{D^{\dagger}}$. It was found in (Gulrajani and Lopez-Paz, 2021) that
different model selection criteria lead to drastically different performance
for the same method, thus making benchmarking of ZS-MDG approaches
challenging.

In the context of MDG with ensemble-based approaches, the choice of model
selection strategy has not been studied before. Note that, with DReaME, we
perform inference for a test sample by averaging the predictions from all $M$
models in the ensemble, ${\mathbf{x}}\in\mathcal{D^{\dagger}}$ ,
$\hat{y}=\frac{1}{M}\sum_{m=1}^{M}f_{\theta_{m}}({\mathbf{x}})$. Defining the
accuracy function as $\mathrm{A}:{\mathbf{x}}\times
y\rightarrow\mathbb{R}^{+}[0,1]$, we can compute the performance of an
individual model for a domain $k$ as
$\mathrm{A}(\mathcal{D}_{k}^{hv};f_{\theta_{m}})$ and that of an ensemble as
$\mathrm{A}(\mathcal{D}_{k}^{hv};\\{f_{\theta_{m}}\\})$. We investigate two
model selection strategies in this study:

Overall Avg: In this strategy, we choose the checkpoint in which each
individual model $f_{\theta_{m}}$ produces high accuracy on each of the $K$
domains, on average. In other words,

$\arg\max_{c}\frac{1}{MK}\sum_{m=1}^{M}\sum_{k=1}^{K}\mathrm{A}(\mathcal{D}_{k}^{hv};f_{\theta_{m}}^{c}),$

where $c$ indicates the training checkpoint index.

Overall Ens: In this case, we choose the checkpoint in which the ensemble
produces the highest accuracy for each of the $K$ domains, on average.

$\arg\max_{c}\frac{1}{K}\sum_{k=1}^{K}\mathrm{A}(\mathcal{D}_{k}^{hv};\\{f_{\theta_{m}}^{c}\\}).$

###  5.4 Augmenting Meta-Validation Data

Our empirical study shows that by including synthetically augmented versions
of the meta-validation batches, one can better leverage the intra-diversity in
the source domains and further enhance the performance of DReaME. Formally, we
use standard image augmentation strategies (details in Section 6) on the meta-
validation batches $\\{\mathcal{V}_{1},\ldots,\mathcal{V}_{K}\\}$ to produce
$\\{\mathcal{V}_{K+1},\ldots,\mathcal{V}_{\bar{K}}\\}$, where $\bar{K}-K$ is
the number of additional batches. Note, through ablation studies, we
demonstrate what role this augmentation for meta-validation data plays
alongside the standard practice of training data augmentation adopted in the
meta-train stage.

##  6 Experiments

Figure 3: Benchmarking DReaME. Our proposed approach significantly improves
upon ERM (left) as well as sophisticated ensemble construction methods (right)
in ZS-MDG, wherein we obtain larger performance gains as the domain
discrepancy becomes more severe.

###  6.1 Dataset Description

We evaluate DReaME using six standard visual MDG benchmarks (i) PACS (Li et
al., 2017) dataset comprising $4$ domains, namely photos, art, cartoon and
sketches, with images belonging to $7$ different classes; (ii) VLCS (Fang et
al., 2013) dataset, which is also comprised of $4$ domains corresponding to
the four benchmark image datasets (Caltech101, LabelMe, SUN09 and VOC2007) and
contains images from $5$ classes; (iii) OfficeHome (Venkateswara et al., 2017)
dataset containing images from $65$ classes, where the images represent $4$
different domains, namely art, clipart, product and real respectively; (iv)
Terra Incognita (Beery et al., 2018) comprised of camera trap images of wild
animals obtained from four different camera angles (i.e., domains) and $10$
different wildlife categories; and (v) Camelyon$17$-WILDS (Bandi et al., 2018;
Koh et al., 2021) consisting roughly 400k images of potentially cancer cells
taken at different hospitals and scanners.

###  6.2 Experimental Setup

Following standard practice in ZS-MDG, for every dataset except for
Camelyon$17$-WILDS, we run experiments by leaving out one of $K$ domains for
testing while using the $K-1$ domains for training. For Camelyon$17$-WILDS
(Bandi et al., 2018; Koh et al., 2021), we use the standard protocol of using
data from first three hospitals as training domains and use data from fourth
and fifth hospitals as validation and testing domains. To enable a fair
comparison with the state-of-the-art, we use ResNet-50 (He et al., 2016), pre-
trained on ImageNet (Russakovsky et al., 2015) as the backbone feature
extractor for all experiments.

For GroupDRO++, we use the following settings: (i) number of training
iterations $n_{iter}$ is set to $5000$ and $T$ to $300$; (ii) number of groups
is fixed at $K=4$; (iii) $\lambda,\gamma,\eta_{q}$ are set to $0.1$, $0.3$ and
$0.2$ respectively; (iv) batch size of $32$ per domain; and (v) Adam optimizer
(Kingma and Ba, 2015) with learning rate $5e-5$.

For DReaME, we use a random 80-20 split from each of the source domains to
obtain the train and validation sets, while the train set itself is further
subdivided (80-20) to construct meta-train and meta-validation data. We report
the mean and standard deviation of performance, obtained across three trials
with different random seeds, for each experiment similar to (Gulrajani and
Lopez-Paz, 2021). Across all experiments we use the following hyper-
parameters: (i) batch size of $32$ per domain; (ii) Adam optimizer (Kingma and
Ba, 2015) with learning rate of $5e-5$ (for both $\alpha$ and $\lambda$);
(iii) number of training iterations set to $5000$ and (iv) ensemble size $M$
is set to $3$. We implement DReaME into the publicly available DomainBed
framework.

Figure 4: Analysis of DReaME design. (left) When compared to approaches that
utilize advanced data augmentation strategies to improve generalization,
DReaME eliminates the need for tailoring the augmentation strategy and is
consistently effective for all benchmarks. In contrast, even sophisticated
approaches such as RandConv (RC) or its combination with RandAug (RA) provide
varying degrees of improvements over ERM (RA) across different datasets; (b)
When compared against different design choices for the MRS function
$\mathrm{S}$, we find that the proposed gradient-matching performs the best.

In DReaME, the training mini-batches are augmented using a composition of the
following augmentation choices: random horizontal flip, random color jitter
and grayscaling with $10\%$ probability, which we refer to as RandAug (RA). As
described in Section 5.2, we also create additional meta-validation batches by
augmenting each batch $\mathcal{V}_{k}$ using subsets of augmentations used
during training. We set $\eta$ in eq. (11) to $1.0$ and study the sensitivity
of this hyper-parameter as part of our ablation study. We report results for
both the proposed model selection strategies and our rigorous empirical study
shows that the Overall Avg. strategy provides a small margin of improvement
over Overall Ens.

Table 1: Summary performance of popular ZS-MDG baselines obtained using
GroupDRO++ and DReaME. While the proposed GroupDRO++ improves over ERM and
vanilla GroupDRO methods, DReaME with Overall (Avg.) model selection
consistently achieves the best generalization performance.

Methods | PACS | VLCS | OfficeHome | TerraIncognita
---|---|---|---|---
ERM (RA) | 85.5 $\pm$ 0.2 | 77.5 $\pm$ 0.4 | 66.5 $\pm$ 0.3 | 46.1 $\pm$ 1.8
IRM (Arjovsky et al., 2019) | 83.5 $\pm$ 0.8 | 78.5 $\pm$ 0.5 | 64.3 $\pm$ 2.2 | 47.6 $\pm$ 0.8
MLDG (Li et al., 2018) | 84.9 $\pm$ 1.0 | 77.2 $\pm$ 0.4 | 66.8 $\pm$ 0.6 | 47.7 $\pm$ 0.9
ARM (Zhang et al., 2020b) | 85.1 $\pm$ 0.4 | 77.6 $\pm$ 0.3 | 64.8 $\pm$ 0.3 | 45.5 $\pm$ 0.3
RSC (Huang et al., 2020) | 85.2 $\pm$ 0.9 | 77.1 $\pm$ 0.5 | 65.5 $\pm$ 0.9 | 46.6 $\pm$ 1.0
CORAL (Sun and Saenko, 2016) | 86.2 $\pm$ 0.3 | 78.8 $\pm$ 0.6 | 68.7 $\pm$ 0.3 | 47.6 $\pm$ 1.0
GroupDRO (Sagawa* et al., 2020) | 84.4 $\pm$ 0.8 | 76.7 $\pm$ 0.6 | 66.0 $\pm$ 0.7 | 43.2 $\pm$ 1.1
GroupDRO++ (ours) | 86.66 $\pm$ 0.4 | 79.81 $\pm$ 0.5 | 67.1$\pm$0.3 | 47.45 $\pm$ 0.3
DReaME (Avg.) (ours) | 87.35 $\pm$ 0.2 | 79.02 $\pm$ 0.3 | 69.76 $\pm$ 0.2 | 48.66 $\pm$ 0.2

Table 2: Performance of DReaME on the challenging WILDS-Camelyon17 benchmark.
In addition to being significantly superior to ERM (RA), DReaME outperforms
best-performing methods such as SagNet (Nam et al., 2019) and CORAL (Sun and
Saenko, 2016).

Method | Accuracy
---|---
ERM | 82.31
ERM (RA) | 85.21
CORAL (Sun and Saenko, 2016) | 92.7
SagNet (Nam et al., 2019) | 92.9
GroupDRO++ (ours) | 86.7
DReaME (Avg.) (ours) | 94.6

###  6.3 Key Findings

We now present a summary of the key findings from our empirical studies. We
first find that re-labeling indeed leads to significant improvements over ERM
thus providing evidence to our core hypothesis and sheds new light on the
competitive behavior of ERM (Gulrajani and Lopez-Paz, 2021). From the results,
we notice that GroupDRO, despite leveraging the domain labels, often performs
poorly compared to the vanilla ERM. In contrast, GroupDRO++ improves upon both
these baselines. Finally, we how that DReaME outperforms other existing
ensembling strategies and produces state-of-the-art domain generalization
performance.

Table 3: Re-labeling domains improves generalization. Here, for each dataset,
we show the detailed generalization results for each of the domains using
models trained with the remaining three domains. Both DReaME and GroupDRO++,
which re-label samples from the different source domains and perform multi-
domain training, lead to significant performance gains over ERM as well as the
standard GroupDRO implementation.

Dataset: PACS
---
Method | A | C | P | S | Average
ERM | $84.7\pm 0.4$ | $80.8\pm 0.6$ | $\pagecolor[HTML]{FFDD86}97.2\pm 0.3$ | $79.3\pm 1.0$ | $85.5$
GroupDRO | $83.5\pm 0.9$ | $79.1\pm 0.6$ | $96.7\pm 0.3$ | $78.3\pm 2.0$ | $84.4$
GroupDRO++ (ours) | ${84.99\pm 0.2}$ | $82.78\pm 0.4$ | $97.4\pm 0.3$ | ${81.2\pm 0.6}$ | ${86.6}$
DReaME (Avg.) (ours) | $\pagecolor[HTML]{b1e9b0}88.88\pm 1.0$ | $81.9\pm 1.7$ | ${96.79\pm 0.4}$ | $81.84\pm 0.1$ | $87.35$
Dataset: VLCS
Method | C | L | S | V | Average
ERM | $97.7\pm 0.4$ | $64.3\pm 0.9$ | $73.4\pm 0.5$ | $74.6\pm 1.3$ | $77.5$
GroupDRO | $97.3\pm 0.3$ | $63.4\pm 0.9$ | $69.5\pm 0.8$ | $76.7\pm 0.7$ | $76.7$
GroupDRO++ (ours) | $98.41\pm 0.5$ | $\pagecolor[HTML]{b1e9b0}67.34\pm 0.8$ | $\pagecolor[HTML]{b1e9b0}75.7\pm 0.2$ | $\pagecolor[HTML]{FFDD86}{77.79\pm 0.4}$ | $\pagecolor[HTML]{b1e9b0}79.81$
DReaME (Avg.) (ours) | $\pagecolor[HTML]{b1e9b0}98.69\pm 0.6$ | $65.99\pm 0.4$ | $73.12\pm 0.5$ | $\pagecolor[HTML]{b1e9b0}78.27\pm 0.6$ | $79.02$
Dataset: OfficeHome
Method | A | C | P | R | Average
ERM | $61.30\pm 0.7$ | $52.40\pm 0.3$ | $75.80\pm 0.1$ | $76.60\pm 0.3$ | $66.53$
GroupDRO | $60.4\pm 0.7$ | $52.7\pm 1.0$ | $75.0\pm 0.7$ | $76.0\pm 0.7$ | $66.0$
GroupDRO++ (ours) | $61.2\pm 0.4$ | $54.4\pm 0.2$ | $75.9\pm 0.3$ | $75.02\pm 0.2$ | $67.1$
DReaME (Avg.) (ours) | $\pagecolor[HTML]{b1e9b0}64.38\pm 1.2$ | $\pagecolor[HTML]{b1e9b0}55.48\pm 0.6$ | $\pagecolor[HTML]{b1e9b0}79.00\pm 0.7$ | $\pagecolor[HTML]{b1e9b0}80.16\pm 0.4$ | $\pagecolor[HTML]{b1e9b0}69.76$
Dataset: Terra Incognita
Method | L100 | L38 | L43 | L46 | Average
ERM | $49.8\pm 4.4$ | $42.1\pm 1.4$ | $56.9\pm 1.8$ | $35.7\pm 3.9$ | $46.13$
GroupDRO | $41.2\pm 0.7$ | $38.6\pm 2.1$ | $56.7\pm 0.9$ | $36.4\pm 2.1$ | $43.2$
GroupDRO++ (ours) | $50.7\pm 0.9$ | $\pagecolor[HTML]{b1e9b0}44.5\pm 1.0$ | $\pagecolor[HTML]{b1e9b0}57.4\pm 0.8$ | $37.2\pm 0.6$ | $47.45$
DReaME (Avg.) (ours) | $\pagecolor[HTML]{b1e9b0}53.65\pm 2.9$ | $44.25\pm 1.7$ | $56.4\pm 0.2$ | $\pagecolor[HTML]{b1e9b0}40.35\pm 0.6$ | $\pagecolor[HTML]{b1e9b0}48.66$

Finding 1: GroupDRO++ and DReaME provide significant gains over ERM.

As can be seen from Figure 3 (left), Tables 1, 3 and 2, using custom domain
groups is highly beneficial. Overall, across the benchmarks, both GroupDRO++
and DReaME improve over ERM by significant margins, in terms of average
generalization performance, regardless of the level of cross-domain gap
inherent to each of the datasets. Furthermore, it is clearly apparent that
GroupDRO++ is consistently better than the vanilla GroupDRO implementation.
Through the use of a deep ensemble backbone, DReaME produces the best
performance among the two proposed methods. For example, datasets such as
TerraIncognita, and Camelyon$17$ are known to contain much higher cross-domain
discrepancies compared to VLCS, thus making zero-shot generalization more
challenging. However, we find that DReaME with $M=3$ achieves large
performance gains over ERM and importantly, the gap with ERM widens as the
severity of domain shift increases. In the challenging Camelyon$17$-WILDS
dataset, DReaME provides $>10\%$ gains over ERM. In order to gain insights
into the behavior of DReaME, in Figure 5, we visualize examples images
assigned to the three models ($M=3$) in the ensemble. For this illustration,
we trained DReaME using the art (A), cartoon (C) and photo (P) domains from
the PACS benchmark. This clearly shows that images from the same input domain
can be assigned to different groups, thus enabling improved generalization.
Interestingly, we observe that each model evolves to specialize for different
styles (in terms of image statistics) and semantic concepts (e.g., images
corresponding to the person class are strongly representative of group 3).

Finding 2: DReaME outperforms existing ensembling methods with the same
complexity.

We perform a comparative analysis of DReaME to a state-of-the-art multi-domain
ensembling method, DMG (Chattopadhyay et al., 2020), which infers individual
masks over the neurons for each of the source domains. Note that both DMG and
DReaME are implemented using the DomainBed framework, and hence use the same
experiment protocol, i.e., architecture, augmentations etc. In Figure
3(right), the superiority of DReaME over DMG is clearly evident across all
benchmarks, with improvements as high as $4.5\%$ on the Office-Home benchmark.
Through our ERM style update in the meta-train stage and by taking into
account the relevance of a member w.r.t. to a domain during the meta-test
stage, we effectively re-label the samples into optimal domain groups.
Detailed results across datasets are reported in Table 1.

Finding 3: DReaME provides non-trivial improvements over SoTA DG methods.

Despite the effectiveness of ERM as a baseline, its inability to leverage
domain discrepancies implies that there is a non-trivial performance gap
between ERM and state-of-the-art MDG methods. Using rigorous comparisons with
benchmarks created by (Gulrajani and Lopez-Paz, 2021), and compiled in Table
1, we find that DReaME is highly competitive with the SoTA methods, which rely
on a variety of strategies to leverage cross-domain discrepancies.
Interestingly, our approach produces state-of-the-art results on PACS
(+$1.1\%$), VLCS(+$1.2\%$), Office-Home (+$1.36\%$) and matches the
performance of the state-of-the-art SagNet on the TerraIncognita dataset.
Similarly, as showed in Table 2, on Camelyon$17$, DReaME achieves an accuracy
of $94.6\%$, outperforming best-performing approaches such as SagNet and CORAL
by a margin of $\sim 2\%$.

(a)

(b)

Figure 5: Visualizing the groups inferred using DReaME. (a) For this
visualization, we trained DReaME using A (art), C (cartoon) and P (photo)
domains from the PACS dataset and show randomly selected example images
assigned to each of the three models in the ensemble. We notice that the MRS
scoring based on gradient-matching assigns different subsets of each input
domain to different groups; (b) We plot the distribution of class labels in
each of the inferred domain groups. Interestingly, while MRS assigns different
subsets of domains to different groups, it does not perform a trivial category
split. In fact, our approach is able to effectively exploit the intra- and
inter-domain discrepancies in order to evolve meaningful groups that maximally
benefit the generalization performance.

###  6.4 Ablations

In this section, we perform ablation experiments to understand (i) the role of
our proposed gradient-matching based MRS; (ii) the impact of meta-validation
and train augmentation; (iii) the choice of $\eta$ and lastly (iv) the impact
of ensemble size on generalization performance.

####  6.4.1 Choice of MRS strategy

As explained in Section 5, we experimented with four different choices for the
scoring function $\mathrm{S}$. In Figure 4(right), we show the effect of these
choices on the performance using the VLCS dataset. We find that the proposed
gradient-matching based MRS performs the best, considerably outperforming
loss-based model assignment.

####  6.4.2 Impact of meta-validation and train augmentation

In Table 4, we show how the performance of DReaME varies when the augmentation
protocol is changed. As discussed earlier, we divide data from each observed
domain into three disjoint sets - train, meta-validation and held-out
validation. We explore whether adding synthetic augmentations to create
additional meta-validation batches can lead to improved implicit grouping.
Note that, in this setup, we applied standard augmentation (following
DomainBed) to the meta-train batches. It is evident from the table that,
removing meta-validation augmentation leads upto $~{}1.5\%$ decrease in
accuracy in the cases of VLCS and Office Home datasets.

We performed another ablation by not including any augmentation to the meta-
train batches, while still considering synthetically augmented meta-validation
batches. We observed that, for benchmarks with larger cross-domain gap such
as, TerraIncognita (Beery et al., 2018), removing augmentation during training
has significant effect on performance with upto $~{}2.8\%$ drop and
unsurprisingly the performance in this case of no train augmentation is lower
than ERM with train augmentation. In contrast, with benchmarks such as VLCS,
not applying train augmentation does not show any apparent impact.

Table 4: Impact of augmenting meta-train and meta-validation sets. Without
meta-validation augmentation and only train augmentation, we observed reduced
performance across all benchmarks. On the other hand, omitting train
augmentation and including only meta-validation leads to bigger performance
drops on more challenging datasets such as TerraIncognita.

Method PACS VLCS Office Home Terra Incognita DReaME (No Meta-Valid Aug) Avg
86.93 $\pm$ 0.73 78.50 $\pm$ 0.17 69.37 $\pm$ 0.12 48.38 $\pm$ 0.73 Ens 86.58
$\pm$ 1.52 77.06 $\pm$ 0.05 68.74 $\pm$ 0.28 47.99 $\pm$ 0.97 DReaME (No Train
Aug) Avg 85.46 $\pm$ 1.29 78.88 $\pm$ 0.45 69.05 $\pm$ 0.28 45.87 $\pm$ 1.13
Ens 85.21 $\pm$ 1.26 78.52 $\pm$ 0.27 69.29 $\pm$ 0.36 46.09 $\pm$ 0.52 DReaME
Avg 87.35 $\pm$ 0.22 79.02 $\pm$ 0.20 69.76 $\pm$ 0.48 48.66 $\pm$ 0.85 Ens
87.21 $\pm$ 0.94 78.40 $\pm$ 0.19 70.06 $\pm$ 0.16 48.48 $\pm$ 0.96

####  6.4.3 Choice of $\eta$

Next, we studied the sensitivity of the penalty $\eta$, a hyper-parameter that
controls the penalty for the meta-validation loss in eq. (11) in terms of the
performance of DReaME using the OfficeHome (Venkateswara et al., 2017)
dataset. As evidenced in Table 5, for values greater than $0.2$ the
performance of DReaME is stable with respect to changes in $\eta$.

Table 5: Impact of the choice of $\eta$. We find that DReaME is stable w.r.t. change in $\eta$ values. Choice of $\eta$ | A | C | P | R | Avg.
---|---|---|---|---|---
$\eta=1$ | Avg | 63.08 | 55.52 | 78.60 | 79.80 | 69.25
Ens | 64.72 | 55.09 | 78.60 | 81.15 | 69.89
$\eta=0.8$ | Avg | 63.44 | 56.41 | 77.42 | 79.06 | 69.08
Ens | 62.98 | 54.85 | 77.43 | 79.23 | 68.62
$\eta=0.6$ | Avg | 62.61 | 56.65 | 76.63 | 78.91 | 68.70
Ens | 62.52 | 57.18 | 76.54 | 78.60 | 68.71
$\eta=0.4$ | Avg | 63.54 | 56.41 | 77.06 | 79.35 | 69.09
Ens | 62.72 | 56.41 | 76.83 | 79.29 | 68.81
$\eta=0.2$ | Avg | 63.13 | 56.67 | 76.60 | 79.98 | 69.10
Ens | 62.05 | 56.16 | 76.46 | 79.58 | 68.56

####  6.4.4 Impact of Ensemble Size.

The ensemble size $M$ is a standard hyper-parameter in any ensembling
approach. The complexity (size) of the ensemble is controlled such that a
simple averaging strategy can still work for unseen test domains. When $M$
becomes large and models become diverse, one might require a “manager” module
to select a specific model from the ensemble. However, the ensemble size is
not necessarily connected to the number of domains and for simplicity, we
fixed $M=3$ for all cases (identified using parameter search on PACS, VLCS).
Table 6 shows the performance of DReaME on VLCS (Fang et al., 2013) at
different ensemble sizes ($M=2,3,4$). We observe that while a three-model
ensemble outperforms two-model ensemble by almost $2\%$, there is no
significant improvement achieved beyond $M=3$. We note that at this time the
choice of $M$ is empirical, and extending this framework to automatically
identify the required number of models, such that simple averaging of
predictions from the ensemble can still be effective, is part of our future
work.

Table 6: Impact of change in ensemble size. For the VLCS benchmark, We observed that there was no improvement beyond $M=3$. Choice of M | C | L | S | V | Avg.
---|---|---|---|---|---
M = 2 | Avg | 98.32 | 62.58 | 69.04 | 77.78 | 76.93
Ens | 98.05 | 62.40 | 72.88 | 76.78 | 77.53
M = 3 | Avg | 98.69 | 65.99 | 73.12 | 78.27 | 79.02
Ens | 98.60 | 65.61 | 72.26 | 77.11 | 78.4
M = 4 | Avg | 98.58 | 65.84 | 72.58 | 78.02 | 78.76
Ens | 98.58 | 65.43 | 72.76 | 78.41 | 78.80

##  7 Related Work

In several computer vision tasks, deep models are often found to generalize
poorly to out-of-distribution (OOD) data. Consequently, several problem
formulations have emerged to address this critical limitation, for example,
unsupervised domain adaptation, domain generalization and test-time
adaptation. These approaches are typically differentiated by the assumptions
made on the access to source (ID) data and target (OOD) data. In this section,
we provide a brief review of different multi-domain generalization approaches
as it relates to our method. In particular, we organize existing methods
broadly into methods that are domain-label agnostic and those that leverage
the domain groups.

##### Domain-Label Agnostic MDG

This class of approaches ignores the domain labels and assumes the observed
data from multiple sources to be drawn from a joint distribution. A
prototypical example is the standard ERM (Vapnik, 1999) training, which pools
data from all source domains and infers a classifier that minimizes the risk
on this pooled set. While this method has been long known to be inferior to
domain-label aware methods, a recent large-scale study showed that with
powerful feature extractors such as Resnet-50 (He et al., 2016) and consistent
model selection criteria, ERM is indeed a strong baseline. Huang _et al_.
(Huang et al., 2020) propose to improve upon ERM by iteratively discarding the
top $p$-percentile gradients in final layers of the network by zeroing them
and then performing an update. They observed that by discarding top-$p$
gradients leads to better OOD generalization as compared to discarding top-$p$
features (Park and Kwak, 2016). Pezeshki _et al_. (Pezeshki et al., 2021)
further studied the interesting gradient starvation phenomenon that arises in
ERM when optimized through gradient descent. They observed that ERM training
captures only a subset of features while other predictive features are
ignored, and hence spectral decoupling can be used to regularize the process.

###  7.1 Domain-Label Aware MDG

More recent approaches in MDG attempt to improve generalization by leveraging
the prior knowledge about the domain groups (_e.g_. photos, art, cartoon,
sketch in PACS (Li et al., 2017) dataset). At the core, these methods learn
invariant features by decreasing the discrepancy between the different
domains. For example, CORAL (Sun and Saenko, 2016) and CAADA (caa, 2020)
simultaneously infer an effective classifier, while also matching the feature
covariance matrices across different training domains either through
customized loss functions or through adversarial learning. On the other hand,
Gretton _et al_. (Gretton et al., 2012) directly minimize the maximum mean
discrepancy between different domains.

In the pioneering work of DANN (Ganin et al., 2016), Ganin _et al_. utilized
adversarial learning to learn domain-invariant features by constructing a
gradient reversal layer. Several follow-up works (Akuzawa et al., 2019;
Albuquerque et al., 2020) improved upon DANN by explicitly considering
statistical dependence between domains. Style-Agnostic network (Nam et al.,
2019) is another recently proposed method that attempts to reduce the style
bias corresponding to each domain by editing the styles through an adversarial
learning paradigm. Building upon mixup (Zhang et al., 2017), a popular
approach in the image-classification setting, many methods (Yan et al., 2020;
Xu et al., 2020) have proposed inter-domain mixup _i.e_., learning from
linearly interpolated examples from random pairs of domains and their labels.
Marginal transfer learning (MTL) (Blanchard et al., 2017, 2011) proposes to
build classifiers that take in an additional input in the form of a domain-
specific prototypical feature embedding. Adaptive risk minimization (ARM)
(Zhang et al., 2020b) further improves upon MTL by using a separate embedding
convolutional neural network. Arjovsky _et al_. (Arjovsky et al., 2019)
subsequently introduced the paradigm of Invariant risk minimization (IRM) and
proposed to learn feature representations such that there is a classifier that
is simultaneously optimal for all domains. In general, causal learning has
inspired MDG approaches such as IRM (Arjovsky et al., 2019), MatchDG (Mahajan
et al., 2021) and Deep CAMA (Zhang et al., 2020a). Meta-learning approaches
such as MLDG (Li et al., 2018) build upon the MAML (Finn et al., 2017a)
framework to learn a feature extractor using only $K-1$ domains and the
$K^{\text{th}}$ domain to perform the meta-update.

Given the inherent limitations of ERM in handling multi-domain data (Blanchard
et al., 2011), researchers have explored the use of distributional robust
optimization methods (Sagawa* et al., 2020; Liu et al., 2021; Krueger et al.,
2021) that attempt to reduce the worst-group error _i.e_., the training domain
with the highest error, by suitably weighting the loss function. In
particular, GroupDRO (Sagawa* et al., 2020) is a popular example that utilizes
an adaptive sample weighting mechanism to suitably adjust the loss function.

Finally, multi-domain ensembling forms an important class of MDG approaches
that are often found to produce state-of-the-art generalization at the cost of
the need to train and infer with multiple models. Existing multi-domain
ensembling methods typically combine models independently trained on each of
the source domains (Mancini et al., 2018; Li et al., 2019; Seo et al., 2020;
Chattopadhyay et al., 2020; Matsuura and Harada, 2020). For example, (Seo et
al., 2020) uses a common model across the members of the ensemble, while
allowing domain-specific normalization layers. On the other hand,
(Chattopadhyay et al., 2020) proposed to build domain-specific masks over
neurons and optimized the masks for minimizing cross-domain feature overlap.
Furthermore, other existing approaches have also focused on building an
additional model to determine the weights to be used while aggregating the
predictions from different models in an ensemble at test time (Wang et al.,
2020). While this approach could be seen as having a manager from the Mixture
of Experts literature, by design, we cannot employ such an approach when
generalizing to an unseen domain in the zero-shot setting. This is due to the
fact that in ZS-MDG problems, we can neither assume access to a collection of
samples from the unseen domain, nor can we expect the manager model to
automatically withstand the complex distribution shifts.

##  8 Conclusion

In this work, we explored the benefits of domain re-labeling in zero-shot
multi-domain generalization. In particular, we considered two popular classes
of MDG algorithms, namely distributional robust optimization and multi-domain
ensembling, to demonstrate our hypothesis. While we utilized an explicit
clustering step to design GroupDRO++, through DReaME we proposed a meta-
optimization strategy to produce multi-domain ensembles with implicit re-
labeling. While the underlying MDG techniques and the strategies used for re-
labeling are vastly different across these two solutions, we find that both
the proposed approaches lead significant improvements in generalization
performance over their standard implementations with original domain labels
from the dataset. Using rigorous empirical studies on standard benchmarks, we
find that both GroupDRO++ and DReaME are able to outperform the ERM baseline,
and on challenging datasets, achieve state-of-the-art MDG performance.
Uncovering the theoretical underpinnings to understand the link between the
choice of domain groups and MDG performance, and designing generic re-labeling
strategies that can be readily integrated into any MDG method are crucial
future research directions.

## Acknowledgements

This work was performed under the auspices of the U.S. Department of Energy by
the Lawrence Livermore National Laboratory under Contract No. DE-
AC52-07NA27344, Lawrence Livermore National Security, LLC.

## References

  * caa (2020) Correlation-aware adversarial domain adaptation and generalization.  _Pattern Recognition_ , 100:107124, 2020.  ISSN 0031-3203.  doi: https://doi.org/10.1016/j.patcog.2019.107124.  URL https://www.sciencedirect.com/science/article/pii/S003132031930425X.
  * Akuzawa et al. (2019) Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo.  Adversarial invariant feature learning with accuracy constraint for domain generalization.  In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , pages 315–331. Springer, 2019.
  * Albuquerque et al. (2020) Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas.  Adversarial target-invariant representation learning for domain generalization.  2020\.
  * Arjovsky et al. (2019) Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  _arXiv preprint arXiv:1907.02893_ , 2019.
  * Ash et al. (2020) Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal.  Deep batch active learning by diverse, uncertain gradient lower bounds.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=ryghZJBKPS.
  * Bandi et al. (2018) Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al.  From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.  _IEEE transactions on medical imaging_ , 38(2):550–560, 2018.
  * Beery et al. (2018) Sara Beery, Grant Van Horn, and Pietro Perona.  Recognition in terra incognita.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 456–473, 2018.
  * Blanchard et al. (2017) G Blanchard, AA Deshmukh, U Dogan, et al.  Domain generalization by marginal transfer learning. arxiv preprint arxiv: 171107910.  2017\.
  * Blanchard et al. (2011) Gilles Blanchard, Gyemin Lee, and Clayton Scott.  Generalizing from several related classification tasks to a new unlabeled sample.  _Advances in neural information processing systems_ , 24:2178–2186, 2011.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 132–149, 2018.
  * Chattopadhyay et al. (2020) Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman.  Learning to balance specificity and invariance for in and out of domain generalization.  In _European Conference in Computer Vision (ECCV)_ , 2020.
  * Deng et al. (2018) Weijian Deng, Liang Zheng, Qixiang Ye, Guoliang Kang, Yi Yang, and Jianbin Jiao.  Image-image domain adaptation with preserved self-similarity and domain-dissimilarity for person re-identification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 994–1003, 2018.
  * Fang et al. (2013) Chen Fang, Ye Xu, and Daniel N Rockmore.  Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 1657–1664, 2013.
  * Finn et al. (2017a) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pages 1126–1135. PMLR, 2017a.
  * Finn et al. (2017b) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pages 1126–1135. PMLR, 2017b.
  * Ganin et al. (2016) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030, 2016.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  volume 13, pages 723–773. JMLR. org, 2012.
  * Gulrajani and Lopez-Paz (2021) Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=lQdXeXDoWtI.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hoffman et al. (2018) Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell.  Cycada: Cycle-consistent adversarial domain adaptation.  In _International conference on machine learning_ , pages 1989–1998. PMLR, 2018.
  * Huang et al. (2020) Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang.  Self-challenging improves cross-domain generalization.  _arXiv preprint arXiv:2007.02454_ , 2, 2020.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba.  Adam: A methodfor stochastic optimization.  In _International Conference onLearning Representations (ICLR)_ , 2015\.
  * Koh et al. (2021) Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR, 2021.
  * Krueger et al. (2021) David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville.  Out-of-distribution generalization via risk extrapolation (rex).  In _International Conference on Machine Learning_ , pages 5815–5826. PMLR, 2021.
  * Li et al. (2017) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales.  Deeper, broader and artier domain generalization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 5542–5550, 2017.
  * Li et al. (2018) Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales.  Learning to generalize: Meta-learning for domain generalization.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Li et al. (2019) Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales.  Episodic training for domain generalization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1446–1455, 2019.
  * Liu et al. (2021) Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn.  Just train twice: Improving group robustness without training group information.  In _International Conference on Machine Learning_ , pages 6781–6792. PMLR, 2021.
  * Mahajan et al. (2021) Divyat Mahajan, Shruti Tople, and Amit Sharma.  Domain generalization using causal matching.  In _International Conference on Machine Learning_ , pages 7313–7324. PMLR, 2021.
  * Mancini et al. (2018) Massimiliano Mancini, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci.  Best sources forward: domain generalization through source-specific nets.  In _2018 25th IEEE international conference on image processing (ICIP)_ , pages 1353–1357. IEEE, 2018.
  * Matsuura and Harada (2020) Toshihiko Matsuura and Tatsuya Harada.  Domain generalization using a mixture of multiple latent domains.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 11749–11756, 2020.
  * Nam et al. (2019) Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo.  Reducing domain gap via style-agnostic networks.  _arXiv preprint arXiv:1910.11645_ , 2019.
  * Park and Kwak (2016) Sungheon Park and Nojun Kwak.  Analysis on the dropout effect in convolutional neural networks.  In _Asian conference on computer vision_ , pages 189–204. Springer, 2016.
  * Pezeshki et al. (2021) Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie.  Gradient starvation: A learning proclivity in neural networks.  _Advances in Neural Information Processing Systems_ , 34:1256–1272, 2021.
  * Robey et al. (2021) Alexander Robey, George J. Pappas, and Hamed Hassani.  Model-based domain generalization.  _ArXiv_ , abs/2102.11436, 2021.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Sagawa* et al. (2020) Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang.  Distributionally robust neural networks.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=ryxGuJrFvS.
  * Seo et al. (2020) Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han.  Learning to optimize domain specific normalization for domain generalization.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_ , pages 68–83. Springer, 2020.
  * Standley et al. (2020) Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Which tasks should be learned together in multi-task learning?  In _International Conference on Machine Learning_ , pages 9120–9132. PMLR, 2020.
  * Sun and Saenko (2016) Baochen Sun and Kate Saenko.  Deep coral: Correlation alignment for deep domain adaptation.  In _European conference on computer vision_ , pages 443–450. Springer, 2016.
  * Torralba and Efros (2011) Antonio Torralba and Alexei A Efros.  Unbiased look at dataset bias.  In _CVPR 2011_ , pages 1521–1528. IEEE, 2011.
  * Vapnik (1999) Vladimir N Vapnik.  An overview of statistical learning theory.  _IEEE transactions on neural networks_ , 10(5):988–999, 1999.
  * Venkateswara et al. (2017) Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5018–5027, 2017.
  * Wang et al. (2021) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin.  Generalizing to unseen domains: A survey on domain generalization.  _CoRR_ , abs/2103.03097, 2021.
  * Wang and Deng (2018) Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  _Neurocomputing_ , 312:135–153, 2018.
  * Wang et al. (2020) Shujun Wang, Lequan Yu, Kang Li, Xin Yang, Chi-Wing Fu, and Pheng-Ann Heng.  Dofe: Domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets.  _IEEE Transactions on Medical Imaging_ , 39(12):4237–4248, 2020.
  * Xu et al. (2020) Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang.  Adversarial domain adaptation with domain mixup.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 6502–6509, 2020.
  * Xu et al. (2021) Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Marc Niethammer.  Robust and generalizable visual representation learning via random convolutions.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=BVSM0x3EDK6.
  * Yan et al. (2020) Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren.  Improve unsupervised domain adaptation with mixup training.  _arXiv preprint arXiv:2001.00677_ , 2020.
  * Zhang et al. (2020a) Cheng Zhang, Kun Zhang, and Yingzhen Li.  A causal view on robustness of neural networks.  _Advances in Neural Information Processing Systems_ , 33:289–301, 2020a.
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ , 2017.
  * Zhang et al. (2020b) Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn.  Adaptive risk minimization: A meta-learning approach for tackling group distribution shift.  _arXiv preprint arXiv:2007.02931_ , 2020b.
  * Zhou et al. (2021) Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=6xHJ37MVxxp.