# IW-GAE: Importance weighted group accuracy estimation for improved
calibration and model selection in unsupervised domain adaptation

Taejong Joo & Diego Klabjan
Department of Industrial Engineering & Management Sciences
Northwestern University
Evanston, IL, USA
<EMAIL_ADDRESS>
###### Abstract

Reasoning about a model’s accuracy on a test sample from its confidence is a
central problem in machine learning, being connected to important applications
such as uncertainty representation, model selection, and exploration. While
these connections have been well-studied in the i.i.d. settings, distribution
shifts pose significant challenges to the traditional methods. Therefore,
model calibration and model selection remain challenging in the unsupervised
domain adaptation problem–a scenario where the goal is to perform well in a
distribution shifted domain without labels. In this work, we tackle
difficulties coming from distribution shifts by developing a novel importance
weighted group accuracy estimator. Specifically, we formulate an optimization
problem for finding an importance weight that leads to an accurate group
accuracy estimation in the distribution shifted domain with theoretical
analyses. Extensive experiments show the effectiveness of group accuracy
estimation on model calibration and model selection. Our results emphasize the
significance of group accuracy estimation for addressing challenges in
unsupervised domain adaptation, as an orthogonal improvement direction with
improving transferability of accuracy.

##  1 Introduction

In this work, we consider a classification problem in unsupervised domain
adaptation (UDA). UDA aims to transfer knowledge from a source domain with
ample labeled data to enhance the performance in a target domain where labeled
data is unavailable. In UDA, the source and target domains have different data
generating distributions, so the core challenge is to transfer knowledge
contained in the labeled dataset in the source domain to the target domain
under the distribution shifts. Over the decades, significant improvements in
the transferability from source to target domains have been made, resulting in
areas like domain alignment (Ben-David et al.,, 2010; Ganin et al.,, 2016;
Long et al.,, 2018; Zhang et al.,, 2019) and self-training (Chen et al.,,
2020; Cai et al.,, 2021; Liu et al.,, 2021).

Improving calibration performance, which is about matching predictions
regarding a random event to the long-term occurrence of the event (Dawid,,
1982), is of central interest in the machine learning community due to its
significance to safe and trustworthy deployment of machine learning models in
critical real-world decision-making systems (Lee and See,, 2004; Amodei et
al.,, 2016). In independent and identically distributed (i.i.d.) settings,
calibration performance has been significantly improved by various approaches
(Guo et al.,, 2017; Gal and Ghahramani,, 2016; Lakshminarayanan et al.,,
2017). However, producing well-calibrated predictions in UDA remains
challenging due to the distribution shifts. Specifically, Wang et al., (2020)
show the discernible compromise in calibration performance as an offset
against the enhancement of target accuracy. A further observation reveals that
state-of-the-art calibrated classifiers in the i.i.d. settings begin to
generate unreliable uncertainty representation in the face of distributional
shifts (Ovadia et al.,, 2019). As such, enhancing the calibration performance
in UDA requires carefully addressing the impacts of the distribution shifts.

Moreover, the model selection task in UDA remains challenging due to the
scarcity of labeled target domain data that are required to evaluate model
performance. In the i.i.d. settings, a standard approach for model selection
is a cross-validation method—constructing a hold-out dataset for selecting the
model that yields the best performance on the hold-out dataset. While cross-
validation provides favorable statistical guarantees (Stone,, 1977; Kohavi et
al.,, 1995), such guarantees falter in the presence of the distribution shifts
due to the violation of the i.i.d. assumption. In practice, it has also been
observed that performances of machine learning models measured in one domain
have significant discrepancy to their performances in another distribution
shifted domain (Hendrycks and Dietterich,, 2019; Ovadia et al.,, 2019; Recht
et al.,, 2019). Therefore, applying model selection techniques in the i.i.d.
settings to the labeled source domain is suboptimal in the target domain.

This paper proposes importance weighted group accuracy estimation (IW-GAE)
that simultaneously addresses these critical aspects in UDA from a new
perspective of predicting a group accuracy (see Figure 1 for the graphical
illustration). We partition predictions into a set of groups and then estimate
the group accuracy–the average accuracy of predictions in a group–by
importance weighting. When the group accuracy estimate accurately represents
the expected accuracy of a model for individual samples in the group, using
the group accuracy estimate as prediction confidence induces a well-calibrated
classifier. When the average of the group accuracy estimates matches the
expected accuracy, the average group accuracy becomes a good model selection
criterion (cf. Figure 1). In this work, we formulate a novel optimization
problem for finding importance weights (IWs) that induce a group accuracy
estimator satisfying these ideal properties under the distribution shifts.
Specifically, we define two estimators for the group accuracy in the source
domain, where only one of them depends on the IW. Then, we find the IW that
makes the two estimators close to each other by solving the optimization
problem (cf. Figure 1). Through a theoretical analysis and several
experiments, we show that the optimization process results in an accurate
group accuracy estimator for the target domain, thereby improving model
calibration and model selection performances.

Figure 1: Figure 1 illustrates the idea of encouraging two estimators of the
source group accuracy close to each other. The shaded areas for the IW-based
estimator represent possible estimations under different IW values in the
confidence interval of the IW. We show that the quantity of interest–the error
of the IW-based estimator–is reduced by optimization in Section 4.1, which can
also reduce its target group accuracy estimation error. Figure 1 illustrates
the idea of estimating the group accuracy for model calibration and model
selection. Nine data points are assigned to three groups. For the model
calibration task, group 1 corresponds to the ideal case where the group
accuracy well represents the expected accuracy of the individual samples in
the group. However, group 2 has a high discrepancy between expected accuracies
of the samples in the group. In this case, even though the group accuracy
estimation is correct, the group accuracy estimation is not favorable for
model calibration. We prevent this case by group construction developed in
Section 4.2. Conversely, group 3 has a small variance of the expected accuracy
within the samples in the group. However, the estimator is not appropriate for
the model calibration task due to its high bias, which we prevent by solving
the novel optimization problem developed in Section 4.1. For the model
selection task, we aim to match mean group accuracy estimation (the blue
dotted line) to the mean expected accuracy (the red dotted line), which can be
induced by accurate group accuracy estimations for each group.

Our contributions can be summarized as follows: 1) We propose a novel
optimization problem for IW estimation that can directly reduce an error of
the quantity of interests in UDA with a theoretical analysis; 2) We show when
and why considering group accuracy, instead of the accuracy for individual
samples, is statistically favorable based on the bias-variance decomposition
of the group accuracy estimation error, which can simultaneously benefit model
calibration and model selection; 3) On average, IW-GAE improves state-of-the-
art by 26% in the model calibration task and 14% in the model selection task.

##  2 Related work

#### Model calibration in the i.i.d. settings

In a classification problem, the maximum value of the softmax output is often
considered as a confidence of a neural network’s prediction. In Guo et al.,
(2017), it is shown that the modern neural networks are poorly calibrated,
tending to produce larger confidences than their accuracies. Based on this
observation, Guo et al., (2017) introduce a post-processing approach that
adjusts a temperature parameter of the softmax function for adjusting the
overall confidence level. In Bayesian approaches (such as Monte-Carlo dropout
(Gal and Ghahramani,, 2016; Gal et al.,, 2017), deep ensemble
(Lakshminarayanan et al.,, 2017; Rahaman et al.,, 2021), and a last-layer
Bayesian approach (Sensoy et al.,, 2018; Joo et al.,, 2020)), the confidence
level adjustment is induced by posterior inference and model averaging. While
both post-hoc calibration methods and Bayesian methods have been achieving
impressive calibration performances in the i.i.d. setting (Maddox et al.,,
2019; Ovadia et al.,, 2019; Ebrahimi et al.,, 2020), it has been shown that
most of the calibration improvement methods fall short under distribution
shifts (Ovadia et al.,, 2019).

#### Model calibration in UDA

While handling model calibration problems under general distribution shifts is
challenging, the availability of unlabelled samples in the distribution
shifted target domain relaxes the difficulty in UDA. In particular, unlabeled
samples in the target domain enable an IW formulation for the quantity of
interests in the shifted domain. Therefore, the post-doc calibration methods
(e.g., Guo et al., (2017)) can be applied by reweighting calibration measures
such as the expected calibration error (Wang et al.,, 2020) and the Brier
score (Park et al.,, 2020) in the source dataset with an IW. However,
estimating the IW brings another difficulty of high-dimensional density
estimation. In this work, instead of concentrating on obtaining accurate
importance weighted calibration measures for matching the maximum softmax
output to the expected accuracy, we aim to directly estimate the accuracy in
the distribution shifted target domain.

#### Model selection in UDA

A standard procedure for model selection in the i.i.d. settings is the cross-
validation, which enjoys favorable statistical guarantees about bias and
variance of model performance (Stone,, 1977; Kohavi et al.,, 1995; Efron and
Tibshirani,, 1997). However, in UDA, the distribution shifts violate
assumptions for the statistical guarantees. Furthermore, in practice, the
accuracy measured in one domain is significantly changed in the face of
natural/adversarial distribution shifts (Goodfellow et al.,, 2015; Hendrycks
and Dietterich,, 2019; Ovadia et al.,, 2019). To tackle the distribution shift
problem, importance weighted cross validation (Sugiyama et al.,, 2007) applies
importance sampling for obtaining an unbiased estimate of model performance in
the distribution shifted target domain. Further, recent work in UDA controls
variance of the importance-weighted cross validation with a control variate
(You et al.,, 2019). These methods aim to accurately estimate the IW and then
use an IW formula for the expected accuracy estimation. In this work, our
method concerns the accuracy estimation error in the target domain during the
process of IW estimation, which can potentially induce an IW estimation error
but resulting in an accurate accuracy estimator.

##  3 Background

#### Notation and problem setup

Let ${\mathcal{X}}\subseteq\mathbb{R}^{r}$ and
${\mathcal{Y}}=[K]:=\\{1,2,\cdots,K\\}$ be input and label spaces. Let
$\hat{Y}:{\mathcal{X}}\rightarrow[K]$ be the prediction function of a model
and $Y(x)$ is a (conditional) $K$-dimensional categorical random variable
related to a label at $X=x$. When there is no ambiguity, we represent $Y(x)$
and $\hat{Y}(x)$ as $Y$ and $\hat{Y}$ for brevity. We are given a labeled
source dataset
${\mathcal{D}}_{S}=\\{(x^{(S)}_{i},y^{(S)}_{i})\\}_{i=1}^{N^{(S)}}$ sampled
from $p_{S_{XY}}$ and an unlabeled target dataset
${\mathcal{D}}_{T}=\\{x^{(T)}_{i}\\}_{i=1}^{N^{(T)}}$ sampled from $p_{T_{X}}$
where $p_{S_{XY}}$ is a joint data generating distribution of the source
domain and $p_{T_{X}}$ is a marginal data generating distribution of the
target domain. We also denote $\mathbb{E}_{p}[\cdot]$ as the population
expectation and $\hat{\mathbb{E}}_{p}[\cdot]$ as its empirical counterpart.
For $p_{S_{XY}}$ and $p_{T_{XY}}$, we consider a covariate shift without a
concept shift; i.e., $p_{S_{X}}(x)\neq p_{T_{X}}(x)$ but
$p_{S_{Y|X}}(y|x)=p_{T_{Y|X}}(y|x)$ for all $x\in{\mathcal{X}}$. For the rest
of the paper, we use the same notation for marginal and joint distributions
when there is no ambiguity; that is,
$\mathbb{E}_{p_{S}}[u_{1}(X)]=\mathbb{E}_{p_{S_{X}}}[u_{1}(X)]$ and
$\mathbb{E}_{p_{S}}[u_{2}(X,Y)]=\mathbb{E}_{p_{S_{XY}}}[u_{2}(X,Y)]$. However,
we use the explicit notation for the conditional distribution as $p_{S_{Y|X}}$
and $p_{T_{Y|X}}$ to avoid confusion.

In this work, we consider an IW estimation problem for improving model
calibration and model selection in UDA. Importance weighting can address many
problems in UDA due to its statistical exactness for dealing with two
different probability distributions under the absolute continuity condition
(Horvitz and Thompson,, 1952; Sugiyama et al.,, 2007) that is often assumed in
the literature. Specifically, for densities $p_{S}$ and $p_{T}$, a quantity of
interest $u(\cdot,\cdot)$ in $p_{T}$ can be computed by
$\mathbb{E}_{p_{T}}\left[u(X,Y)\right]=\mathbb{E}_{p_{S}}\left[w^{*}(X)u(X,Y)\right]$
where $w^{*}(x):=\frac{p_{T}(x)}{p_{S}(x)}$ is the IW of $x$. We next review
two main approaches for IW estimation, which circumvent the challenges of
directly estimating the IW, or the densities $p_{S}$ and $p_{T}$, in a high-
dimensional space.

#### Estimating IW by discriminative learning

Bickel et al., (2007) formulate IW estimation as a discriminative learning
problem by applying the Bayes’ rule, which is more sample efficient (Ng and
Jordan,, 2001; Tu,, 2007; Long and Servedio,, 2006). Specifically, with a
discriminative model that classifies source and target samples, the IW can be
computed as
$w^{*}(x)=\frac{p_{T}(x)}{p_{S}(x)}=\frac{\nu(x|d=1)}{\nu(x|d=0)}=\frac{P(d=0)}{P(d=1)}\frac{P(d=1|x)}{P(d=0|x)}$
where $\nu$ is a distribution over $(x,d)\in({\mathcal{X}}\times\\{0,1\\})$
and $d$ is a Bernoulli random variable indicating whether $x$ belongs to the
target domain or not. For IW estimation, ${P(d=0)}/{P(d=1)}$ can be estimated
by counting sample sizes of ${\mathcal{D}}_{S}$ and ${\mathcal{D}}_{T}$. Also,
to estimate ${P(d=1|x)}/{P(d=0|x)}$, a logistic regression model can be
trained by assigning a domain index of zero to $x_{S}\in{\mathcal{D}}_{S}$ and
one to $x_{T}\in{\mathcal{D}}_{T}$, and maximizing log-likelihood with respect
to the domain datasets.

#### Estimating confidence interval of importance weight

Recently, nonparametric estimation of the IW is proposed in the context of
generating a probably approximately correct (PAC) prediction set (Park et
al.,, 2022). In this approach, ${\mathcal{X}}$ is partitioned into $B$ number
of bins (${\mathcal{X}}=\cup_{i=1}^{B}{\mathcal{B}}_{i}$) with

$I^{(B)}:{\mathcal{X}}\rightarrow[B]\text{ such that
}{\mathcal{B}}_{i}=\\{x\in{\mathcal{X}}|I^{(B)}(x)=i\\},\quad i\in[B].$ (1)

Then, confidence intervals (CIs) of the binned probabilities
$\bar{p}_{S}(x)=\bar{p}_{S_{I^{(B)}(x)}}\text{ with
}\bar{p}_{S_{j}}=\int_{{\mathcal{B}}_{j}}p_{S}(x)dx$ and
$\bar{p}_{T}(x)=\bar{p}_{T_{I^{(B)}(x)}}\text{ with
}\bar{p}_{T_{j}}=\int_{{\mathcal{B}}_{j}}p_{T}(x)dx$ are constructed.
Specifically, for $\bar{p}_{S_{j}}$, the number of samples in a bin
$n^{(S)}_{j}:=\sum_{i=1}^{N^{(S)}}\textbf{1}(x^{(S)}_{i}\in{\mathcal{B}}_{j})$
is interpreted as a sample from $\text{Binom}(N^{(S)},\bar{p}_{S_{j}})$. Then,
the Clopper–Pearson CI (Clopper and Pearson,, 1934) provides the CI of
$\bar{p}_{S_{j}}$ as
$\underaccent{\bar}{\theta}_{S}(n^{(S)}_{j};N^{(S)},\delta/2)\leq\bar{p}_{S_{j}}\leq\bar{\theta}_{S}(n^{(S)}_{j};N^{(S)},\delta/2)$
with probability at least $1-\delta$ where
$\bar{\theta}(k;m,\delta):=\inf\\{\theta\in[0,1]|F(k;m,\theta)\leq\delta\\}$
and
$\underaccent{\bar}{\theta}(k;m,\delta):=\sup\\{\theta\in[0,1]|F(k;m,\theta)\geq\delta\\}$
with $F$ being the cumulative distribution function of the binomial
distribution. Similarly, we can obtain the CIs of $\bar{p}_{T_{j}}$ by
collecting
$n^{(T)}_{j}:=\sum_{i=1}^{N^{(T)}}\textbf{1}(x_{i}^{(T)}\in{\mathcal{B}}_{j})$
and following the same procedure, which are denoted as
$\underaccent{\bar}{\theta}_{T}(n^{(T)}_{j};N^{(T)},\delta/2)$ and
$\bar{\theta}_{T}(n^{(T)}_{j};N^{(T)},\delta/2)$.

With the CIs of $p_{S_{j}}$ and $p_{T_{j}}$ for $j\in[B]$, the CI of the IW in
${\mathcal{B}}_{j}$ can be obtained. Specifically, for
$\bar{\delta}:=\delta/2B$, the following inequality holds with probability at
least $1-\delta$ (Park et al.,, 2022):

$\frac{[\underaccent{\bar}{\theta}_{T}(n^{(T)}_{j};N^{(T)},\bar{\delta})-G]^{+}}{\bar{\theta}_{S}(n_{j}^{(S)};N^{(S)},\bar{\delta})+G}\leq
w_{j}^{*}:=\frac{\bar{p}_{T_{j}}}{\bar{p}_{S_{j}}}\leq\frac{\bar{\theta}_{T}(n^{(T)}_{j};N^{(T)},\bar{\delta})+G}{[\underaccent{\bar}{\theta}_{S}(n_{j}^{(S)};N^{(S)},\bar{\delta})-G]^{+}}$
(2)

where $G\in\mathbb{R}_{+}$ is a constant that satisfies
$\int_{{\mathcal{B}}_{j}}|p_{S}(x)-p_{S}(x^{\prime})|dx^{\prime}\leq G$ and
$\int_{{\mathcal{B}}_{j}}|p_{T}(x)-p_{T}(x^{\prime})|dx^{\prime}\leq G$ for
all $x\in{\mathcal{B}}_{j}$ and $j\in[B]$. For the rest of the paper, we refer
to $\\{w_{i}^{*}\\}_{i\in B}$ as binned IWs. Also, we let
$\Phi_{j}:=\left[\frac{[\underaccent{\bar}{\theta}_{T}(n^{(T)}_{j};N^{(T)},\bar{\delta})-G]^{+}}{\bar{\theta}_{S}(n_{j}^{(S)};N^{(S)},\bar{\delta})+G},\frac{\bar{\theta}_{T}(n^{(T)}_{j};N^{(T)},\bar{\delta})+G}{[\underaccent{\bar}{\theta}_{S}(n_{j}^{(S)};N^{(S)},\bar{\delta})-G]^{+}}\right]$
be the CI of $w^{*}_{j}$.

##  4 Importance weighted group accuracy estimation

In this section, we propose IW-GAE that estimates the group accuracy in the
target domain for addressing model calibration and selection tasks in UDA.
Specifically, we construct $M$ groups denoted by
$\\{{\mathcal{G}}_{i}\\}_{i\in[M]}$ and then estimate the average accuracy of
each group in the target domain with an IW. To this end, we define the target
group accuracy of a group ${\mathcal{G}}_{n}$ with the true IW $w^{*}$ as

$\alpha_{T}({\mathcal{G}}_{n};w^{*}):=\mathbb{E}_{p_{S}}\left[w^{*}(X)\textbf{1}(Y=\hat{Y})|X\in{\mathcal{G}}_{n}\right]\frac{P(X_{S}\in{\mathcal{G}}_{n})}{P(X_{T}\in{\mathcal{G}}_{n})}$
(3)

where $X_{S}$ and $X_{T}$ are random variables having densities $p_{S}$ and
$p_{T}$, respectively. It is called the group accuracy because
$\mathbb{E}_{p_{S}}\left[w^{*}(X)\textbf{1}(Y=\hat{Y})|X\in{\mathcal{G}}_{n}\right]\frac{P(X_{S}\in{\mathcal{G}}_{n})}{P(X_{T}\in{\mathcal{G}}_{n})}=\int_{x\in{\mathcal{X}}}\textbf{1}(Y=\hat{Y})\frac{p_{T}(x)\textbf{1}(x\in{\mathcal{G}}_{n})}{P(X_{T}\in{\mathcal{G}}_{n})}dx=\mathbb{E}_{p_{T}}\left[\textbf{1}(Y(X)=\hat{Y}(X))|X\in{\mathcal{G}}_{n}\right]$.
We denote $\hat{\alpha}_{T}({\mathcal{G}}_{n};w^{*})$ to be the expectation
with respect to the empirical measure. In Appendix B.1, we present a
sufficient condition that the group accuracy estimator is better than the
individual accuracy estimator under a relatively loose condition of group
constructions. We also define the source group accuracy as

$\alpha_{S}({\mathcal{G}}_{n};w^{*}):=\mathbb{E}_{p_{T}}\left[\frac{\textbf{1}(Y(X)=\hat{Y}(X))}{w^{*}(X)}\bigg{|}X\in{\mathcal{G}}_{n}\right]\frac{P(X_{T}\in{\mathcal{G}}_{n})}{P(X_{S}\in{\mathcal{G}}_{n})}.$
(4)

In the following, we explain IW estimation and group construction for
estimating the group accuracy. We denote an estimate for group accuracy with
an estimated IW $\hat{w}:{\mathcal{X}}\rightarrow\mathbb{R}_{+}$ and a group
assignment $I^{(g)}:{\mathcal{X}}\rightarrow[M]$ as
$\hat{\alpha}_{T}({\mathcal{G}}_{i};\hat{w})$ with
${\mathcal{G}}_{i}=\\{x\in{\mathcal{X}}:I^{(g)}(x)=i\\}$ for $i\in[M]$. We
discuss how to obtain $\hat{w}$ in Section 4.1 and $I^{(g)}$ in Section 4.2,
respectively.

After obtaining $\hat{w}$ and $I^{(g)}$,
$\hat{\alpha}_{T}({\mathcal{G}}_{i};\hat{w})$ can be used to perform the
following tasks: 1) Model calibration: For a test sample $x\sim p_{T}$,
$\hat{\alpha}_{T}({\mathcal{G}}_{I^{(g)}(x)};\hat{w})$ provides an estimate of
confidence, which is preferably close to
$\mathbb{E}_{p_{T_{Y|x}}}[\textbf{1}(Y(x)=\hat{Y}(x))]$; 2) Model selection:
For a validation dataset from the target domain ${\mathcal{D}}_{T}$, we
compare models based on
$\hat{\mathbb{E}}_{p_{T}}[\hat{\alpha}_{T}({\mathcal{G}}_{I^{(g)}(X)};\hat{w})]$.

###  4.1 Importance weight estimation

Our goal is to obtain IW $\hat{w}$ that leads to
$\alpha_{T}({\mathcal{G}}_{n};w^{*})\approx\alpha_{T}({\mathcal{G}}_{n};\hat{w})$
for $n\in[M]$. The proposed method is based on a CI estimation method
developed for producing the PAC prediction set discussed in Section 3 (Park et
al.,, 2022)111However, we want to emphasize that IW-GAE can be applied to any
valid CI estimators (cf. Appendix B.2).. Specifically, given the CI of binned
IWs $\\{\Phi_{i}\\}_{i\in B}$ in (2), our goal is to find binned IWs
$\\{w_{i}\in\Phi_{i}\\}_{i\in[B]}$ that give an accurate group accuracy
estimation. We let $\tilde{w}(x):=w_{I^{(B)}(x)}$ be the induced IW estimation
from binned IWs where $I^{(B)}$ is the partition in (1).

Our idea for accurately estimating the “target” group accuracy with IW
estimator $\tilde{w}$ is to define two estimators for the “source” group
accuracy defined in (4), with one estimator dependent on a target accuracy
estimate, and to encourage the two estimators to agree with each other. This
approach can be validated because the target accuracy estimation error of
$\tilde{w}$ can be upper bounded by its source accuracy estimation error (see
Appendix E.7 for empirical analyses); that is,

$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};\tilde{w})|=\bigg{|}\mathbb{E}_{p_{T}}\left[\tilde{w}(X)\left(\frac{1}{w^{*}(X)}-\frac{1}{\tilde{w}(X)}\right)\textbf{1}(Y=\hat{Y})\bigg{|}X\in{\mathcal{G}}_{n}\right]\bigg{|}\frac{P(X_{S}\in{\mathcal{G}}_{n})}{P(X_{T}\in{\mathcal{G}}_{n})}\\\
\leq\bar{M}\cdot|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};\tilde{w})|\left(\frac{P(X_{T}\in{\mathcal{G}}_{n})}{P(X_{S}\in{\mathcal{G}}_{n})}\right)^{2}$
(5)

where $\bar{M}=\max_{x\in Supp(p_{T})}\tilde{w}(x)$ and
$\alpha_{S}({\mathcal{G}}_{n};\tilde{w})$ is obtained by replacing $w^{*}$
with $\tilde{w}$ in (4).

To develop the first estimator, note that we can reliably approximate the
source group accuracy with ${\mathcal{D}}_{S}$ by Monte-Carlo estimation with
the error of ${\mathcal{O}}(1/\sqrt{|{\mathcal{G}}_{n}({\mathcal{D}}_{S})|})$
where
${\mathcal{G}}_{n}({\mathcal{D}}_{S}):=\\{(x_{k},y_{k})\in{\mathcal{D}}_{S}:x_{k}\in{\mathcal{G}}_{n}\\}$;
we denote the Monte-Carlo estimate as
$\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})=\hat{\mathbb{E}}_{p_{S}}[\textbf{1}(Y=\hat{Y})|X\in{\mathcal{G}}_{n}]$.

Based on (4), we define a second estimator for
$\alpha_{S}({\mathcal{G}}_{i};w^{*})$, as a function of binned IWs
$\\{w_{i}\\}_{i\in[B]}$, by assuming
$\mathbb{E}_{p_{T_{Y|x}}}[\textbf{1}(Y(x)=\hat{Y}(x))]=\hat{\alpha}_{T}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})$
for all $x\in{\mathcal{G}}_{n}$:

$\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]}):=\frac{\hat{P}(X_{T}\in{\mathcal{G}}_{n})}{\hat{P}(X_{S}\in{\mathcal{G}}_{n})}\cdot\hat{\mathbb{E}}_{p_{T}}\left[\frac{\hat{\alpha}_{T}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})}{\tilde{w}(X)}\bigg{|}X\in{\mathcal{G}}_{n}\right]\\\
=\hat{\mathbb{E}}_{p_{T}}\left[\frac{1}{\tilde{w}(X)}\bigg{|}X\in{\mathcal{G}}_{n}\right]\hat{\mathbb{E}}_{p_{S}}\left[\textbf{1}(Y=\hat{Y})\tilde{w}(X)\bigg{|}X\in{\mathcal{G}}_{n}\right]$
(6)

where $\hat{\alpha}_{T}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})$ is an
empirical estimate of the target accuracy with $\\{w_{i}\\}_{i\in[B]}$ in (3),
$\hat{P}(X_{T}\in{\mathcal{G}}_{n}):=\hat{E}_{p_{T}}[\textbf{1}(X\in{\mathcal{G}}_{n})]$,
and
$\hat{P}(X_{S}\in{\mathcal{G}}_{n}):=\hat{E}_{p_{S}}[\textbf{1}(X\in{\mathcal{G}}_{n})]$.

We aim to formulate an optimization problem to choose binned IWs from CIs such
that

$\min_{\\{w_{i}\in\Phi_{i}\\}_{i\in[B]}}(\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})-\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n}))^{2}.$
(7)

However, note that
$\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})$ in (6) is
non-convex with respect to $w_{i}$’s (see Appendix A.1 for the derivation),
which is in general not effectively solvable with optimization methods (Jain
et al.,, 2017). Therefore, we introduce a relaxed reformulation of (6) by
separating binned IWs for source and target, which introduces coordinatewise
convexity. Specifically, we redefine the estimator in (6) as

$\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]}):=\hat{\mathbb{E}}_{p_{T}}\left[\frac{1}{\tilde{w}^{(T)}(X)}\bigg{|}X\in{\mathcal{G}}_{n}\right]\hat{\mathbb{E}}_{p_{S}}\left[\textbf{1}(Y=\hat{Y})\tilde{w}^{(S)}(X)\bigg{|}X\in{\mathcal{G}}_{n}\right]$
(8)

where $\tilde{w}^{(S)}(X):=w^{(S)}_{I^{(B)}(X)}$ and
$\tilde{w}^{(T)}(X):=w^{(T)}_{I^{(B)}(X)}$. Then, we encourage agreements of
$w_{i}^{(S)}$ and $w_{i}^{(T)}$ for $i\in[B]$ through constraints.
Specifically, for each group ${\mathcal{G}}_{n}$, we find binned IWs
$w^{\dagger}(n)\in\mathbb{R}^{2B}_{+}$ by solving the following optimization
problem:

$\displaystyle
w^{\dagger}(n)\in\operatorname*{arg\,min}_{\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]}}$
$\displaystyle\quad\left(\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})-\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]})\right)^{2}$
(9) s.t. $\displaystyle\quad w_{i}^{(S)}\in\Phi_{i},\quad\text{for }i\in[B]$
(10) $\displaystyle\quad w_{i}^{(T)}\in\Phi_{i},\quad\text{for }i\in[B]$ (11)
$\displaystyle\quad\parallel
w_{i}^{(T)}-w_{i}^{(S)}\parallel_{2}^{2}\leq\delta^{(tol)}\quad\text{for
}i\in[B]$ (12)
$\displaystyle\quad\left|\hat{\mathbb{E}}_{p_{S}}[\tilde{w}^{(S)}(X)|X\in{\mathcal{G}}_{n}]-\frac{\hat{P}(X_{T}\in{\mathcal{G}}_{n})}{\hat{P}(X_{S}\in{\mathcal{G}}_{n})}\right|\leq\delta^{(prob)}$
(13)
$\displaystyle\quad\left|\hat{\mathbb{E}}_{p_{T}}[1/\tilde{w}^{(T)}(X)|X\in{\mathcal{G}}_{n}]-\frac{\hat{P}(X_{S}\in{\mathcal{G}}_{n})}{\hat{P}(X_{T}\in{\mathcal{G}}_{n})}\right|\leq\delta^{(prob)}$
(14)

where $\delta^{(tol)}$ and $\delta^{(prob)}$ are small constants. Box
constraints (10) and (11) ensure that the obtained solution is in the CI,
which bounds the estimation error of $w_{i}^{(S)}$ and $w_{i}^{(T)}$ by
$|\Phi_{i}|$. This can also bound the target group accuracy estimation error
as
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};\\{w^{(S)}_{i}\\}_{i\in
B})|\leq\max_{b\in[B]}|\Phi_{b}|P(X_{S}\in{\mathcal{G}}_{n})/P(X_{T}\in{\mathcal{G}}_{n})$.
Constraint (12) corresponds to the relaxation for removing non-convexity of
the original objective, and setting $\delta^{(tol)}=0$ recovers the original
objective. Constraints (13) and (14) are based on the equalities that the true
IW $w^{*}(\cdot)$ satisfies:
$\mathbb{E}_{p_{S}}[w^{*}(X)|X\in{\mathcal{G}}_{n}]=\frac{P(X_{T}\in{\mathcal{G}}_{n})}{P(X_{S}\in{\mathcal{G}}_{n})}$
and
$\mathbb{E}_{p_{T}}[1/w^{*}(X)|X\in{\mathcal{G}}_{n}]=\frac{P(X_{S}\in{\mathcal{G}}_{n})}{P(X_{T}\in{\mathcal{G}}_{n})}$.

Since the above optimization problem is a constrained nonlinear optimization
problem with box constraints, we solve it through sequential least square
programming (Kraft,, 1988). Note that the objective (9) is convex with respect
to the block $(w^{(S)}_{1},w^{(S)}_{2},\cdots,w^{(S)}_{B})$ and the block
$(w^{(T)}_{1},w^{(T)}_{2},\cdots,w^{(T)}_{B})$, but not jointly convex.
Therefore, using a quasi-Newton method can guarantee only convergence to a
local optimum (Nocedal and Wright,, 1999). Nevertheless, due to constraints
(13) and (14), the asymptotic convergence $(w^{\dagger}(n))_{i}\rightarrow
w_{i}^{*}$ and $(w^{\dagger}(n))_{i+B}\rightarrow w_{i}^{*}$ as
$N^{(S)}\rightarrow\infty$ and $N^{(T)}\rightarrow\infty$ can be trivially
guaranteed because $|\Phi_{i}|\rightarrow 0$ for $i\in[B]$ (Thulin,, 2014).
Values $w^{\dagger}(n)$ depend on $N^{(S)}$ and $N^{(T)}$ but we omit this
dependency in the notation.

The above optimization problem can be thought of as aiming to estimate the
truncated IW
$w(x|X\in{\mathcal{G}}_{n}):=\frac{p_{T}(x|X\in{\mathcal{G}}_{n})}{p_{S}(x|X\in{\mathcal{G}}_{n})}$
for each ${\mathcal{G}}_{n}$ that can induce an accurate source group accuracy
estimator. However, the objective in (9) does not measure the source group
accuracy estimation error. In the following proposition, we show that the
above optimization minimizes the upper bound of the source group accuracy
estimation error, thereby the target group accuracy estimation error due to
(5).

######  Proposition 4.1 (Upper bound of source group accuracy estimation
error).

Let $w^{\dagger}(n)$ be a solution to the optimization problem for
${\mathcal{G}}_{n}$ defined in (9)-(14) with $\delta^{(tol)}=0$ and
$\delta^{(prob)}=0$. Let
$\epsilon_{opt}(w^{\dagger}(n)):=\left(\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})-\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};w^{\dagger}(n))\right)^{2}$
be the objective value. For $\tilde{\delta}>0$, the following inequality holds
with probability at least $1-\tilde{\delta}$:

$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|\leq\epsilon_{opt}(w^{\dagger}(n))+\epsilon_{stat}+IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$
(15)

where
$\epsilon_{stat}\in{\mathcal{O}}(\log(1/{\tilde{\delta}}))/\sqrt{|{\mathcal{G}}_{n}({\mathcal{D}}_{S})|})$
and $IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})=$

$\frac{P(X_{T}\in{\mathcal{G}}_{n})}{2P(X_{S}\in{\mathcal{G}}_{n})}\left(\mathbb{E}_{p_{T}}\left[(\textbf{1}(Y(X)=\hat{Y}(X))-\alpha_{T}({\mathcal{G}}_{n};w^{*}))^{2}\big{|}X\in{\mathcal{G}}_{n}\right]+\frac{1}{\underaccent{\bar}{w}^{\dagger}(n)^{2}}\right)$
(16)

for
$\underaccent{\bar}{w}^{\dagger}(n):=\min_{i\in[2B]}\\{w^{\dagger}_{i}(n)\\}$.

The proof is based on the Cauchy-Schwarz inequality, which is provided in
Appendix A. Proposition 4.1 shows that we can reduce the source accuracy
estimation error by reducing $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$. In Section 4.2, we deal with
the design of $I^{(g)}$ that can reduce this term. Note that the number of
groups $M$ could highly influence $\epsilon_{stat}$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ (see the discussion in Appendix
B.3). However, we found that IW-GAE is not sensitive to even large changes in
$M$ (cf. Appendix E.6). Also, we empirically analyze the tightness of (15) in
Appendix E.7. Finally, for tightening the upper bounds in (5) and (15), we
bound the maximum and minimum values of IW, which is a common technique in IW-
based estimations (Wang et al.,, 2020; Park et al.,, 2022).

###  4.2 Group construction

In this section, we design a group assignment function $I^{(g)}$ for
constructing $\\{{\mathcal{G}}_{i}\\}_{i\in[M]}$ that can lead to an accurate
group accuracy estimation. Proposition 4.1 provides the bias of the identical
accuracy assumption. Now, we show that
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ depends on the bias of the
target group accuracy estimator and the variance of prediction correctness
within the group.

######  Proposition 4.2 (Bias-variance decomposition).

Let $\hat{\alpha}_{T}({\mathcal{G}}_{n})$ be an estimate for
$\alpha_{T}({\mathcal{G}}_{n};w^{*})$. Then, the bias of the identical
accuracy assumption is given by

$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})=\frac{P(X_{T}\in{\mathcal{G}}_{n})}{2P(X_{S}\in{\mathcal{G}}_{n})}\left(\frac{1}{\underaccent{\bar}{w}^{\dagger}(n)^{2}}+Bias(\hat{\alpha}_{T}({\mathcal{G}}_{n}))^{2}+Var(\textbf{1}(Y=\hat{Y})|{\mathcal{G}}_{n})\right)$
(17)

where
$Bias(\hat{\alpha}_{T}({\mathcal{G}}_{n})):=|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{T}({\mathcal{G}}_{n})|$
is the bias of the estimate $\hat{\alpha}_{T}({\mathcal{G}}_{n})$ and
$Var(\textbf{1}(Y=\hat{Y})|{\mathcal{G}}_{n}):=\mathbb{E}_{p_{T}}\left(\textbf{1}(Y=\hat{Y})-\alpha_{T}({\mathcal{G}}_{n};w^{*})\right)^{2}$
is the variance of the correctness of predictions in ${\mathcal{G}}_{n}$.

The proof is similar to the bias-variance decomposition technique in
statistics, and we defer it to Appendix A. Proposition 4.2 suggests forming a
group having a small variance in the correctness within the group. Further, a
lower variance would be beneficial for preventing potential misspecification
of the accuracy within a group (cf. Figure 1). Therefore, we group examples by
the maximum value of the softmax output based on strong empirical evidence
that the maximum value of the softmax output is highly correlated with
accuracy (Guo et al.,, 2017; Wang et al.,, 2020); i.e.,
${\mathcal{G}}_{n}({\mathcal{D}}):=\\{x_{i}\in{\mathcal{D}}|\frac{n-1}{M}\leq(\phi(g(x_{i})))_{j}\leq\frac{n}{M},j\in
m(x_{i})\\}$ for ${\mathcal{D}}\in\\{{\mathcal{D}}_{S},{\mathcal{D}}_{T}\\}$
and $n\in[M]$ where $g:{\mathcal{X}}\rightarrow\mathbb{R}^{K}$ is the logit
function of a neural network, $\phi$ is the $K$-dimensional softmax function,
and
$m(x_{i}):=\operatorname*{arg\,max}_{k\in[K]}\left(\phi(g(x_{i}))\right)_{k}$.

In addition, we use a separate criterion for grouping of source and target
samples inspired by results that an overall scale of the maximum value of the
softmax output significantly varies from one domain to another (Yu et al.,,
2022). Therefore, we introduce a temperature scale parameter for target
samples that adjusts the sharpness of the softmax output for target
predictions. This results in the nested optimization of
$\min_{t\in{\mathcal{T}}}\left\\{\text{optimization problem in
(\ref{eq:opt_prob})-(\ref{eq:const2}) with
${\mathcal{G}}_{n}({\mathcal{D}}_{S})$ and
${\mathcal{G}}_{n}^{(t)}({\mathcal{D}}_{T})$}\right\\}$ where
${\mathcal{G}}_{n}^{(t)}({\mathcal{D}}_{T}):=\\{x_{i}\in{\mathcal{D}}_{T}|\frac{n-1}{M}\leq(\phi(g(x_{i})/t))_{j}\leq\frac{n}{M},j\in
m(x_{i})\\}$ are the samples in ${\mathcal{G}}_{n}$ under the temperature-
scaled prediction (see Appendix C.1 for the complete optimization form). Based
on the facts that a group separation is not sensitive to small changes in the
temperature and the inner optimization is not smooth with respect to $t$, we
use a discrete set for ${\mathcal{T}}$; that is,
${\mathcal{T}}=\\{t_{1},t_{2},\cdots,t_{n}\\}$. We note that the inner
optimization problem is readily solvable, so the discrete optimization can be
performed without much additional computational overhead.

##  5 Experiments

We evaluate IW-GAE on model calibration and selection tasks. Since both tasks
are based on UDA classification tasks, we first provide the common setup and
task-specific setup such as the baselines and evaluation metrics in the
corresponding sections. For all experiments in this section, we evaluate our
method on the popular Office-Home (Venkateswara et al.,, 2017) dataset, which
contains around 15,000 images of 65 categories from four domains (Art,
Clipart, Product, Real-World). We also show robustness of our findings by
performing a large-scale experiment with the VisDa-2017 (Peng et al.,, 2017)
dataset containing around 280,000 images in Appendix E.1.

A base model is required for implementing the baseline methods and IW-GAE,
which serve as the test objectives for the model calibration and selection
tasks. We consider maximum mean discrepancy (MDD; (Zhang et al.,, 2019)) with
ResNet-50 (He et al.,, 2016) as the backbone neural network, which is the most
popular high-performing UDA method. MDD aims to learn domain invariant
representation while learning a classification task in the source domain. In
implementation, we use the popular open source project Transfer Learning
Library (Jiang et al.,, 2020). We use the default hyperparameters in all
experiments. Further details are explained in Appendix D.

IW estimation is required for implementing baseline methods and construct bins
for estimating the CI of the IW. We adopt a linear logistic regression model
on top of the neural network’s representation as the discriminative learning-
based estimation, following Wang et al., (2020). Specifically, it first
upsamples from one domain to make $|{\mathcal{D}}_{S}|=|{\mathcal{D}}_{T}|$,
and then it labels samples with the domain index:
$\\{(h(x),1)|x\in{\mathcal{D}}_{T}\\}$ and
$\\{(h(x),0)|x\in{\mathcal{D}}_{S}\\}$ where $h$ is the feature map of the
neural network. Then, logistic regression is trained with a quasi-Newton
method until convergence.

###  5.1 Model calibration performance

#### Setup & Metric

In this experiment, our goal is to match the confidence of a prediction to its
expected accuracy in the target domain. Following the standard (Guo et al.,,
2017; Park et al.,, 2020; Wang et al.,, 2020), we use expected calibration
error (ECE) on the test dataset as a measure of calibration performance. The
ECE measures the average absolute difference between the confidence and
accuracy of binned groups, which is defined as
$ECE({\mathcal{D}}_{T})=\sum_{n=1}^{m}\frac{|{\mathcal{G}}_{n}|}{|{\mathcal{D}}_{T}|}|\hat{\text{Acc}}({\mathcal{G}}_{n}({\mathcal{D}}_{T}))-\hat{\text{Conf}}({\mathcal{G}}_{n}({\mathcal{D}}_{T}))|$
where $\hat{\text{Acc}}({\mathcal{G}}_{n}({\mathcal{D}}_{T}))$ is the average
accuracy in ${\mathcal{G}}_{n}({\mathcal{D}}_{T})$ and
$\hat{\text{Conf}}({\mathcal{G}}_{n}({\mathcal{D}}_{T}))$ is the average
confidence in ${\mathcal{G}}_{n}({\mathcal{D}}_{T})$. We use $M=15$ following
the standard value (Guo et al.,, 2017; Wang et al.,, 2020).

#### Baselines

We consider the following five different baselines: The vanilla method uses a
maximum value of the softmax output as the confidence of the prediction. We
also consider temperature scaling-based methods that adjust the temperature
parameter by maximizing the following calibration measures: Temperature
scaling (TS) (Guo et al.,, 2017): the log-likelihood on the source validation
dataset; IW temperature scaling (IW-TS): the log-likelihood on the importance
weighted source validation dataset; Calibrated prediction with covariate shift
(CPCS): the Brier score (Brier,, 1950) on the importance weighted source
validation dataset; TransCal (Wang et al.,, 2020): the ECE on the importance
weighted source validation dataset with a bias and variance reduction
technique. These methods also use a maximum value of the (temperature-scaled)
softmax output as the confidence.

#### Results

As shown in Table 1, IW-GAE achieves the best average ECEs across different
base models. For individual domains, IW-GAE achieves the best ECE among 11 out
of the 12 cases. We note that IW-Mid, which selects the middle point in the CI
as IW estimation and originates herein, is a strong baseline, outperforming
other baselines. IW-GAE improves this strong baseline for every case. This
shows that the process of reducing $\epsilon_{opt}(w^{\dagger}(n))$ reduces
the group accuracy estimation error in the target domain, which is consistent
with the result in Proposition 4.1.

Task | Method | Ar-Cl | Ar-Pr | Ar-Rw | Cl-Ar | Cl-Pr | Cl-Rw | Pr-Ar | Pr-Cl | Pr-Rw | Rw-Ar | Rw-Cl | Rw-Pr | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Model | Vanilla | 40.61 | 25.62 | 15.56 | 33.83 | 25.34 | 24.75 | 33.45 | 38.62 | 16.76 | 23.37 | 36.51 | 14.01 | 27.37
calibration | TS | 35.86 | 22.84 | 10.60 | 28.24 | 20.74 | 20.06 | 32.47 | 37.20 | 14.89 | 18.36 | 34.62 | 12.28 | 24.01
| CPCS | 22.93 | 22.07 | 10.19 | 26.88 | 18.36 | 14.05 | 28.28 | 29.20 | 12.06 | 15.76 | 26.54 | 11.14 | 19.79
| IW-TS | 32.63 | 22.90 | 11.27 | 28.05 | 19.65 | 18.67 | 30.77 | 38.46 | 15.10 | 17.69 | 32.20 | 11.77 | 23.26
| TransCal | 33.57 | 20.27 | 8.88 | 26.36 | 18.81 | 18.42 | 27.35 | 29.86 | 10.48 | 16.17 | 29.90 | 10.00 | 20.84
| IW-Mid | 23.25 | 31.62 | 12.99 | 17.15 | 18.71 | 9.23 | 27.75 | 30.35 | 9.02 | 13.64 | 26.32 | 10.60 | 19.22
| IW-GAE | 12.78 | 4.70 | 12.93 | 7.52 | 4.42 | 4.11 | 9.50 | 17.49 | 8.40 | 7.62 | 9.52 | 8.14 | 8.93
| Oracle | 10.45 | 10.72 | 6.47 | 8.10 | 7.62 | 6.55 | 11.88 | 9.39 | 5.93 | 7.54 | 10.72 | 5.70 | 8.42
Model | Vanilla | 53.31 | 70.96 | 77.44 | 59.70 | 65.17 | 69.96 | 57.07 | 50.95 | 74.75 | 68.81 | 57.11 | 80.13 | 65.45
selection | IWCV | 53.24 | 69.61 | 72.50 | 59.70 | 65.17 | 67.50 | 57.07 | 55.21 | 74.75 | 68.81 | 58.51 | 80.13 | 65.18
| DEV | 53.31 | 70.72 | 77.44 | 59.79 | 67.99 | 69.96 | 57.07 | 52.50 | 77.12 | 70.50 | 53.38 | 82.27 | 66.00
| IW-Mid | 54.13 | 69.27 | 78.47 | 61.48 | 68.03 | 71.06 | 59.99 | 55.21 | 78.79 | 70.50 | 57.11 | 83.10 | 67.26
| IW-GAE | 54.34 | 70.96 | 78.47 | 61.48 | 69.93 | 71.06 | 62.79 | 55.21 | 78.79 | 70.50 | 58.51 | 83.31 | 67.95
| Lower bound | 52.51 | 69.27 | 72.50 | 59.70 | 65.17 | 67.50 | 57.07 | 50.95 | 74.75 | 68.81 | 50.90 | 80.13 | 64.10
| Oracle | 54.34 | 70.96 | 78.47 | 61.48 | 69.93 | 71.06 | 62.79 | 55.21 | 78.79 | 71.32 | 58.51 | 83.31 | 68.01

Table 1: Model calibration and selection benchmark results of MDD with
ResNet-50 on Office-Home. We repeat experiments ten times and report the
average value. For model calibration, the numbers indicate the mean ECE with
boldface for the minimum mean ECE. For model selection, the numbers indicate
the mean test accuracy of selected model with boldface for the maximum mean
test accuracy. For the model calibration task, Oracle is obtained by applying
TS with labeled test samples in the target domain. For the model selection
task, Lower bound and Oracle indicate the accuracy of the models with the
worst and best test accuracy, respectively.

###  5.2 Model Selection

#### Setup & Metric

In this experiment, we perform model selection for choosing the best
hyperparameter. To this end, we repeat training the MDD method by changing its
key hyperparameter of margin coefficient from 1 to 8 (the default value is 4).
After training several models under different values of the margin
coefficient, we choose one model based on a model selection criterion. For
evaluation, we compare the test target accuracy of the chosen model under
different model selection methods.

#### Baselines

We consider three baselines that evaluate the model’s performance in terms of
the following criterion: Vanilla: the minimum classification error on the
source validation dataset; Importance weighted cross validation (IWCV)
(Sugiyama et al.,, 2007): the minimum importance-weighted classification error
on the source validation dataset; Deep embedded validation (DEV)) (You et
al.,, 2019): the minimum deep embedded validation risk on the source
validation dataset.

#### Results

Table 1 shows that model selection with IW-GAE achieves the best average
accuracy, improving state-of-the-art by 18% in terms of the relative scale of
lower and upper bounds of accuracy. Specifically, IW-GAE achieves the best
performance in all cases. We also note that IW-Mid performs the model
selection task surprisingly well. This means that, on average, the true IW
could be located near the middle point of the CI, while the exact location
varies from one group to another.

Note that plain IWCV does not improve the vanilla method on average, which
could be due to the inaccurate estimation of the IW by the discriminative
learning-based approach. In this sense, IW-GAE has an advantage of depending
less on the performance of the IW estimator since the estimated value is used
to construct bins for the CI, and then the exact value is found by solving the
separate optimization problem. We also remark that our experimental results
reveal dangers of the current practice of using the vanilla method or IWCV in
model selection in UDA.

###  5.3 Additional experiments

To more rigorously evaluate IW-GAE, we perform the following additional
experiments and analyze the results in Appendix E. We apply IW-GAE to a
different base model called conditional adversarial domain adaptation (CDAN;
(Long et al.,, 2018)) in E.2. For these experiments, IW-GAE improves state-of-
the-art performances by 21% and 2%, respectively. In Appendix E.3, we perform
another model selection task of choosing the best checkpoint, and IW-GAE
improves the state-of-the-art performance by 9%. In Appendix E.4, we
qualitatively evaluate IW-GAE by visually comparing the group accuracy
estimation and the average group accuracy, which shows an accurate estimation
ability of IW-GAE. In Appendix E.5, we show that the group construction
criterion and nested optimization with temperature scaling developed in
Section 4.2 work effectively for IW-GAE. In Appendix E.6, a sensitivity
analysis shows that IW-GAE consistently performs well even under large changes
in the number of bins $B$ and the number of accuracy groups $M$. In Appendix
E.7, we analyze $\epsilon_{opt}(w^{\dagger}(n))$,
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$, and their impacts on source and
target group accuracy estimation errors. The analysis shows a strong
correlation between the target and source group accuracy estimation errors. In
addition, we found that $\epsilon_{opt}(w^{\dagger}(n))$ is strongly
correlated with the source group accuracy estimation error. These results
support the idea of minimizing $\epsilon_{opt}(w^{\dagger}(n))$ for obtaining
an accurate target group accuracy estimator.

##  6 Conclusion

In this work, we formulate an optimization problem to choose IW estimation
from its CI for accurately estimating group accuracy. Specifically, we define
a Monte-Carlo estimator and an IW-based estimator of group accuracy in the
source domain and find the IW that makes the two estimators close to each
other. Solving the optimization problem not only reduces the source group
accuracy estimation error but also leads to an accurate group accuracy
estimation in the target domain. We show that our method achieves state-of-
the-art performances in both model calibration and selection tasks in UDA
across a wide range of benchmark problems. We believe that the impressive
performance gains by our method show a promising future direction of research,
which is orthogonal to improving the transferability of accuracy–the main
focus in the UDA literature. Finally, we remark that applying IW-GAE and other
baselines to the large-language models (XLM-R (Conneau et al.,, 2019) and
GPT-2 (Solaiman et al.,, 2019)) did not result in improvements. We conjecture
that the pre-trained large-language model is less subject to the distribution
shifts, so applying the methods in the i.i.d. settings can serve as a strong
baseline. We leave this as an important future direction of research.

## Acknowledgement

We would like to thank Jihyeon Hyeong, Yuchen Lou, and Jiezhong Wu for
insightful discussions and helpful suggestions in writing the manuscript.

## References

  *   * Amodei et al., (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016).  Concrete problems in AI safety.  arXiv preprint arXiv:1606.06565.
  * Ben-David et al., (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. (2010).  A theory of learning from different domains.  Machine Learning, 79:151–175.
  * Bhatia and Davis, (2000) Bhatia, R. and Davis, C. (2000).  A better bound on the variance.  The American Mathematical Monthly, 107(4):353–357.
  * Bickel et al., (2007) Bickel, S., Brückner, M., and Scheffer, T. (2007).  Discriminative learning for differing training and test distributions.  In International Conference on Machine Learning.
  * Brier, (1950) Brier, G. W. (1950).  Verification of forecasts expressed in terms of probability.  Monthly Weather Review, 78(1):1–3.
  * Cai et al., (2021) Cai, T., Gao, R., Lee, J., and Lei, Q. (2021).  A theory of label propagation for subpopulation shift.  In International Conference on Machine Learning.
  * Chen et al., (2020) Chen, Y., Wei, C., Kumar, A., and Ma, T. (2020).  Self-training avoids using spurious features under domain shift.  In Advances in Neural Information Processing Systems.
  * Clopper and Pearson, (1934) Clopper, C. J. and Pearson, E. S. (1934).  The use of confidence or fiducial limits illustrated in the case of the binomial.  Biometrika, 26(4):404–413.
  * Conneau et al., (2019) Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. (2019).  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * Dawid, (1982) Dawid, A. P. (1982).  The well-calibrated Bayesian.  Journal of the American Statistical Association, 77(379):605–610.
  * Ebrahimi et al., (2020) Ebrahimi, S., Elhoseiny, M., Darrell, T., and Rohrbach, M. (2020).  Uncertainty-guided continual learning with Bayesian neural networks.  In International Conference on Learning Representations.
  * Efron and Tibshirani, (1997) Efron, B. and Tibshirani, R. (1997).  Improvements on cross-validation: the 632+ bootstrap method.  Journal of the American Statistical Association, 92(438):548–560.
  * Gal and Ghahramani, (2016) Gal, Y. and Ghahramani, Z. (2016).  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning, pages 1050–1059.
  * Gal et al., (2017) Gal, Y., Hron, J., and Kendall, A. (2017).  Concrete dropout.  In Advances in Neural Information Processing Systems.
  * Ganin et al., (2016) Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016).  Domain-adversarial training of neural networks.  Journal of Machine Learning Research, 17(1):2096–2030.
  * Goodfellow et al., (2015) Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015).  Explaining and harnessing adversarial examples.  In International Conference on Learning Representations.
  * Guo et al., (2017) Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017).  On calibration of modern neural networks.  In International Conference on Machine Learning.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition.
  * Hendrycks and Dietterich, (2019) Hendrycks, D. and Dietterich, T. (2019).  Benchmarking neural network robustness to common corruptions and perturbations.  In International Conference on Learning Representations.
  * Horvitz and Thompson, (1952) Horvitz, D. G. and Thompson, D. J. (1952).  A generalization of sampling without replacement from a finite universe.  Journal of the American Statistical Association, 47(260):663–685.
  * Jain et al., (2017) Jain, P., Kar, P., et al. (2017).  Non-convex optimization for machine learning.  Foundations and Trends® in Machine Learning, 10(3-4):142–363.
  * Jiang et al., (2020) Jiang, J., Chen, B., Fu, B., and Long, M. (2020).  Transfer-learning-library.  https://github.com/thuml/Transfer-Learning-Library.
  * Joo et al., (2020) Joo, T., Chung, U., and Seo, M.-G. (2020).  Being Bayesian about categorical probability.  In International Conference on Machine Learning, pages 4950–4961.
  * Kohavi et al., (1995) Kohavi, R. et al. (1995).  A study of cross-validation and bootstrap for accuracy estimation and model selection.  In International Joint Conferences on Artificial Intelligence.
  * Kraft, (1988) Kraft, D. (1988).  A software package for sequential quadratic programming.  Forschungsbericht- Deutsche Forschungs- und Versuchsanstalt fur Luft- und Raumfahrt.
  * Lakshminarayanan et al., (2017) Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2017).  Simple and scalable predictive uncertainty estimation using deep ensembles.  In Advances in Neural Information Processing Systems.
  * Lee and See, (2004) Lee, J. D. and See, K. A. (2004).  Trust in automation: Designing for appropriate reliance.  Human Factors, 46(1):50–80.
  * Liu et al., (2021) Liu, H., Wang, J., and Long, M. (2021).  Cycle self-training for domain adaptation.  In Advances in Neural Information Processing Systems.
  * Long et al., (2018) Long, M., Cao, Z., Wang, J., and Jordan, M. I. (2018).  Conditional adversarial domain adaptation.  In Advances in Neural Information Processing Systems.
  * Long and Servedio, (2006) Long, P. M. and Servedio, R. A. (2006).  Discriminative learning can succeed where generative learning fails.  In Annual Conference on Learning Theory.
  * Maddox et al., (2019) Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. (2019).  A simple baseline for Bayesian uncertainty in deep learning.  In Advances in Neural Information Processing Systems.
  * Ng and Jordan, (2001) Ng, A. and Jordan, M. (2001).  On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes.  In Advances in Neural Information Processing Systems.
  * Nocedal and Wright, (1999) Nocedal, J. and Wright, S. J. (1999).  Numerical Optimization.  Springer.
  * Ovadia et al., (2019) Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. (2019).  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In Advances in Neural Information Processing Systems.
  * Park et al., (2020) Park, S., Bastani, O., Weimer, J., and Lee, I. (2020).  Calibrated prediction with covariate shift via unsupervised domain adaptation.  In International Conference on Artificial Intelligence and Statistics.
  * Park et al., (2022) Park, S., Dobriban, E., Lee, I., and Bastani, O. (2022).  PAC prediction sets under covariate shift.  In International Conference on Learning Representations.
  * Peng et al., (2017) Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., and Saenko, K. (2017).  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924.
  * Popoviciu, (1965) Popoviciu, T. (1965).  Sur certaines inégalités qui caractérisent les fonctions convexes.  Analele Stiintifice Univ.“Al. I. Cuza”, Iasi, Sectia Mat, 11:155–164.
  * Rahaman et al., (2021) Rahaman, R. et al. (2021).  Uncertainty quantification and deep ensembles.  In Advances in Neural Information Processing Systems.
  * Recht et al., (2019) Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019).  Do imagenet classifiers generalize to imagenet?  In International Conference on Machine Learning.
  * Russakovsky et al., (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015).  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 115:211–252.
  * Salvador et al., (2021) Salvador, T., Voleti, V., Iannantuono, A., and Oberman, A. (2021).  Frustratingly easy uncertainty estimation for distribution shift.  arXiv preprint arXiv:2106.03762.
  * Sensoy et al., (2018) Sensoy, M., Kaplan, L., and Kandemir, M. (2018).  Evidential deep learning to quantify classification uncertainty.  In Advances in Neural Information Processing Systems.
  * Sharma et al., (2010) Sharma, R., Gupta, M., and Kapoor, G. (2010).  Some better bounds on the variance with applications.  Journal of Mathematical Inequalities, 4(3):355–363.
  * Solaiman et al., (2019) Solaiman, I., Brundage, M., Clark, J., Askell, A., Herbert-Voss, A., Wu, J., Radford, A., Krueger, G., Kim, J. W., Kreps, S., et al. (2019).  Release strategies and the social impacts of language models.  arXiv preprint arXiv:1908.09203.
  * Stone, (1977) Stone, M. (1977).  Asymptotics for and against cross-validation.  Biometrika, pages 29–35.
  * Sugiyama et al., (2007) Sugiyama, M., Krauledat, M., and Müller, K.-R. (2007).  Covariate shift adaptation by importance weighted cross validation.  Journal of Machine Learning Research, 8(5).
  * Thulin, (2014) Thulin, M. (2014).  The cost of using exact confidence intervals for a binomial proportion.  Electronic Journal of Statistics, 8:817–840.
  * Tishby et al., (2000) Tishby, N., Pereira, F. C., and Bialek, W. (2000).  The information bottleneck method.  arXiv preprint physics/0004057.
  * Tishby and Zaslavsky, (2015) Tishby, N. and Zaslavsky, N. (2015).  Deep learning and the information bottleneck principle.  In 2015 IEEE Information Theory Workshop (itw), pages 1–5.
  * Tu, (2007) Tu, Z. (2007).  Learning generative models via discriminative approaches.  In IEEE Conference on Computer Vision and Pattern Recognition.
  * Venkateswara et al., (2017) Venkateswara, H., Eusebio, J., Chakraborty, S., and Panchanathan, S. (2017).  Deep hashing network for unsupervised domain adaptation.  In IEEE Conference on Computer Vision and Pattern Recognition.
  * Virtanen et al., (2020) Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, İ., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors (2020).  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods, 17:261–272.
  * Wang et al., (2020) Wang, X., Long, M., Wang, J., and Jordan, M. (2020).  Transferable calibration with lower bias and variance in domain adaptation.  In Advances in Neural Information Processing Systems.
  * You et al., (2019) You, K., Wang, X., Long, M., and Jordan, M. (2019).  Towards accurate model selection in deep unsupervised domain adaptation.  In International Conference on Machine Learning.
  * Yu et al., (2022) Yu, Y., Bates, S., Ma, Y., and Jordan, M. (2022).  Robust calibration with multi-domain temperature scaling.  In Advances in Neural Information Processing Systems.
  * Zhang et al., (2019) Zhang, Y., Liu, T., Long, M., and Jordan, M. (2019).  Bridging theory and algorithm for domain adaptation.  In International Conference on Machine Learning.

##  Appendix A Proof of claims

###  A.1 Non-convexity of the optimization problem

Let
${\mathcal{G}}_{n}({\mathcal{D}}_{S}):=\\{(x_{k},y_{k})\in{\mathcal{D}}_{S}:x_{k}\in{\mathcal{G}}_{n}\\}$
and
${\mathcal{G}}_{n}({\mathcal{D}}_{T}):=\\{x_{k}\in{\mathcal{D}}_{T}:x_{k}\in{\mathcal{G}}_{n}\\}$
for $n\in[M]$. By elementary algebra, we obtain the following

$\displaystyle\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})$
$\displaystyle=\left(\frac{1}{|{\mathcal{G}}_{n}({\mathcal{D}}_{T})|}\sum_{(x,y))\in{\mathcal{G}}_{n}({\mathcal{D}}_{T})}\frac{1}{w(x)}\right)\left(\frac{1}{|{\mathcal{G}}_{n}({\mathcal{D}}_{S})|}\sum_{x\in{\mathcal{G}}_{n}({\mathcal{D}}_{S})}\textbf{1}(y=\hat{Y}(x))w(x)\right)$
(18)
$\displaystyle=\left(\frac{1}{|{\mathcal{G}}_{n}({\mathcal{D}}_{T})|}\sum_{i=1}^{B}\frac{a_{i}}{w_{i}}\right)\left(\frac{1}{|{\mathcal{G}}_{n}({\mathcal{D}}_{S})|}\sum_{i=1}^{B}b_{i}w_{i}\right)$
(19)

where $a_{i}=|{\mathcal{G}}_{n}({\mathcal{D}}_{T})\cap{\mathcal{B}}_{i}|$ and
$b_{i}=|\\{(x_{k},y_{k})\in{\mathcal{G}}_{n}({\mathcal{D}}_{S})\cap{\mathcal{B}}_{i}|y_{k}=\hat{Y}(x_{k})\\}|$.
Therefore, $\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}\\}_{i\in[B]})$
is non-convex with respect to $w_{i}$ for $i\in[B]$.

###  A.2 Proof of Proposition 4.1

###### Proof.

By applying the triangle inequalities, we get the following inequality:

$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|\leq|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})|+|\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})-\hat{\alpha}^{(IW)}_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|\\\
+|\hat{\alpha}^{(IW)}_{S}({\mathcal{G}}_{n};w^{\dagger}(n))-{\alpha}^{(IW)}_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|+|\alpha^{(IW)}_{S}({\mathcal{G}}_{n};w^{\dagger}(n))-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|.$
(20)

Note that the first and third terms in the right hand side are coming from the
Monte-Carlo approximation, so they can be bounded by
${\mathcal{O}}(\log(1/{\tilde{\delta}})/|{\mathcal{D}}_{n}^{S}|)$ with
probability at least $1-{\tilde{\delta}}$ based on a concentration inequality
such as the Hoeffding’s inequality. Also, the second term is bounded by the
optimization error $\epsilon_{opt}(w^{\dagger}(n))$. Therefore, it is enough
to analyze the fourth term.

The fourth term is coming from the bias of
$\mathbb{E}_{T_{{Y|X}}}[\textbf{1}(Y(X)=\hat{Y}(X))]=\hat{\alpha}_{T}({\mathcal{G}}_{n};w^{\dagger}(n))$,
which we refer to as the bias of the identical accuracy assumption. It can be
bounded by

$\displaystyle|\alpha^{(IW)}_{S}({\mathcal{G}}_{n};w^{\dagger}(n))-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$
(21)
$\displaystyle=\frac{P(X_{T}\in{\mathcal{G}}_{n})}{P(X_{S}\in{\mathcal{G}}_{n})}\bigg{|}\mathbb{E}_{p_{T}}\left[\frac{\textbf{1}(Y(X)=\hat{Y}(X))-\hat{\alpha}_{T}({\mathcal{G}}_{n};w^{\dagger}(n))}{w^{\dagger}(n)(X)}\big{|}X\in{\mathcal{G}}_{n}\right]\bigg{|}$
(22)
$\displaystyle\leq\frac{P(X_{T}\in{\mathcal{G}}_{n})}{P(X_{S}\in{\mathcal{G}}_{n})}\left(\mathbb{E}_{p_{T}}\left[(\textbf{1}(Y(X)=\hat{Y}(X))-\hat{\alpha}_{T}({\mathcal{G}}_{n};w^{\dagger}(n)))^{2}\big{|}X\in{\mathcal{G}}_{n}\right]\mathbb{E}_{p_{T}}\left[\frac{1}{w^{\dagger}(n)(X)^{2}}\big{|}X\in{\mathcal{G}}_{n}\right]\right)^{1/2}$
(23)
$\displaystyle\leq\frac{P(X_{T}\in{\mathcal{G}}_{n})}{2P(X_{S}\in{\mathcal{G}}_{n})}\left(\mathbb{E}_{p_{T}}\left[(\textbf{1}(Y(X)=\hat{Y}(X))-\hat{\alpha}_{T}({\mathcal{G}}_{n};w^{\dagger}(n)))^{2}\big{|}X\in{\mathcal{G}}_{n}\right]+\mathbb{E}_{p_{T}}\left[\frac{1}{w^{\dagger}(n)(X)^{2}}\big{|}X\in{\mathcal{G}}_{n}\right]\right)$
(24)
$\displaystyle\leq\frac{P(X_{T}\in{\mathcal{G}}_{n})}{2P(X_{S}\in{\mathcal{G}}_{n})}\left(\mathbb{E}_{p_{T}}\left[(\textbf{1}(Y=\hat{Y})-\alpha_{T}({\mathcal{G}}_{n};w^{*}))^{2}\big{|}{\mathcal{G}}_{n}\right]+\frac{1}{\underaccent{\bar}{w}^{\dagger}(n)^{2}}\right)$
(25)

where (23) holds due to the Cauchy-Schwarz inequality, (24) holds due to the
AM-GM inequality, and
$\underaccent{\bar}{w}^{\dagger}(n):=\min_{i\in[2B]}\\{w^{\dagger}_{i}(n)\\}$.
∎

###  A.3 Proof of Proposition 4.2

###### Proof.

Based on the proof of Proposition 4.1, it is enough to decompose
$\mathbb{E}_{p_{T}}\left(\textbf{1}(Y(X)=\hat{Y}(X))-\hat{\alpha}_{T}({\mathcal{G}}_{n})\right)^{2}$
as follows

$\displaystyle\mathbb{E}_{p_{T}}\left(\textbf{1}(Y(X)=\hat{Y}(X))-\hat{\alpha}_{T}({\mathcal{G}}_{n})\right)^{2}$
(26)
$\displaystyle=\mathbb{E}_{p_{T}}\left(\textbf{1}(Y(X)=\hat{Y}(X))-\alpha_{T}({\mathcal{G}}_{n};w^{*})+\alpha_{T}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{T}({\mathcal{G}}_{n})\right)^{2}$
(27)
$\displaystyle=\mathbb{E}_{p_{T}}\left[\left(\textbf{1}(Y(X)=\hat{Y}(X))-\alpha_{T}({\mathcal{G}}_{n};w^{*})\right)^{2}+\left(\alpha_{T}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{T}({\mathcal{G}}_{n})\right)^{2}\right]$
(28)
$\displaystyle=\mathbb{E}_{p_{T}}\left(\textbf{1}(Y(X)=\hat{Y}(X))-\alpha_{T}({\mathcal{G}}_{n};w^{*})\right)^{2}+\left(\alpha_{T}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{T}({\mathcal{G}}_{n})\right)^{2}$
(29)

where (28) holds due to
$\mathbb{E}_{T_{X}}\mathbb{E}_{T_{Y|X}}\left[(\textbf{1}(Y(X)=\hat{Y}(X))-\alpha_{T}({\mathcal{G}}_{n};w^{*}))(\alpha_{T}({\mathcal{G}}_{n};w^{*})-\hat{\alpha}_{T}({\mathcal{G}}_{n}))\right]=0$.
∎

##  Appendix B Discussions

###  B.1 On idea of estimating group accuracy

Our idea of predicting group accuracy, instead of an expected accuracy for
each sample, can be motivated by a bias and variance decomposition analysis.
Suppose we are given samples $\\{x_{i}\\}_{i=1}^{N_{n}}\in{\mathcal{G}}_{n}$
and a classifier $f$. Let $\beta(x):=\mathbb{E}_{Y|X}[\textbf{1}(Y(x)=f(x))]$
be an expected accuracy of a classifier $f$ at $x$, which is our goal to
estimate. Let $D=\\{(x,y)|y\sim Y|x,x\in{\mathcal{G}}_{n}\\}$ be random
samples with the realization of a label, which is the case in most machine
learning cases. Then, the observed accuracy is
$\hat{\beta}(x):=\textbf{1}(y=f(x))=\beta(x)+\epsilon_{x}$ where
$\epsilon_{x}$ is random noise with zero mean and variance $\sigma^{2}$. Note
that the random noise $\epsilon_{x}$ exists unless the label is a
deterministic function of $x$. While this might be true in the raw input
space, this is highly unlikely in the representation space where most
estimations are usually performed unless the representation contains all
information about the input random variable $X$ conditional on the label
random variable $Y$ (cf. (Tishby et al.,, 2000; Tishby and Zaslavsky,, 2015)).

In this setting, the expected mean-square error of an estimator $g(x)$ for
$\beta(x)$ at $x_{i}\in{\mathcal{G}}_{n}$ with respect to the realization of a
label $y_{i}\sim Y|x$ can be decomposed by

$\mathbb{E}_{D,\epsilon}[(\hat{\beta}(x_{i})-g(x_{i}))^{2}]=Var_{D}(g(x_{i};D))+(Bias_{D}(g(x_{i};D)))^{2}+\sigma^{2}_{x_{i}}$
(30)

where
$Var_{D}(g(x_{i};D)):=\mathbb{E}_{D}[(g(x_{i};D)-\mathbb{E}_{D}[g(x_{i};D)])^{2}]$
is the variance of the estimator and
$Bias_{D}(g(x_{i};D)):=\mathbb{E}_{D}[g(x_{i};D)]-\beta(x)$ is the bias of the
estimator.

With the bias-variance decomposition in (30), we compare two maximum
likelihood estimators (MLEs) where one predicts the expected accuracy at each
$x_{i}$ for $i\in[N_{n}]$ and the other one predicts the group accuracy for
all $x_{i}$ for $i\in[N_{n}]$ in terms of the expected mean-square accuracy
estimation error.

#### Individual accuracy estimator:

Let $g_{1}(x;D)$ be an estimator that predicts an accuracy at sample $x$ given
$D$. For this case, an MLE estimator is $g_{1}(x)=\hat{\beta}(x)$, which is
unbiased because $\mathbb{E}_{D}(g_{1}(x;D))=\beta(x)$ for each
$x\in{\mathcal{G}}_{n}$. Therefore, this estimator has the average of expected
errors

$\frac{1}{N_{n}}\sum_{k=1}^{N_{n}}\mathbb{E}_{D,\epsilon}[(\hat{\beta}(x_{k})-g_{1}(x_{k};D))^{2}]=\bar{\sigma}^{2}+\bar{\sigma}^{2}$
(31)

where $\bar{\sigma}^{2}:=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\sigma^{2}_{x_{i}}$.

#### Group accuracy estimator:

Let $g_{2}(x;D)$ be a group accuracy estimator that predicts the same group
accuracy estimate for all $x\in{\mathcal{G}}_{n}$. An MLE estimator can be
defined by $g_{2}(x;D)=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\hat{\beta}(x_{i})$,
which is a biased estimator because
$\mathbb{E}_{D}(g_{2}(x;D))=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\beta(x_{i})$ for
each $x\in{\mathcal{G}}_{n}$. Therefore, this estimator has the average of
expected errors

$\displaystyle\frac{1}{N_{n}}\sum_{k=1}^{N_{n}}\mathbb{E}_{D,\epsilon}[(\hat{\beta}(x_{k})-g_{2}(x_{k}))^{2}]$
$\displaystyle=\frac{1}{N_{n}}\bar{\sigma}^{2}+\frac{1}{N_{n}}\sum_{k=1}^{N_{n}}\left(\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\beta(x_{i})-\beta(x_{k})\right)^{2}+\bar{\sigma}^{2}$
(32)
$\displaystyle=\frac{1}{N_{n}}\bar{\sigma}^{2}+Var(\beta;D)+\bar{\sigma}^{2}$
(33)

where $Var(\beta;D)$ is the variance of the accuracy in group
${\mathcal{G}}_{n}$.

#### Comparison of the two estimators:

The Popoviciu’s inequality (Popoviciu,, 1965) provides a sufficient condition
for the group accuracy estimator $g_{2}$ to have a lower expected mean-squared
error than the individual accuracy estimator $g_{1}$ as follows:

$\frac{1}{4}\left(\max_{x^{\prime}\in{\mathcal{G}}_{n}}\beta(x^{\prime})-\min_{x^{\prime}\in{\mathcal{G}}_{n}}\beta(x^{\prime})\right)^{2}\leq\frac{N_{n}-1}{N_{n}}\bar{\sigma}^{2}=\frac{N_{n}-1}{N_{n}}\left(\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\beta(x_{i})(1-\beta(x_{i}))\right)$
(34)

where the equality comes from
$\bar{\sigma}^{2}=\frac{1}{N_{n}}\sum_{i=1}^{N_{n}}\sigma^{2}_{x_{i}}$ with
$\sigma^{2}_{x_{i}}$ is the variance of the Bernoulli distribution with a
parameter $\beta(x_{i})$.

Recall that $\bar{\sigma}^{2}$ is due to the realization of only a single
label at each sample, so it cannot be controlled unless we gather more labels
of $x_{i}$ or removing the stochasticity of a label by learning the perfect
representation. However, we can reduce the term
$\max_{x\in{\mathcal{G}}_{n}}\beta(x)-\min_{x\in{\mathcal{G}}_{n}}\beta(x)$ by
constructing a group in a creative way discussed in Section 4.2. We also note
that the sufficient condition is somewhat loose as we can see in Figure A1,
which means that the group accuracy estimator could be better than the
individual accuracy estimator in various cases. We finally note that the
inequality can be further sharpened, i.e., the sufficient condition can be
further relaxed, with tighter upper bounds of the variance (Bhatia and Davis,,
2000; Sharma et al.,, 2010).

Figure A1: The shaded area includes values of group accuracy disparity
satisfying the sufficient condition that the group accuracy estimator $g_{2}$
outperforms the individual group accuracy estimator $g_{1}$.

###  B.2 On choice of non-parametric estimators

Our concept of determining the IW from its CI can be applied with any other
valid CI estimators. For example, by analyzing a CI of the odds ratio of the
logistic regression used as a domain classifier (Bickel et al.,, 2007; Park et
al.,, 2020; Salvador et al.,, 2021), a CI of the IW can be obtained. Then, IW-
GAE can be applied in the same way as developed in Section 4. In this regard,
advancements in IW estimation or CI estimation would be beneficial for
accurately estimating the group accuracy, thereby model selection and
uncertainty estimation. Therefore, we leave combining IW-GAE with advanced IW
estimation techniques as an important future direction of research.

###  B.3 On choice of the number of groups

In this work, we estimate the group accuracy by grouping predictions based on
the confidence of the prediction. Therefore, a natural question to ask is how
to select the number of groups. If we use a small number of groups, then there
would be high $IdentBias(w^{\dagger};{\mathcal{G}}_{n})$ because of the large
variance of prediction correctness within a group. In addition, reporting the
same accuracy estimate for a large number of predictions could be inaccurate
in terms of representing uncertainty for individual predictions. Conversely,
if we use a large number of bins, there would be high Monte-Carlo
approximation errors, $\epsilon_{stat}$. Therefore, it would result in a loose
connection between the source group accuracy estimation error and the
objective in the optimization problem (cf. Proposition 4.1). Therefore, it is
important to choose a proper number of bins.

##  Appendix C Additional details

###  C.1 The nested optimization problem under the temperature scaling in the
target domain

$\displaystyle(t^{\dagger},w^{\dagger}(n))\in\operatorname*{arg\,min}_{t\in{\mathcal{T}}}\operatorname*{arg\,min}_{\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]}}$
$\displaystyle\quad\left(\hat{\alpha}_{S}^{(MC)}({\mathcal{G}}_{n})-\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]})\right)^{2}$
(35) s.t. $\displaystyle\quad w_{i}^{(S)}\in\Phi_{i},\quad\text{for }i\in[B]$
(36) $\displaystyle\quad w_{i}^{(T)}\in\Phi_{i},\quad\text{for }i\in[B]$ (37)
$\displaystyle\quad\parallel
w_{i}^{(T)}-w_{i}^{(S)}\parallel_{2}^{2}\leq\delta^{(tol)}\quad\text{for
}i\in[B]$ (38)
$\displaystyle\quad\left|\hat{\mathbb{E}}_{p_{S}}[\tilde{w}^{(S)}(X)|X\in{\mathcal{G}}_{n}]-\frac{\hat{P}(X_{T}\in{\mathcal{G}}_{n}^{(t)})}{\hat{P}(X_{S}\in{\mathcal{G}}_{n})}\right|\leq\delta^{(prob)}$
(39)
$\displaystyle\quad\left|\hat{\mathbb{E}}_{p_{T}}[1/\tilde{w}^{(T)}(X)|X\in{\mathcal{G}}_{n}^{(t)}]-\frac{\hat{P}(X_{S}\in{\mathcal{G}}_{n})}{\hat{P}(X_{T}\in{\mathcal{G}}_{n}^{(t)})}\right|\leq\delta^{(prob)}$
(40)

In the above optimization problem, we note that temperature scaling also
changes the IW based estimator by

$\hat{\alpha}_{S}^{(IW)}({\mathcal{G}}_{n};\\{w_{i}^{(S)},w_{i}^{(T)}\\}_{i\in[B]})=\hat{\mathbb{E}}_{p_{T}}\left[\frac{1}{\tilde{w}^{(T)}(X)}\bigg{|}X\in{\mathcal{G}}_{n}^{(t)}\right]\hat{\mathbb{E}}_{p_{S}}\left[\textbf{1}(Y=\hat{Y})\tilde{w}^{(S)}(X)\bigg{|}X\in{\mathcal{G}}_{n}\right].$
(41)

##  Appendix D Experimental details

We follow the exact same training configurations as those used in the Transfer
Learning Library, except we separate 20% as the validation dataset from the
source domain (in the original implementation, validation is performed with
the test dataset for Office-Home).

The configuration of training MDD for Office-Home is as follows: MDD is
trained for 30 epochs with SGD with momentum parameter 0.9 and weight decay of
$0.0005$. The learning rate is schedule by $\alpha\cdot(1+\gamma\cdot
t)^{-\eta}$ where $t$ is the iteration counter, $\alpha=0.004$,
$\gamma=0.0002$, $\eta=0.75$, and the stochastic gradient is computed with
minibatch of 32 samples from the source domain and 32 samples from the target
domain. Also, it uses the margin coefficient of 4 as the MDD-specific
hyperparamter. For the model architecture, it uses ResNet-50 pre-trained on
ImageNet (Russakovsky et al.,, 2015) with the bottleneck dimension of 2,048.

#### IW-GAE specific details

For CI estimation, we follow the same configuration with the original method
(Park et al.,, 2022). Specifically, we use constant $G=0.001$, CI level
$\bar{\delta}=0.05$, and the number of bins $B=10$. In addition, we use the
maximum IW value $\bar{M}=6.0$ and the minimum IW value
$\underaccent{\bar}{w}^{\dagger}(n)=1/6$ for $n\in[M]$ for tightening the
upper bounds in (5) and (15) (cf. Section 4.1).

In addition, for IW-GAE, we use the constraint relaxation constants
$\delta^{(tol)}=0.1$ and $\delta^{(prob)}=0.3$. We also use the number of
accuracy group $M=10$ and analyze the sensitivity for $M$ in Appendix E.6. For
implementing sequential least square programming, we use the SciPy Library
(Virtanen et al.,, 2020) with tolerance $10^{-8}$ that is used to check a
convergence condition (other optimizer-specific values follow the default
values in SciPy) and choose the middle points from CIs of binned IW as an
initial solution. For nested optimization, we set
${\mathcal{T}}:=\\{0.85,0.90,0.95,1.00,1.05,1.10\\}$.

##  Appendix E Additional experiments

###  E.1 Large-scale image classification task

We perform an additional large-scale experiment with the VisDA-2017 (Peng et
al.,, 2017) containing around 280,000 images of 12 categories from two domains
(real and synthetic images). For this experiment, we use ResNet-101 with the
bottleneck dimension of 1,024, following the default setting in the Transfer
Learning Library. All other configurations are the same as the training
configuration of the Office-Home dataset (cf. Appendix D). Figure A2
represents the result of the VisDa-2017 experiment, which shows that the IW-
GAE achieves the best performance among all baselines. Specifically, IW-GAE
reduces the ECE of the state-of-the-art (TransCal) by 21%. We note that the
IW-Mid achieves a comparable performance with TransCal, unlike the OfficeHome
experiment. This result could be explained by an accurate CI estimation under
a large number of samples. We leave analyzing the impacts of the number of
samples on the CI estimation and the performance of IW-GAE as an important
future direction of research, which could provide a strategy for choosing
which CI estimator to use under which condition.

Figure A2: Large-scale model calibration benchmark result of MDD with
ResNet-101 on VisDA 2017. The numbers indicates the mean ECE across ten
repetitions with boldface for the minimum mean ECE.

###  E.2 CDAN

In this section, we perform additional experiments with the conditional domain
adversarial network (Long et al.,, 2018) which is also a popular UDA method.
As in the experiments with MDD, we use ResNet-50 as the backbone network and
OfficeHome as the dataset. The learning rate schedule for CDAN is
$\alpha\cdot(1+\gamma\cdot t)^{-\eta}$ where $t$ is the iteration counter,
$\alpha=0.01$, $\gamma=0.001$, and $\eta=0.75$. The remaining training
configuration for CDAN is the same as the MDD training configuration except it
uses the bottleneck dimension of 256 and weight decay of $0.0005$ (cf.
Appendix D). As we can see from Table A1, IW-GAE achieves the best performance
among all considered methods, achieving the best ECE in 8 out of the 12 cases
as well as the lowest mean ECE. We note that TransCal achieves a performance
comparable to IW-GAE in this experiment, but considering the results in the
other tasks, IW-GAE is still an appealing method for performing the model
calibration task.

Method | Ar-Cl | Ar-Pr | Ar-Rw | Cl-Ar | Cl-Pr | Cl-Rw | Pr-Ar | Pr-Cl | Pr-Rw | Rw-Ar | Rw-Cl | Rw-Pr | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Vanilla | 30.73 | 18.38 | 14.37 | 25.63 | 22.44 | 19.10 | 27.54 | 36.72 | 12.48 | 19.93 | 31.12 | 10.88 | 22.44
TS | 29.68 | 19.40 | 14.40 | 22.15 | 19.97 | 16.88 | 28.82 | 38.03 | 12.99 | 20.46 | 31.91 | 11.83 | 22.21
CPCS | 18.78 | 18.09 | 14.74 | 22.18 | 20.74 | 16.33 | 29.30 | 34.92 | 11.92 | 20.99 | 31.41 | 11.07 | 20.87
IW-TS | 12.38 | 16.79 | 14.85 | 21.75 | 20.06 | 16.92 | 29.30 | 38.84 | 13.30 | 20.82 | 31.10 | 11.37 | 20.62
TransCal | 7.94 | 14.05 | 12.91 | 7.82 | 9.25 | 10.23 | 9.37 | 12.60 | 14.29 | 9.92 | 9.76 | 17.51 | 11.30
IW-Mid | 36.05 | 47.70 | 26.82 | 21.08 | 22.95 | 21.55 | 18.88 | 28.99 | 15.39 | 21.16 | 28.16 | 25.27 | 26.17
IW-GAE | 13.98 | 29.82 | 9.44 | 6.55 | 5.59 | 10.16 | 5.29 | 13.47 | 11.01 | 11.12 | 7.26 | 9.84 | 11.13
Oracle | 7.91 | 8.80 | 6.05 | 7.57 | 7.93 | 6.76 | 9.07 | 9.14 | 4.04 | 7.16 | 9.19 | 5.65 | 7.44

Table A1: Model calibration benchmark results of CDAN with ResNet-50 on
Office-Home. The numbers indicates the mean ECE across ten repetitions with
boldface for the minimum mean ECE.

###  E.3 Checkpoint selection

In this section, we perform the task of choosing the best checkpoint during
training for examining IW-GAE’s model selection performance. Specifically, we
first train MDD on the OfficeHome dataset for 30 epochs and save the
checkpoint at the end of each epoch. Then, we choose the best checkpoint based
on IW-GAE and baselines described in Section 5.2. As shown in Table A2, the
model selected based on IW-GAE achieves the best average test accuracy, which
is consistent with the results in the hyperparameter selection task (cf. the
model selection task in Table 1). Specifically, IW-GAE improves the second-
best method (IWCV) by 9% and achieves the best checkpoint selection for 3 out
of the 12 domains.

Method | Ar-Cl | Ar-Pr | Ar-Rw | Cl-Ar | Cl-Pr | Cl-Rw | Pr-Ar | Pr-Cl | Pr-Rw | Rw-Ar | Rw-Cl | Rw-Pr | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Vanilla | 47.22 | 74.14 | 77.76 | 61.85 | 70.96 | 71.59 | 60.98 | 53.63 | 78.93 | 71.57 | 57.04 | 83.96 | 67.47
IWCV | 54.46 | 74.22 | 72.27 | 61.48 | 70.49 | 70.62 | 61.30 | 51.13 | 78.37 | 72.94 | 58.43 | 84.00 | 67.48
DEV | 54.04 | 73.94 | 78.16 | 61.52 | 63.19 | 70.7 | 60.43 | 53.63 | 78.93 | 71.57 | 58.62 | 83.89 | 67.39
IW-Mid | 54.04 | 72.63 | 78.37 | 62.05 | 71.28 | 71.45 | 61.25 | 54.39 | 79.07 | 73.19 | 58.75 | 80.06 | 68.04
IW-GAE | 54.32 | 73.98 | 78.51 | 61.96 | 71.25 | 71.70 | 61.10 | 54.30 | 78.91 | 73.22 | 58.70 | 83.86 | 68.48
Lower bound | 41.90 | 64.88 | 72.27 | 52.00 | 58.48 | 62.13 | 53.52 | 38.33 | 70.92 | 63.41 | 44.81 | 75.83 | 58.21
Oracle | 54.80 | 74.79 | 78.61 | 62.46 | 71.59 | 72.18 | 61.64 | 54.64 | 79.44 | 73.42 | 59.43 | 84.12 | 68.93

Table A2: Checkpoint selection benchmark results of MDD with ResNet-50 on
Office-Home. The numbers indicate the mean test accuracy of selected model
across ten repetitions with boldface for the maximum mean test accuracy.

###  E.4 Qualitative evaluation of IW-GAE

To qualitatively analyze IW-GAE, we also visualize reliability curves that
compare the estimated group accuracy with the average accuracy in Figure A3.
We first note that IW-GAE tends to accurately estimate the true group accuracy
for most groups under different cases compared to IW-Mid. The accurate group
accuracy estimation behavior of IW-GAE explains the results that the IW-GAE
improves IW-Mid for most cases in the model calibration and selection tasks
(cf. Table 1). For most cases, true accuracy is in between the lower and upper
IW estimators, albeit the interval length tends to increase for high-
confidence groups. This means that the CI of the IW based on the Clopper-
Pearson method successfully captures the IW in the CI. We also note that the
true accuracy is close to the lower IW estimator in the lower confidence group
and the middle IW estimator in the high confidence group. An observation that
the true accuracy’s relative positions in CIs varies from one group to another
group motivates why an adaptive selection of binned IWs as ours is needed.

Figure A3: True group accuracy and estimated group accuracy of IW-GAE and IW-
Mid under MDD. The shaded areas represent possible group accuracy estimation
with binned IWs in the CI. The title of a figure represents “Source-Target.”
For IW-Mid and IW-GAE, we clip the accuracy estimations when they exceed 1,
which can occur when the upper bound of CI is large. Also, the number of
groups in the figure is different for some domains because there can be a
group that contains no target samples (we set $M=10$ for all cases).

###  E.5 Ablation study

In this section, we perform an ablation study of our key design choices for
group construction (cf. Section 4.2). The first ablation study examines group
construction based on the maximum value of the softmax output by constructing
a group function based on IW. The second ablation study examines the
effectiveness of our nested optimization with temperature scaling (cf. Section
4.2), by excluding the outer optimization in the nested optimization; i.e.,
setting ${\mathcal{T}}=\\{1\\}$ in Appendix C.1. Specifically, we repeat the
model calibration experiment with MDD on four randomly selected domains in the
OfficeHome dataset (Ar-Pr, Pr-Cl, Rw-Cl, Rw-Pr).

Table A3 presents the results of the ablation study. First, note that the
grouping by the maximum value of softmax significantly impacts the performance
of IW-GAE. If we assume the group accuracy estimation ability of IW-GAE is not
significantly reduced after changing the grouping function $I^{(g)}$, the
reduction in the performance could be due to a large variance of prediction
accuracy within a group (cf. the case of group 2 in Figure 1). Specifically,
the large value of $Var(\textbf{1}(Y=\hat{Y})|{\mathcal{G}}_{n})$ increases
the $IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$, which can loosen the upper
bound of the source group accuracy estimation error in (15). Given its
significant impact on the calibration performance, we want to remark few
challenging aspects of developing an ideal group function for future work.
Specifically, note that the core factor in
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ impacted by $I^{(g)}$ is
$Var(\textbf{1}(Y=\hat{Y})|{\mathcal{G}}_{n})$. This term depends on the
labeled information in the target domain, so it is hard to foretell changes in
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ as we change $I^{(g)}$.
Furthermore, even if we have a labeled dataset in the target domain, finding
the optimal $I^{(g)}$ that minimizes
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ is a combinatorial optimization
problem, which is one of the most challenging optimization problems.

Next, note that nested optimization results in improvement of the ECE on
average but it is effective only in four out of eight cases. This is
surprising because the nested optimization problem is guaranteed to reduce
$\epsilon_{opt}(w^{\dagger}(n))$ as ${\mathcal{T}}$ contains the case that
corresponds to the setting without the nested optimization;
$1\in{\mathcal{T}}$. This motivates our further investigation of the
relationship between $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ in Appendix E.7. In a nutshell,
we found the cases when reducing $\epsilon_{opt}(w^{\dagger}(n))$ increases
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$. Therefore, it can increase the
upper bound of a source group accuracy estimation error in (15). We also found
a (weak) correlation between $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$, which can explain the average
improvement by the nested optimization problem. We present more details in
Appendix E.7.

Method | Ar-Pr | Pr-Cl | Rw-Cl | Rw-Pr | Avg
---|---|---|---|---|---
IW-GAE w/ grouping by IW and w/o the nested optimization | 15.07 | 36.02 | 35.01 | 5.23 | 22.83
IW-GAE w/ grouping by IW | 14.18 | 34.67 | 35.08 | 5.30 | 22.31
IW-GAE w/o the nested optimization | 11.43 | 16.57 | 9.38 | 5.91 | 10.82
IW-GAE | 4.70 | 17.49 | 9.52 | 8.14 | 9.97

Table A3: An ablation study of key design choices of constructing a group by
the maximum value of the softmax and using nested optimization with
temperature scaling on the target domain for IW-GAE. We use MDD with ResNet-50
on Office-Home. The numbers indicate the mean ECE across ten repetitions.

Figure A4: Sensitivity analysis of IW-GAE with respect to the number of groups
$M$ and the number of bins $B$ (b) on four domains (Ar-Pr, Pr-Cl, Rw-Cl, Rw-
Pr) in the OfficeHome dataset. Default hyperparameters for $M$ and $B$ are 10,
and we normalize the accuracy for each domain by its performance under the
default hyperparameters. Average represents the average relative accuracy for
each value of the hyperparameter and the shaded areas represent areas between
the minimum and the maximum relative accuracy over the 4 domains.

###  E.6 Sensitivity analysis

In this section, we conduct a sensitivity analysis with respect to our key
hyperparameters of the number of accuracy groups $M$ and the number of bins
$B$. As in the ablation study in Appendix E.5, we perform the model
calibration experiment with MDD on the four domains in the OfficeHome dataset
(Ar-Pr, Pr-Cl, Rw-Cl, Rw-Pr). Figure A4 shows the sensitivity analysis results
with hyperparameter values $M\in[4,27]$ and $B\in[4,27]$ (the default value
for both $M$ and $B$ is 10). Within the search space, the average performances
do not change more than 10%, which means that IW-GAE would outperform state-
of-the-art (cf. Table 1) also under such altered settings. Also, we can see
that the performance changes under different hyperparameter values are
somewhat stable; the maximum and minimum changes are within the range of 10%
for most cases, even though a large variance in the performances appears for
extreme values such as $M=4$, $B=4$, and $B=27$. The results show the
robustness of IW-GAE with respect to small changes in the key hyperparameter
values.

###  E.7 Analysis of $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ of IW and their relation to
source and target group accuracy estimation errors

In this section, we aim to answer the following question about the central
idea of this work: “Does solving the optimization problem in (9)-(14) result
in an accurate target group accuracy estimator?” Specifically, we analyze the
relationship between the optimization error $\epsilon_{opt}(w^{\dagger}(n))$,
the bias of the identical accuracy assumption
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$, the source group accuracy
estimation error
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$,
and the target group accuracy estimation error
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$
from the perspective of (5) and (15)222Technically speaking, the computed
values in this experiment are the empirical expectation which can contain a
statistical error. However, since we have no access to the data generating
distribution, we perform the analysis as if these values are the population
expectations.. To this end, we gather $w^{\dagger}(n)$ obtained by solving the
optimization problem under all temperature parameters in the search space
$t\in{\mathcal{T}}$ with MDD on the OfficeHome dataset (720 IWs from 6 values
of the temperature parameter, 12 cases, and 10 groups). Then, by using the
test dataset in the source and the target domains, we obtain the following
observations.

In (5), we show that
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$
is upper bounded by
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$.
However, the inequality could be loose since the inequality is obtained by
taking the maximum over the IW values. Considering that the optimization
problem is formulated for finding $w^{\dagger}(n)$ that achieves small
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$
(cf. Proposition 4.1), the loose connection between the source and target
group accuracy estimation errors can potentially enlighten a fundamental
difficulty to our approach. However, as we can see from Figure A5, it turns
out that
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$
is strongly correlated with
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$.
This result validates our approach of reducing the source accuracy estimation
error of the IW-based estimator for obtaining an accurate group accuracy
estimator in the target domain.

Figure A5:  The relationship between source and target group accuracy
estimation errors.

In (15), we show that
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|\leq\epsilon_{opt}(w^{\dagger}(n))+\epsilon_{stat}+IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$,
which motivates us to solve the optimization problem for reducing
$\epsilon_{opt}(w^{\dagger}(n))$ (cf. Section 4.1) and to construct groups
based on the maximum value of softmax for reducing
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ (cf. Section 4.2). Again, if
these terms are loosely connected to
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$,
a fundamental difficulty arises for our approach. In this regard, we analyze
the relationship between $\epsilon_{opt}(w^{\dagger}(n))$,
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$, and
$|\alpha_{S}({\mathcal{G}}_{n};w^{*})-\alpha_{S}({\mathcal{G}}_{n};w^{\dagger}(n))|$.
From Figure A6, we can see that both $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ are strongly correlated to the
source group accuracy estimation error. Combined with the observation in
Figure A5, this observation explains the impressive performance gains by IW-
GAE developed for reducing $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$.

Figure A6:  The relationship between between $\epsilon_{opt}(w^{\dagger}(n))$
and the source group accuracy estimation error (left) and the relationship
between $IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ and the source group
accuracy estimation error (right).

Next, we analyze the efficacy of solving the optimization problem for
obtaining an accurate target group accuracy estimator. To this end, we analyze
the relationship between $\epsilon_{opt}(w^{\dagger}(n))$ and
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$.
From Figure A7, $\epsilon_{opt}(w^{\dagger}(n))$ is correlated with
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$,
which explains the performance gains in the model calibration and selection
tasks by IW-GAE. However, the correlation is weaker than the cases analyzed in
Figure A5 and Figure A6. We conjecture that this is because
$\epsilon_{opt}(w^{\dagger}(n))$ is connected to
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$
through two inequalities (5) and (15), and this results in a somewhat loose
connection between $\epsilon_{opt}(w^{\dagger}(n))$ and
$|\alpha_{T}({\mathcal{G}}_{n};w^{*})-\alpha_{T}({\mathcal{G}}_{n};w^{\dagger}(n))|$.

In Figure A7, we also note that the optimization problem is subject to a non-
identifiability issue that the solutions with the same optimization error can
have significantly different target group accuracy estimation errors (e.g.,
points achieving the zero optimization error in Figure A7). We remark that the
non-identifiability issue motivates an important future direction of research
that develops a more sophisticated objective function and a regularization
function that can distinguish estimators with different target group accuracy
estimation errors.

Finally, we analyze the relationship between $\epsilon_{opt}(w^{\dagger}(n))$
and $IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ in Figure A8. In general, we
can see a weak correlation between $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$. This means that, in general,
finding a better solution in terms of $\epsilon_{opt}(w^{\dagger}(n))$ could
also reduce $IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$. However, there are
many cases in which different IWs have the same
$\epsilon_{opt}(w^{\dagger}(n))$ but significantly different
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$. Furthermore, reducing
$\epsilon_{opt}(w^{\dagger}(n))$ increases
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$ for some cases. This observation
explains small performance gains by the nested optimization problem as seen in
the ablation study (cf. Appendix E.5).

Figure A7: The relationship between $\epsilon_{opt}(w^{\dagger}(n))$ and the
target group accuracy estimation error.
Figure A8: The relationship between $\epsilon_{opt}(w^{\dagger}(n))$ and
$IdentBias(w^{\dagger}(n);{\mathcal{G}}_{n})$.