# Selective classification using a robust meta-learning approach

Nishant Jain Pradeep Shenoy
Google Research India
<EMAIL_ADDRESS>
###### Abstract

Selective classification involves identifying the subset of test samples that
a model can classify with high accuracy, and is important for applications
such as automated medical diagnosis. We argue that this capability of
identifying uncertain samples is valuable for training classifiers as well,
with the aim of building more accurate classifiers. We unify these dual roles
by training a single auxiliary meta-network to output an importance weight as
a function of the instance. This measure is used at train time to reweight
training data, and at test-time to rank test instances for selective
classification. A second, key component of our proposal is the meta-objective
of minimizing dropout variance (the variance of classifier output when
subjected to random weight dropout) for training the metanetwork. We train the
classifier together with its metanetwork using a nested objective of
minimizing classifier loss on training data and meta-loss on a separate meta-
training dataset. We outperform current state-of-the-art on selective
classification by substantial margins–for instance, upto 1.9% AUC and 2%
accuracy on a real-world diabetic retinopathy dataset. Finally, our meta-
learning framework extends naturally to unsupervised domain adaptation, given
our unsupervised variance minimization meta-objective. We show cumulative
absolute gains of 3.4% / 3.3% accuracy and AUC over the other baselines in
domain shift settings on the Retinopathy dataset using unsupervised domain
adaptation.

##  1 Introduction

In applications with significant cost of error, such as medical diagnosis,
classifiers need to accurately quantify and communicate uncertainty about
their predictions. In the setting of selective classification [4], for
instance, classifiers only make predictions on test instances associated with
high model confidence. The “rejected” instances could be routed to a human
expert, with rejection rates adjusted according to application requirements,
cost of human intervention, and other factors influencing the design of human-
AI cooperative systems [3].

Proposals for selective classification include first-principles models of
uncertainty (bayesian NNs [6]) and custom loss functions [7, 11, 14]. A
popular and surprisingly strong baseline is Monte-carlo dropout (MCD [8]),
which measures test-time variance of classifier prediction for an instance
across different network weight dropout masks. We also note the parallel
between selective classification–a test-time application of prediction
uncertainty–and approaches that leverage some correlates of uncertainty at
training time; this includes proposals for robust classification objectives
[20, 24], as well as curricula (carefully defined schedules of training
instances, see e.g., [15]) based on various definitions of instance
“difficulty”.

Inspired by this prior work, we propose two key ideas to learn effective
selective classification models. First, we combine the two applications of
model uncertainty into a unified, meta-learning framework for selective
classification, where the same measure of model uncertainty is used for
optimizing model training as well as for test-time instance rejection. Second,
we devise a novel meta-objective for learning this measure of instance
uncertainty, based on Monte-Carlo dropout variance minimization–by directly
minimizing the well-known MCD uncertainty metric, we learn a classifier that
can make more confident judgments about instances. Putting these together, we
propose ReVaR: Reweighting for Dropout Variance Reduction, a new approach for
selective classification with the following contributions:

  * •

We propose a unified view of training & testing instance uncertainty by
learning an auxiliary network (a “MetaScorer”) that scores input instances
according to model uncertainty. The MetaScorer’s output reweights training
loss to improve model generalization, excludes low-confidence instances from
prediction at test time.

  * •

We frame a meta-learning approach for learning the MetaScorer alongside the
classifier, with a novel dropout variance reduction meta-objective. We derive
update rules for learning, with model parameters optimized on training data,
and MetaScorer parameters optimized on a separate meta-training dataset.

  * •

We show significant gains (upto 1.5% AUC and 1% accuracy) over SOTA in
selective classification on a key real-world dataset of diabetic retinopathy
classification[1] and upto 1.9% AUC and 2% accuracy when evaluated on an out-
of-distribution dataset [21]. These gains are replicated on other benchmarks
from the selective classification literature, including the ImageNet-100
dataset [22].

  * •

Extensive analyses demonstrate the independent contributions of our MetaScorer
and the novel dropout variance minimization objective. In particular, we show
both overall improvement in classifier accuracy, as well as the superior
uncertainty representation capability of MetaScorer over and above that of our
classifier.

  * •

We demonstrate the additional value of our novel dropout variance reduction
meta-objective for unsupervised domain adaptation, with large improvements in
selective classification under domain shift for retinopathy (additional gains
of upto 1.5% AUC and 2.5% accuracy over and above our own non-adaptive model
performance; this yields cumulative absolute gain of 3.4% / 3.3% accuracy and
AUC over the other baselines ).

##  2 Related Work

###  2.1 Selective Classification

In selective classification, a model needs to signal whether it will predict a
label for a given instance, and if applicable, the associated label. Typically
the model has a target selection rate, and is expected to maximize accuracy on
the selected instances; if the model has access to a reliable measure of
predictive uncertainty, that can be used for selection.

Approaches to selective classification differ in the manner in which this
uncertainty is modeled–for instance, directly leveraging the classifier’s
logits or using some auxiliary metric–and in the manner in which the
classifier is trained, e.g., through various modifications to the training
objective. For instance, [4] introduced an explicit objective of risk
minimization with constraints for trading off target coverage against
performance guarantees on instances selected for prediction. Selective
classification is also used as a benchmark for evaluating the intrinsic
uncertainty measures of Bayesian neural networks [6]. [7] proposes to directly
optimize selective calibration error (the calibration of only selected test
instances) and thereby improve selective classification. These approaches
largely depend on the classifier’s own model of uncertainty, and propose
various ways of improving the classifier itself for better selective
classification. In similar vein, other recent work aims to improve
classification generalization, with better selective classification as a
byproduct, using entropy regularization [5] and training-dynamics-based
ensembles [17], among other techniques. Other work proposes secondary measures
to select samples at test time–for instance, Monte-Carlo Dropout (MCD [8])
computes the variance across different weight dropout masks for an instance at
test-time as a measure of uncertainty. Deep Gamblers [14] proposes learning
additional output head for directly classifying uncertain examples into a
background class given a target selection rate. Selective Net [11] predicts a
separate score for each instance using a neural network and optimizes it
together with the target network for a specific coverage. Self-adaptive
training (SAT [12]) aggregates predictions over the timecourse of learning for
robustness to noise and improved selective classification. One-sided
prediction (OSP [9]) uses certain relaxations of training objective to choose
class-wise data points for selective classification using empirical risk
minimization. Our approach of combining training and test-time applications of
uncertainty is unique in selective classification to the best of our
knowledge, as is the use of MCD minimization for the meta-objective. Although
Selective Net [11] uses a network for predicting uncertainty, it does not use
these two features of our design. We compare against it and a range of other
competitive baselines in our experiments.

###  2.2 Instance-specific adaptation

A number of recent papers learn instance-specific parameters for robustifying
classifiers–for instance, a per-instance label noise parameter [2]; per-
instance and per-class softmax temperatures [19]; and per-instance weights for
a weighted loss optimization procedure [18]. These parameters are learned
using a meta-learning setup in order to optimize accuracy on a validation
dataset. More recent work [20, 24] uses an auxiliary network, as a function of
instance loss, to produce the training instance weights in a weighted loss
formulation–this drastically reduces the number of parameters required for
training. These approaches are not intended for, or evaluated on the selective
classification problem, although they share many aspects with our proposal
including reweighting and meta-learning. In additional differences: approaches
that learn free parameters as instance weights [18, 19] cannot generalize to
test samples and are inapplicable to test-time selection. Approaches that use
meta-networks for reweighting [20, 24] only use instance loss as input to the
network; our proposal uses the whole instance as input to the meta-network,
and is strictly more general. More importantly, our choice of dropout variance
minimization as meta-objective is critical for our performance gains; we
demonstrate this with lesion studies in experiments.

##  3 ReVaR: dropout variance reduction

###  3.1 Weighted training loss

Given a dataset with inputs as $X\in\mathcal{X}$ and corresponding outputs,
$Y\in\mathcal{Y}$, the problem of selective prediction is to learn a model,
comprising functions $f:\mathcal{X}\mapsto\mathcal{Y}$ and
$g:\mathcal{X}\mapsto\\{0,1\\}$, which for any given input example $x\in X$,
outputs the prediction of $f$ if $g$ approves, otherwise it simply abstains
from making a prediction:

$(f,g)(x)=\begin{cases}f(x)&\text{if $g(x)$=$1$};\\\ \text{can't
predict}&\text{if $g(x)$=$0$};\end{cases}$ (1)

Our primary design choice is the use of an auxiliary network conditioned on
the input (which we term a MetaScorer) to indicate prediction uncertainty
associated with an input $x$. This is a key difference from approaches that
use classifier logits [10] or test-time dropout variance of predicted logits
[8]. The uncertainty measure $g_{\Theta}(\cdot)$ is thresholded in order to
make test-time predictions at a given coverage, as is commonly done in most
selective classification approaches. We indicate the classifier and MetaScorer
parameters $\theta$ and $\Theta$ using subscripts, i.e., $f_{\theta}(\cdot)$
and $g_{\Theta}(\cdot)$.

Since we desire the uncertainty measure to be relevant for both training and
test-time scoring, we use the MetaScorer in a weighted loss formulation for
training the classifier:

$\mathcal{L}_{train}=\frac{1}{N}\sum_{i=1}^{N}g_{\Theta}(x_{i})\cdot
l(f_{\theta}(x_{i}),y_{i})$ (2)

where $N$ is the train set size, and $l$ corresponds to classification loss
for the predicted label for the instance $x_{i}$. This weighted loss encodes
the belief that highly uncertain instances need to be weighted more heavily in
the training objective. Other work in standard supervised learning has taken
this approach [20, 24]; however, they only use a function of the instance loss
on the classifier rather than the full input instance $x_{i}$, and also do not
use our variance-minimization meta-objective.

###  3.2 Learning the MetaScorer

We now discuss the training objective for $g_{\Theta}$. We measure and
optimize the MetaScorer’s performance on a specialized set (($X^{s}$,$Y^{s}$),
also referred to as a validation set), separate from the training set
($X$,$Y$) of the classifier. The model $g_{\Theta}$ is learned so as to
optimize a combination of classification accuracy and instance uncertainty.
The first component of this objective corresponds to the unweighted loss
computed on the specialized set:

$\mathcal{L}_{1}^{s}=\frac{1}{M}\sum_{i=1}^{M}l(f_{\theta}(x^{s}_{i}),y^{s}_{i})$
(3)

where M is the size of $X_{s}$, and ($x^{s}_{i}$,$y^{s}_{i}$) correspond to
the $i^{th}$ input-output pair in the specialized set.

Along with this loss component, the MetaScorer also aims to minimize a measure
of instance uncertainty measure. Based on the effectiveness of Monte-Carlo
dropout as a test-time uncertainty measure [8], we propose to reduce the total
MCD variance over the specialized set, i.e., the variance of predictions made
by the classifier when subjected to different drop out masks. Denoting the
$k^{th}$ dropout mask as $\mathcal{D}_{k}$, this secondary objective
$\mathcal{L}_{2}^{s}$ is formalized as follows:

$\frac{1}{MK}\sum_{i=1}^{M}\left(\sum_{k=1}^{K}(\mathcal{D}_{k}\odot
f_{\theta}(x^{s}_{i})-E[\mathcal{D}_{k}\odot
f_{\theta}(x^{s}_{i})])^{2}\right)$ (4)

where $\odot$ denotes the application of the dropout mask on the classifier,
and $E[\cdot]$ denotes the expected value (average) over all masks
$\mathcal{D}_{k}$ for the instance $x^{s}_{i}$. This results in the following
combined minimization objective for the MetaScorer:

$\mathcal{L}_{meta}=a\mathcal{L}_{1}^{s}+(1-a)\mathcal{L}_{2}^{s}$ (5)

where $a$ is a hyperparameter controlling the degree of contribution of the
two loss components.

###  3.3 Unsupervised domain adaptation

We note that the second loss component in the meta-objective,
$\mathcal{L}_{2}^{s}$, does not require any label information. This makes the
meta-objective an excellent candidate for unsupervised adaptation to novel
domains. Given unlabeled data $X^{d}$ from a new domain, we consider the
following two ways of using it in domain adaption. In both scenarios, we start
with a classifier learned on in-domain data $(X,Y);(X^{s},Y^{s})$, and then
update the classifier and metanet parameters using the new data $X^{d}$

Domain shift validation (DSV): In this scenario, we continue training ReVaR
with training data $(X,Y)$, and validation data $(X^{d},nil)$, and only MCD
minimization as the meta-objective.

Pooled validation (PV): Here, we continue training ReVaR with training data
$(X,Y)$, and pooled validation data $(X^{d}\cup X^{s},Y^{s})$. For the meta-
objective, $X^{d}$ only contributes to the variance minimization term.

We test both of these approaches in our experiments on domain adaptation for
selective classification.

###  3.4 Meta-learning with bilevel loss

The modeling choices we have laid out above result in a bi-level optimization
scheme involving the meta-network and classifier parameters. This is because
the values of each parameter set $\theta$ and $\Theta$ influence the
optimization objective of the other. Expressing $\mathcal{L}_{meta}$ as a
function $\mathcal{L}$ of inputs $X^{s}$, $Y^{s}$, $\theta$, this bi-level
optimization scheme can be formalized as:

$\displaystyle\theta^{*}=\arg\min_{\theta}\frac{1}{N}\sum_{t=1}^{N}g_{\Theta^{*}}(x_{t})\cdot
l(y_{t},f_{\theta}(x_{t}))$ (6) $\displaystyle\textit{s.t.
}\Theta^{*}=\arg\min_{\Theta}\mathcal{L}(X^{s},Y^{s},\theta)$

Calculating updates. Instead of solving completely for the inner loop
(optimizing $\Theta$) for every setting of the outer parameter $\theta$, we
aim to solve this bilevel optimization using alternating stochastic gradient
descent updates. At a high level, the updates are:

$\displaystyle\Theta_{t+1}$
$\displaystyle=\Theta_{t}-\alpha_{1}\nabla_{\Theta}\mathcal{L}(X^{s},Y^{s},\theta_{t})$
(7) $\displaystyle\theta_{t+1}$
$\displaystyle=\theta_{t}-\alpha_{2}G_{\Theta}^{T}(X)\nabla_{\theta}\mathcal{L}_{c}(X,Y,\theta_{t})$

where $\alpha_{1}$ and $\alpha_{2}$ are the learning rates corresponding to
these networks, $\mathcal{L}_{c}$ is the classification loss on the dataset
($X$,$Y$) using a network with params $\theta$, and $G_{\Theta}$ is the vector
of weights predicted by $g_{\Theta}$ for each input sample in $X$, with
$\Theta:=\Theta_{t+1}$. This style of stochastic optimization is commonly used
for solving bilevel optimization problems in a variety of settings [2, 20].
Further details, including all approximations used for deriving these
equations, are provided in the appendix.

Dataset | #labels | Train | Val | Test | Size
---|---|---|---|---|---
Retinopathy (Kaggle) | 5 (binarized) | 35,697 | 2617 | 42,690 | 512 $\times$ 512
Retinopathy (APTOS) | 5 (binarized) | - | - | 2917 | 512 $\times$ 512
ImageNet 100 | 100 | 130,000 | - | 5000 | 256 $\times$ 256
CIFAR 10 | 10 | 50000 | - | 10000 | 28 $\times$ 28
Table 1: Dataset statistics

##  4 Experiment setup

We compare ReVaR with a range of existing baselines for selective
classification on a number of benchmark datasets and under various coverage
levels for prediction. We also evaluate our method under domain shift
conditions, and finally compare the calibration errors of our method against
other selective classification baselines.

###  4.1 Baselines

We compare our approach against key baselines for selective classification
from the literature. Softmax Response (SR) [10] and SelectiveNet (SN) [11]
directly minimize a coverage based objective, with instance scores based on
classifier softmax probabilities or explicit modeling through a neural
network. Monte Carlo Dropout (MCD [8]) is a test-time instance scoring method
that uses dropout variance as a measure of uncertainty. We also include
abstention based methods which classify ambigous examples to a background
class according to a train-time configured coverage rate; this includes the
popular Deep Gamblers (DG) [14] approach and the recently proposed Self-
Adaptive Training (SAT) [12]. Additional baseline comparisons are presented in
the supplementary materials.

###  4.2 Datasets

Table 1 summarizes the various datasets used in our experiments, and their
characteristics.

(a) DR dataset (in-distribution AUC)

(b) DR dataset (in-distribution Accuracy)

(c) DR dataset (in-distribution calibration)

(d) DR dataset (country shift AUC)

(e) DR dataset (country shift Accuracy)

(f) DR dataset (country shift calibration)

Figure 1: Selective classification results on diabetic retinopathy dataset.
ReVaR shows robust improvement in AUC in both in-domain and domain-shift
scenarios (panels (a,b)). Accuracy measures also show similar trends, with
large improvements in domain shift conditions (panels (c,d)). Finally,
selective calibration error measures (calibration of selected data points,
panels (e,f)) show that better calibration is a key underlying factor for
ReVaR’s performance. See text for details. (a) ReVaR Lesion Study: Accuracy
Figure 2: Lesion study on the DR dataset.

Kaggle Dataset (in-distribution)
---
| 40% retained | 50% retained | 60% retained | 80% retained | 100% retained
|  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%)
MCD | 96.3 $\pm$ 0.1 | 97.8 $\pm$ 0.0 | 95.2 $\pm$ 0.1 | 97.1 $\pm$ 0.1 | 93.7 $\pm$ 0.3 | 95.3 $\pm$ 0.2 | 92.3 $\pm$ 0.2 | 92.8 $\pm$ 0.1 | 91.2 $\pm$ 0.2 | 90.6 $\pm$ 0.1
DG | 95.9 $\pm$ 0.1 | 97.2 $\pm$ 0.1 | 94.4 $\pm$ 0.2 | 96.4 $\pm$ 0.1 | 93.3 $\pm$ 0.2 | 95.1 $\pm$ 0.1 | 92.5 $\pm$ 0.3 | 93.1 $\pm$ 0.1 | 91.3 $\pm$ 0.3 | 90.8 $\pm$ 0.1
SN | 95.8 $\pm$ 0.1 | 97.0 $\pm$ 0.1 | 94.2 $\pm$ 0.2 | 96.1 $\pm$ 0.1 | 93.5 $\pm$ 0.3 | 95.2 $\pm$ 0.1 | 92.8 $\pm$ 0.1 | 93.4 $\pm$ 0.1 | 91.4 $\pm$ 0.3 | 90.9 $\pm$ 0.2
SAT | 96.5 $\pm$ 0.0 | 97.9 $\pm$ 0.0 | 95.0 $\pm$ 0.1 | 96.8 $\pm$ 0.1 | 93.9 $\pm$ 0.2 |  95.6 $\pm$ 0.1 | 92.7 $\pm$ 0.2 | 93.6 $\pm$ 0.2 | 91.7 $\pm$ 0.3 | 91.1 $\pm$ 0.2
Ours | 97.5 $\pm$ 0.1 | 98.4 $\pm$ 0.0 | 96.3 $\pm$ 0.2 | 97.4 $\pm$ 0.1 | 94.4 $\pm$ 0.3 | 95.5 $\pm$ 0.2 | 92.9 $\pm$ 0.2 | 93.8 $\pm$ 0.2 | 91.5 $\pm$ 0.3 | 91.0 $\pm$ 0.1
APTOS Dataset (country shift)
MCD | 79.8 $\pm$ 0.8 | 87.9 $\pm$ 0.5 | 87.2 $\pm$ 0.4 | 87.8 $\pm$ 0.2 | 89.1 $\pm$ 0.2 | 87.3 $\pm$ 0.2 | 91.4 $\pm$ 0.3 | 86.9 $\pm$ 0.2 | 93.6 $\pm$ 0.3 | 86.2 $\pm$ 0.2
DG | 83.7 $\pm$ 0.6 | 87.5 $\pm$ 0.3 | 88.1 $\pm$ 0.3 | 87.1 $\pm$ 0.2 | 90.1 $\pm$ 0.6 | 86.9 $\pm$ 0.2 | 91.9 $\pm$ 0.2 | 86.2 $\pm$ 0.2 | 93.7 $\pm$ 0.6 | 86.1 $\pm$ 0.1
SN | 86.2 $\pm$ 0.4 | 88.4 $\pm$ 0.4 | 88.1 $\pm$ 0.2 | 88.3 $\pm$ 0.2 | 89.7 $\pm$ 0.3 | 87.5 $\pm$ 0.1 | 91.1 $\pm$ 0.2 | 87.2 $\pm$ 0.1 | 93.2 $\pm$ 0.2 | 86.3 $\pm$ 0.1
SAT | 87.3 $\pm$ 0.3 | 89.8 $\pm$ 0.3 | 88.7 $\pm$ 0.2 | 89.2 $\pm$ 0.2 | 89.3 $\pm$ 0.2 | 87.9 $\pm$ 0.1 | 91.3 $\pm$ 0.3 | 87.1 $\pm$ 0.2 | 92.7 $\pm$ 0.3 | 86.9 $\pm$ 0.2
Ours | 89.2 $\pm$ 0.4 | 91.4 $\pm$ 0.2 | 90.2 $\pm$ 0.3 | 90.7 $\pm$ 0.3 | 90.9 $\pm$ 0.2 | 89.9 $\pm$ 0.1 | 91.8 $\pm$ 0.2 | 88.1 $\pm$ 0.2 | 92.3 $\pm$ 0.3 | 86.1 $\pm$ 0.2

Table 2: Comparison on the Kaggle dataset and the APTOS dataset under the country shift setup. | ReVaR |  ReVaR-PV |  ReVaR-DSV
---|---|---|---
Coverage |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%)
1.0 | 92.3 | 86.1 | 93.1 | 88.3 | 91.4 | 85.3
0.8 | 91.8 | 88.1 | 92.9 | 90.6 | 90.9 | 87.4
0.6 | 90.9 | 89.9 | 92.1 | 91.7 | 90.3 | 88.2
0.4 | 89.2 | 91.4 | 90.7 | 93.1 | 88.7 | 88.9
Table 3: Comparison of the Accuracy of ReVaR and its variants for unsupervised domain adaptation on Country Shift Setting. ReVaR-PV uses pooled in-domain and out-of-domain validation data, while ReVaR-DSV only uses domain-shift validation data in the meta-objective. See text for details.  | 0.7 | 0.8 | 0.9 | 0.95 | 1
---|---|---|---|---|---
MCD | 0.49 $\pm$ 0.0 | 1.10 $\pm$ 0.1 | 2.98 $\pm$ 0.1 | 4.91 $\pm$ 0.1 | 6.37 $\pm$ 0.1
DG | 0.46 $\pm$ 0.0 | 0.80 $\pm$ 0.1 | 2.38 $\pm$ 0.0 | 3.80 $\pm$ 0.1 | 6.43 $\pm$ 0.1
SN | 0.36 $\pm$ 0.0 | 0.92 $\pm$ 0.0 | 2.51 $\pm$ 0.1 | 4.54 $\pm$ 0.1 | 6.47 $\pm$ 0.1
SAT | 0.37 $\pm$ 0.0 | 0.78 $\pm$ 0.0 | 2.14 $\pm$ 0.1 | 3.63 $\pm$ 0.1 | 6.38 $\pm$ 0.1
Ours | 0.30 $\pm$ 0.0 | 0.67 $\pm$ 0.0 | 1.97 $\pm$ 0.1 | 3.22 $\pm$ 0.1 | 6.42 $\pm$ 0.1
Table 4: Comparison on the Cifar-10 dataset. | 0.6 | 0.7 | 0.8 | 0.9 | 1
---|---|---|---|---|---
MCD | 2.55 $\pm$ 0.4 | 3.62 $\pm$ 0.3 | 6.34 $\pm$ 0.3 | 9.34 $\pm$ 0.4 | 13.74 $\pm$ 0.3
DG | 2.31 $\pm$ 0.4 | 3.41 $\pm$ 0.3 | 5.36 $\pm$ 0.4 | 8.58 $\pm$ 0.4 | 13.62 $\pm$ 0.5
SN | 2.13 $\pm$ 0.2 | 3.51 $\pm$ 0.2 | 6.07 $\pm$ 0.1 | 9.56 $\pm$ 0.2 | 13.88 $\pm$ 0.2
SAT | 1.89 $\pm$ 0.2 | 2.86 $\pm$ 0.3 | 5.38 $\pm$ 0.2 | 8.89 $\pm$ 0.3 | 13.70 $\pm$ 0.4
Ours | 1.48 $\pm$ 0.2 | 2.32 $\pm$ 0.4 | 5.08 $\pm$ 0.3 | 8.67 $\pm$ 0.2 | 13.73 $\pm$ 0.3
Table 5: Comparison on the ImageNet-100 dataset.

Diabetic Retinopathy. The recently proposed Kaggle Diabetic Retinopathy (DR)
Detection Challenge [1] dataset and the APTOS dataset [21] are used as an
uncertainty benchmark for Bayesian learning [6]. The Kaggle dataset consists
of 35,697 training images, 2617 validation and 42,690 testing images, whereas
the APTOS dataset consists of 2917 evaluation images for testing ddomain
generalization for DR detection. In particular, the APTOS dataset is collected
from labs in India using different equipment, and is distributionally
different from the Kaggle dataset. Both datasets label input retina images
with the severity of diabetic retinopathy at 5 grades– 0-4 as No DR, mild,
moderate, severe and proliferative. Similar to [6], we formulate a binary
classification problem grouping grades 0-2 as negative class and 3,4 as
positive class. We focus primarily on this dataset given its real-world value
and role as a benchmark specifically relevant to selective classification [6].

Other image datasets. We study other image classification datasets commonly
used for evaluating selective classification methods. including the CIFAR-10
dataset [13] (10 categories of natural images). We also evaluate on a subset
of the widely used ImageNet dataset–ImageNet-100 [22]–consisting of 100
randomly selected classes from the 1k classes in the original ImageNet
dataset. This serves as a larger-scale stress-test of selective classification
given the relatively larger image size, dataset size, and number & complexity
of categories.

###  4.3 Metrics

For selective classification, we measure and report accuracy for all datasets.
In addition, for the DR dataset, we also measure AUC, a measure of performance
that is robust to class imbalance, and to potential class imbalance under
selective classification. The accuracy and AUC are measured on those data
points selected by each method for prediction, at the specified target
coverage. We also measure selective calibration, i.e., calibration error (ECE
[16]) measured on only the data points selected by the method for the
specified coverage. All metrics reported are averages $\pm$ standard deviation
over 5 runs of the method with different random initializations.

###  4.4 Training & evaluation

We use the ResNet-50 architecture as classifier for the diabetic retinopathy
experiments. Each experiment has been run 5 times and the associated mean and
standard deviation are reported. For MC-Dropout baseline and our meta-
objective, we calculate the uncertainty by 10 forward passes through the
classifier. We used a learning rate of 0.003 and a batch size of 64 to train
each model in each of the experiments. For our re-weighting scheme, we
separate 10 percent of the data as the validation set. For CiFAR-10 and
ImageNet-100 we use VGG-16 in all experiments, inspired by recent work [12,
14]. A batch size of 128 is used and an initial learning rate of 0.1 with a
momentum of 0.9 is used. A weight decay of $10^{-4}$ is used.

For all experiments, training is done for 300 epochs. For the unsupervised
domain adaptation from Kaggle data to APTOS data, we split the training into
230 epochs without unlabelled images and 70 epochs combining the unlabelled
images along with the source data for a total of 300 epochs.

##  5 Results

###  5.1 Diabetic Retinopathy Dataset

We present our main results on a large real-world application of selective
classification: Diabetic retinopathy detection [1]. Our evaluation considers
both in-distribution data, as well as a separate test set from a different
geographic region collected using different equipment– this is an evaluation
of test-time generalization under domain shift, without any additional
learning.

Figure 1 shows a comparison of all methods on selective classification for the
Kaggle DR dataset (first row) alongwith domain generalization results (Country
shift evaluated using APTOS dataset, second row)). The columns present
different metrics for each task: AUC (column 1), accuracy (column 2), and
selective calibration error (column 3). We see that ReVaR consistently
outperforms the other methods on both tasks and all metrics. In particular,
the robust gains on AUC (column 1, upto 1.5% absolute, see Table 7) for both
in-distribution and domain shift tasks are compelling. Note that although the
results are reflected in accuracy metrics as well (column 2, upto 2% absolute
gains on the domain shift task, see Table 7), AUC is less susceptible to class
imbalances and therefore a more reliable metric. Also, column 3 shows robust
improvement in calibration on both in-domain and out-of-domain data (ECE
metric, lower is better), suggesting that the MetaScorer indeed better
represents classifier uncertainty, and thereby improves on selective
classification. Finally, we note that the improvement in calibration, a widely
used metric of classifiers’ ability to capture and represent uncertainty,
suggests that ReVaR may have broad applications beyond selective
classification (see e.g., [23]).

(a) DR test-time unsup adapt (AUC)

(b) DR test-time unsup adapt (Acc)

Figure 3: Test-time unsupervised adaptation. Figure shows classifier
performance on domain shift data for the retinopathy benchmark (including
comparisons from Figure 1). The results show substantial additional gains from
the unsupervised test-time adaptation, for a total of upto 3.4% absolute
accuracy over baselines.

A note of interest is that AUC for all methods reduces in the domain shift
task as the selectivity is increased. This is the opposite of expected
behavior, where accuracy and AUC should generally increase as the classifier
becomes more selective. The data suggests a significant change in the two data
distributions that appears to partially invert the ranking order–i.e., all
classifiers appear to be more accurate for instances they are less confident
about. The robust gains of ReVaR suggest that it is less susceptible to such
drastic shifts in distribution.

###  5.2 Unsupervised domain adaptation

We explore the value of our meta-learning framework and (label-free) MCD-
minimization meta-objective in unsupervised domain adaptation for the
retinopathy dataset. For this, we took our ReVaR classifier trained on the
Kaggle dataset, then performed continued training using both the Kaggle data
and the APTOS test-time (domain-shift) dataset. Crucially, we did not use
labels from the domain-shift data. Section 3.3 lays out the details of the
adaptaion procedure; in short, we can calculate the MCD objective on unlabeled
data and use it during the meta-learning loop. As described there, we compare
the following variants, alongside our original proposal ReVaR:

ReVaR-PV: uses pooled validation data from in-domain (with labels) and domain
shift (without labels) in the meta-objective.
ReVaR-DSV: only uses domain shift validation data (without labels) in the
meta-objective.

Table 3 shows the results of our comparison. We see that while ReVaR-DSV fails
to improve upon ReVaR, ReVaR-PV shows handsome improvements over and above the
already substantial gains of ReVaR on top of the other baselines. In
particular, we see additional AUC gains of upto 1.5% and accuracy gains of
upto 2.5%. This results in a total of upto 3.4% absolute accuracy & 3.3%
absolute AUC gains over the other baselines (Figure 3). Our results suggest
that the labels from the source data serve a necessary regularization role
during domain adaptation, although more experimentation is needed to explore
the limits and potential of ReVaR-DSV.

###  5.3 Additional datasets

We replicated our findings on other datasets commonly used for studying
selective classification in the literature. This includes CIFAR-10 (Table 4),
and Imagenet-100 (Table 8). ReVaR retains an edge over the other baselines in
each of these datasets. In particular, the Iamgenet-100 dataset is relatively
more complex, given the much larger larger number of classes (100) on a
substantial training and evaluation set of higher-resolution images. ReVaR’s
superior performance on this dataset shows its potential for scaling to harder
selective classification problems.

In all datasets we see a pattern of increasing gap as the coverage is reduced,
suggesting that ReVaR is able to identify and retain the highest-confidence
test instances better than the other methods.

###  5.4 ReVaR: lesion analysis

We examined the contribution of the various components of our proposal to the
overal efficacy of ReVaR. To do this, we compare the following variants
against the full algorithm:
ReVaR-NoVarMin: This variant drops the dropout variance minimization component
of the meta-objective, and brings our proposal closer to previous robust
classification approaches [20, 24].111Note that since our MetaScorer operates
on the input instance $x$ rather than the loss on that instance, this variant
is still strictly more powerful in its representation capacity compared to
previous work [20, 24].
ReVaR-SR: This variant uses our meta-training objective (including dropout
variance minimization), but uses classifier logits at test time for selection,
similar to the SR baseline [10].
ReVaR-MCD: This variant also uses our meta-training objective, but uses MCD at
test time for scoring and selecting instances.
MCD: This baseline allows us to compare overall classifier quality separate
from uncertainty metric.

Figure 2 shows the result of this comparison on the DR dataset under country
shift, for the accuracy metric.

  * •

ReVaR performs best, showing that each of the components contribute to its
success–in particular, dropout variance minimization plays a very large role.

  * •

ReVaR-MCD performs better than MCD, suggesting that our meta-objective
improves the classifier’s performance at selective classification in general.

  * •

ReVaR performs better than ReVaR-SR and ReVaR-MCD (differing only in test-time
scoring). This shows that not only is the overall classifier better for
selective classification (see previous point), but also our MetaScorer’s
scores are a better measure of uncertainty than MCD or logits on the improved
classifier.

##  6 Conclusion

We proposed a method to train neural networks for better uncertainty
estimation using meta-learning. Our approach, ReVaR, unifies uncertainty
estimates at train-time for reweighting training data and robustifying
classifiers, and at test time, for instance in selective classification. We
proposed a novel meta-objective of dropout variance minimization on a
validation set, with the intuition of encouraging the learned classifier (via
selecting and reweighting the right training instances) to provide more
confident test-time predictions. Our experiments on the large, real-world
diabetic retinopathy dataset demnostrates not only gains over SOTA methods in
an in-domain prediction problem, but substantial improvements when challenged
with test-time domain shift. We show that our meta-learned uncertainty score
is indeed better calibrated than classifier logits and MCD, even on our
classifier. We replicate the findings on a number of benchmark datasets from
the literature including the large-scale Imagenet-100 dataset. Our results
suggest that ReVaR may have broad applicability in learning and using robust
measures of uncertainty–we will explore these directions in future work. We
are also interested in developing a conceptual framework within which we can
understand the relationship between classifier robustness, measures of
uncertainty, and variance minimization.

## References

  * [1] Kaggle. diabetic retinopathy detection challenge, 2015, 2015.
  * [2] Görkem Algan and Ilkay Ulusoy.  Meta Soft Label Generation for Noisy Labels.  In ICPR, 2020.
  * [3] Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham.  Role of human-ai interaction in selective prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5286–5294, 2022.
  * [4] Ran El-Yaniv et al.  On the foundations of noise-free selective classification.  Journal of Machine Learning Research, 11(5), 2010.
  * [5] Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir Abdi.  Stop overcomplicating selective classification: Use max-logit.  arXiv preprint arXiv:2206.09034, 2022.
  * [6] Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud De Kroon, and Yarin Gal.  A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks.  arXiv preprint arXiv:1912.10481, 2019.
  * [7] Adam Fisch, Tommi Jaakkola, and Regina Barzilay.  Calibrated selective classification.  arXiv preprint arXiv:2208.12084, 2022.
  * [8] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In international conference on machine learning, pages 1050–1059. PMLR, 2016.
  * [9] Aditya Gangrade, Anil Kag, and Venkatesh Saligrama.  Selective classification via one-sided prediction.  In International Conference on Artificial Intelligence and Statistics, pages 2179–2187. PMLR, 2021.
  * [10] Yonatan Geifman and Ran El-Yaniv.  Selective classification for deep neural networks.  Advances in neural information processing systems, 30, 2017.
  * [11] Yonatan Geifman and Ran El-Yaniv.  Selectivenet: A deep neural network with an integrated reject option.  In International conference on machine learning, pages 2151–2159. PMLR, 2019.
  * [12] Lang Huang, Chao Zhang, and Hongyang Zhang.  Self-adaptive training: beyond empirical risk minimization.  Advances in neural information processing systems, 33:19365–19376, 2020.
  * [13] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [14] Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda.  Deep gamblers: Learning to abstain with portfolio theory.  Advances in Neural Information Processing Systems, 32, 2019.
  * [15] Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al.  Prioritized training on points that are learnable, worth learning, and not yet learnt.  In International Conference on Machine Learning, pages 15630–15649. PMLR, 2022.
  * [16] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015\.
  * [17] Stephan Rabanser, Anvith Thudi, Kimia Hamidieh, Adam Dziedzic, and Nicolas Papernot.  Selective classification via neural network training dynamics.  arXiv preprint arXiv:2205.13532, 2022.
  * [18] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In International conference on machine learning, pages 4334–4343. PMLR, 2018.
  * [19] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste.  Data Parameters: A New Family of Parameters for Learning a Differentiable Curriculum.  NeurIPS, (3):11095–11105, 2019.
  * [20] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng.  Meta-weight-net: Learning an explicit mapping for sample weighting.  Advances in neural information processing systems, 32, 2019.
  * [21] Asia Pacific Tele-Ophthalmology Society.  Aptos 2019 blindness detection dataset, 2019.
  * [22] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In European conference on computer vision, pages 776–794. Springer, 2020.
  * [23] Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al.  Plex: Towards reliability using pretrained large model extensions.  arXiv preprint arXiv:2207.07411, 2022.
  * [24] Zizhao Zhang and Tomas Pfister.  Learning fast sample re-weighting without reward data.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 725–734, 2021.

Supplementary Materials

##  Appendix A Updates for the bilevel optimization

Here, we further discuss the bilevel optimization scheme we have used in the
paper. Revisiting the bi-level optimization objective proposed in the paper:

$\displaystyle\theta^{*}=\arg\min_{\theta}\frac{1}{N}\sum_{t=1}^{N}g_{\Theta^{*}}(x_{t})\cdot
l(y_{t},f_{\theta}(x_{t}))$ (8) $\displaystyle\textit{s.t.
}\Theta^{*}=\arg\min_{\Theta}\mathcal{L}(X^{s},Y^{s},\theta)$

where $\theta,\Theta$ correspond to model parameters for the primary &
MetaScorer models ($f_{\theta}$ and $g_{\Theta}$ respectively),
$(x_{t},y_{t})$ denote the input-output pair corresponding to the training set
and $l$ is the cross-entropy cost function for the classifier. Each loop for
this complete setup involves firstly updating the MetaScorer($\Theta$) using
the cross-entropy loss and variance of the classifiers on the val set,
generated by sampling different dropout masks for the classifier trained up
till now in presence of dropout. The backpropagation based update equation for
$\Theta$ at epoch $t$ is as follows:

$\Theta_{t+1}=\Theta_{t}-\frac{\alpha}{M}\nabla_{\Theta}\left(\frac{}{}\mathcal{V}(X^{s})_{\\{\theta^{k}_{t}\\}}+\mathcal{L}_{c}(X^{s},Y^{s},\theta_{t})\right)$
(9)

where $\alpha$ is the step size;
$\theta^{k}_{t}=\theta_{t}\odot\mathcal{D}_{k}$ denotes the parameter set
obtained from applying dropout mask $\mathcal{D}_{k}$ to the classifier
parameters $\theta_{t}$; $\mathcal{V}$ is the variance of model predictions on
the validation set inputs $X^{s}$ calculated over the model parameter
distribution $\\{\theta^{k}_{t}\\}_{k=1}^{K}$; and $\mathcal{L}_{c}$ denotes
the cross-entropy loss using classifier $\theta_{t}$ on the complete val set.
The gradient term in the above equation can be further simplified to:

$\displaystyle\left.\left.\nabla_{\theta}\left(\frac{}{}\mathcal{V}(X^{s})_{\\{\theta^{k}_{t}\\}}+\mathcal{L}_{c}(X^{s},Y^{s},\theta)\right)\right|_{\theta_{t}}\nabla_{\Theta}(\theta)\right|_{\Theta_{t}}$
(10)

where $\left.\nabla_{\theta}(.)\frac{}{}\right|_{\theta_{t}}$ denotes
evaluating the gradient at $\theta=\theta_{t}$. To calculate the the last term
in the equation, involving differentiation of classifier parameters w.r.t. re-
weighting network parameters, we now write the update equation for classifier
parameters $\theta$ at time $t+1$ involving the re-weighting network
parameters $\Theta_{t+1}$:

$\displaystyle\theta_{t+1}=\theta_{t}-\left.\frac{\beta}{N}\cdot\nabla_{\theta}G_{\Theta}^{T}(X)\cdot\textit{{L}}(X,Y,\theta)\right|_{\theta_{t},\Theta_{t+1}}$
(11)

where $\beta$ is the step size for MetaScorer, L denotes the vector containing
classification loss for each pair in $(X,Y)$ and $G_{\Theta}$ denotes the
vector of instance weights predicted by MetaScorer. The equation can be
further simplified based on the approximation that for a given epoch $\theta$
is dependent only on the weights predicted using $\Theta$, not directly
dependent on $\Theta$, in contrast to the case for updating $\Theta$:

$\displaystyle\theta_{t+1}=\theta_{t}-\left.\frac{\beta}{N}\cdot
G_{\Theta}^{T}(X)\cdot\nabla_{\theta}\textit{{L}}(X,Y,\theta)\right|_{\theta_{t},\Theta_{t+1}}$
(12)

Now, differentiating this w.r.t. $\Theta$:

$\displaystyle\nabla_{\Theta}(\theta)=\left.\left.-\frac{\beta}{N}\cdot\nabla_{\Theta}G_{\Theta}^{T}(X)\right|_{\Theta_{t+1}}\cdot\nabla_{\theta}\textit{{L}}(X,Y,\theta)\right|_{\theta_{t}}$
(13)

Thus, the last term in eq. 10 can be replaced by this last equation which will
modify the equation 10 to:

$\displaystyle\left.\left.\nabla_{\theta}\left(\frac{}{}\mathcal{L}(X^{s},Y^{s},\theta)\right)\cdot\nabla_{\theta}\textit{{L}}(X,Y,\theta)\right|_{\theta_{t}}\cdot\nabla_{\Theta}G_{\Theta}^{T}(X)\right|_{\Theta_{t+1}}$
(14)
$\displaystyle\text{where}\quad\mathcal{L}(X^{s},Y^{s},\theta)=\mathcal{V}(X^{s})_{\\{\theta^{k}\\}}+\mathcal{L}_{c}(X^{s},Y^{s},\theta)$

Kaggle Dataset (in-distribution)
---
| 40% retained | 50% retained | 60% retained | 80% retained | 100% retained
|  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%)
SN | 96.1 $\pm$ 0.1 | 97.2 $\pm$ 0.1 | 94.8 $\pm$ 0.2 | 96.6 $\pm$ 0.1 | 93.9 $\pm$ 0.2 | 95.5 $\pm$ 0.1 | 93.1 $\pm$ 0.2 | 93.8 $\pm$ 0.1 | 92.4 $\pm$ 0.3 | 91.7 $\pm$ 0.3
SAT | 96.8 $\pm$ 0.1 | 98.0 $\pm$ 0.1 | 95.7 $\pm$ 0.1 | 97.1 $\pm$ 0.1 | 94.2 $\pm$ 0.2 | 95.7 $\pm$ 0.1 | 93.5 $\pm$ 0.3 | 94.1 $\pm$ 0.2 | 92.6 $\pm$ 0.3 | 91.8 $\pm$ 0.2
Ours | 97.7 $\pm$ 0.1 | 98.6 $\pm$ 0.0 | 96.9 $\pm$ 0.2 | 97.8 $\pm$ 0.1 | 94.8 $\pm$ 0.2 | 95.9 $\pm$ 0.3 | 92.9 $\pm$ 0.2 | 93.8 $\pm$ 0.2 | 92.1 $\pm$ 0.2 | 91.5 $\pm$ 0.2
APTOS Dataset (country shift)
SN | 87.4 $\pm$ 0.4 | 89.7 $\pm$ 0.4 | 89.8 $\pm$ 0.3 | 89.5 $\pm$ 0.2 | 89.7 $\pm$ 0.3 | 87.5 $\pm$ 0.1 | 91.1 $\pm$ 0.2 | 87.2 $\pm$ 0.1 | 93.2 $\pm$ 0.2 | 86.44 $\pm$ 0.1
SAT | 88.3 $\pm$ 0.4 | 90.9 $\pm$ 0.3 | 90.2 $\pm$ 0.2 | 90.3 $\pm$ 0.2 | 89.8 $\pm$ 0.2 | 88.5 $\pm$ 0.2 | 91.8 $\pm$ 0.3 | 87.7 $\pm$ 0.2 | 92.6 $\pm$ 0.3 | 86.7 $\pm$ 0.2
Ours | 90.1 $\pm$ 0.5 | 92.5 $\pm$ 0.3 | 91.4 $\pm$ 0.3 | 91.9 $\pm$ 0.3 | 91.7 $\pm$ 0.2 | 89.9 $\pm$ 0.1 | 92.4 $\pm$ 0.3 | 88.9 $\pm$ 0.2 | 92.4 $\pm$ 0.3 | 86.2 $\pm$ 0.2

Table 6: Comparison on the Kaggle dataset and the APTOS dataset under the
country shift setup trained using entropy regularizer and then selecting based
on SR.

Kaggle Dataset (in-distribution)
---
| 40% retained | 50% retained | 60% retained | 80% retained | 100% retained
|  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%) |  AUC(%) |  Acc.(%)
SN | 95.9 $\pm$ 0.1 | 97.2 $\pm$ 0.1 | 94.1 $\pm$ 0.1 | 96.2 $\pm$ 0.2 | 93.8 $\pm$ 0.2 | 95.5 $\pm$ 0.2 | 92.6 $\pm$ 0.1 | 93.8 $\pm$ 0.1 | 91.6 $\pm$ 0.3 | 91.2 $\pm$ 0.2
SAT | 96.8 $\pm$ 0.1 | 97.2 $\pm$ 0.1 | 95.3 $\pm$ 0.2 | 96.2 $\pm$ 0.1 | 94.1 $\pm$ 0.3 |  95.8 $\pm$ 0.1 | 93.1 $\pm$ 0.1 | 93.8 $\pm$ 0.2 | 92.1 $\pm$ 0.3 | 91.6 $\pm$ 0.2
Ours | 97.7 $\pm$ 0.1 | 98.5 $\pm$ 0.0 | 96.7 $\pm$ 0.2 | 97.1 $\pm$ 0.1 | 94.2 $\pm$ 0.3 | 95.2 $\pm$ 0.1 | 93.2 $\pm$ 0.2 | 94.2 $\pm$ 0.2 | 91.8 $\pm$ 0.3 | 91.3 $\pm$ 0.1
APTOS Dataset (country shift)
SN | 86.4 $\pm$ 0.3 | 88.1 $\pm$ 0.5 | 87.7 $\pm$ 0.1 | 88.8 $\pm$ 0.3 | 90.1 $\pm$ 0.4 | 87.9 $\pm$ 0.2 | 91.6 $\pm$ 0.4 | 87.8 $\pm$ 0.2 | 93.5 $\pm$ 0.1 | 86.9 $\pm$ 0.1
SAT | 87.5 $\pm$ 0.2 | 89.6 $\pm$ 0.4 | 88.2 $\pm$ 0.2 | 89.4 $\pm$ 0.2 | 89.4 $\pm$ 0.2 | 88.2 $\pm$ 0.1 | 91.6 $\pm$ 0.3 | 87.3 $\pm$ 0.2 | 92.9 $\pm$ 0.3 | 87.3 $\pm$ 0.2
Ours | 89.2 $\pm$ 0.5 | 91.2 $\pm$ 0.2 | 90.5 $\pm$ 0.3 | 91.1 $\pm$ 0.3 | 91.2 $\pm$ 0.2 | 90.4 $\pm$ 0.2 | 92.2 $\pm$ 0.3 | 88.4 $\pm$ 0.3 | 92.5 $\pm$ 0.4 | 86.4 $\pm$ 0.2

Table 7: Comparison on the Kaggle dataset and the APTOS dataset under the country shift setup. | 0.6 | 0.7 | 0.8 | 0.9 | 1
---|---|---|---|---|---
SN | 2.04 $\pm$ 0.2 | 3.32 $\pm$ 0.1 | 5.89 $\pm$ 0.2 | 8.96 $\pm$ 0.1 | 13.07 $\pm$ 0.3
SAT | 1.76 $\pm$ 0.1 | 2.56 $\pm$ 0.2 | 5.07 $\pm$ 0.3 | 8.63 $\pm$ 0.2 | 13.02 $\pm$ 0.4
Ours | 1.42 $\pm$ 0.1 | 2.11 $\pm$ 0.3 | 4.78 $\pm$ 0.4 | 8.17 $\pm$ 0.3 | 13.23 $\pm$ 0.3
Table 8: Comparison on the ImageNet-100 dataset.

##  Appendix B Comparison with Entropy based regularization

A recent work [5] proposed using the maximum logit directly as the uncertainty
measure, of the methods trained with selective classification/learning to
abstain objectives, instead of their predicted scores. So for a given method,
e.g. SelectiveNet[11], it just combines classifier trained with that method
with Softmax Response (SR)[10] at response time. It further proposes an
entropy regularization loss in addition to cross entropy loss to penalize low
max-logit scores. We now analyze the effect of this entropy regularization on
the selective classification baselines and our method, comparing them for
Kaggle Diabetic Retinopathy data [1] used in the paper. For the baselines at
the inference time, we follow the strategy proposed in this method, using SR,
whereas for ours we go with the MetaScorer at the inference time. Table 6
shows the analysis for this experiment. It can be observed that our method
(with the MetaScorer ) is still significantly more effective when trained with
entropy regularization as compared to these baselines. Also, using variance
reduction based prediction scores are a better criteria as compared to
directly applying SR technique for these selective classifiers.

##  Appendix C Analyzing a different architecture

We examine the effect of backbone in evaluation of our proposed scheme.
Specifically, we compare the top-2 performing baselines namely SelectiveNet
(SN) and Self-Adaptive Training (SAT) with our method using a Wide-
ResNet-28-10 backbone with around 1.5 times parameters compared to the
ResNet-50 backbone used in the paper along with a different architecture. We
do this for the Diabetic Retinopathy data as well as the Imagenet-100 data.
Table 7 shows the analysis for diabetic retinopathy, both in-doamin and
country shift. Again we see the trend of performance is similar as compared to
Table 2 in the paper with accuracy improvements of aorund 0.3-0.5% for most
cases and 0.2-0.3 % decrease for a few cases. However, the performance gap is
similar to using the ResNet-50 baseline. Similarly, the trend for Imagenet-100
is approximately same as the paper with errors improved in the range 0.3-0.8
as compared to Table 3 in the paper. This change is visible for all the
methods. This can lead to a conclusion that architecture might not be playing
a major role in analyzing relative performance for selective classification.
However, any concrete claims require a more rigorous testing with various
state-of-the-art architectures proposed recently.