\begin{table}[h!]
\centering
\setlength{\tabcolsep}{2pt}
\caption{Selective classification results on CIFAR-10, SVHN, and Cats vs. Dogs datasets for different coverage values.}
\begin{tabular}{c|c|cccccccccc}
\multirow{2}{*}{Dataset}      & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Target\\ Coverage\end{tabular}} & \multicolumn{2}{c}{SR} & \multicolumn{2}{c}{SN} & \multicolumn{2}{c}{DG} & \multicolumn{2}{c}{OSP} & \multicolumn{2}{c}{MaskTune} \\
                              &                                                                            & Cov.       & Err.     & Cov.      & Err.      & Cov.      & Err.      & Cov.   & Err.          & Cov.      & Err.            \\ \midrule
\multirow{3}{*}{Cifar-10}     & 100\%                                                                      & 99.99      & 9.58      & 100       & 11.07      & 100       & 10.81      & 100    & 9.74           & 99.99$\pm$0.02     & \textbf{8.96}$\pm$\textbf{0.48}    \\
                              & 95\%                                                                       & 95.2       & 8.74      & 94.7      & 8.34       & 95.1      & 8.21       & 95.1   & 6.98           & 94.86$\pm$0.18     & \textbf{6.54}$\pm$\textbf{0.39}    \\
                              & 90\%                                                                       & 90.5       & 6.52      & 89.6      & 6.45       & 90.1      & 6.14       & 90.0   & \textbf{4.67}           & 89.73$\pm$0.22     & 4.74$\pm$0.31    \\ \midrule
\multirow{3}{*}{SVHN}         & 100\%                                                                      & 99.97      & 3.86      & 100       & 4.27       & 100       & 4.03       & 100    & 4.27           & 100.0$\pm$0.00       & \textbf{3.68}$\pm$\textbf{0.16}    \\
                              & 95\%                                                                       & 95.1       & 1.86      & 95.1      & 2.53       & 95.0      & 2.05       & 95.1   & \textbf{1.83}  & 95.19$\pm$0.09     & 1.84$\pm$0.23             \\
                              & 90\%                                                                       & 90.0       & 1.04      & 90.1      & 1.31       & 90.0      & 1.06       & 90.1   & 1.01           & 89.55$\pm$0.26     & \textbf{0.96}$\pm$\textbf{0.11}     \\ \midrule
\multirow{3}{*}{Cats vs. Dogs} & 100\%                                                                      & 100        & 5.72      & 100       & 7.36       & 100       & 6.16       & 100    & 5.93           & 99.98$\pm$0.00     & \textbf{4.83}$\pm$\textbf{0.17}    \\
                              & 95\%                                                                       & 95.0       & 3.46      & 95.2      & 5.1        & 95.1      & 4.28       & 95.1   & 2.97           & 95.01$\pm$0.14     & \textbf{2.96}$\pm$\textbf{0.15}    \\
                              & 90\%                                                                       & 90.0       & 2.28      & 90.2      & 3.3        & 90.0      & 2.5        & 90.0   & \textbf{1.74}  & 90.78$\pm$0.16   & 1.94$\pm$0.18            
\end{tabular}
\label{tab:selective}
\end{table}
\section{Method}
\textbf{Setup.} 
We consider the supervised learning setting with inputs $x \in \mathcal{X} \subset \mathbb{R}^d$ and corresponding labels $y \in \mathcal{Y} = \{1, \ldots, k\}$. We assume having access to samples $\mathcal{D}^0 = \{(x_i, y_i)\}_{i=1}^n$ drawn from an unknown underlying distribution $p_{data}(x,y)$.

where $\mathcal{T}$ refers to a thresholding function by the threshold factor $\tau$ (i.e., $ \mathcal{T}=\mathbbm{1}_{\mathcal{A}_{x_i} \le \tau }$), and $\odot$ denotes element-wise multiplication. As the resolution of $\mathcal{A}$ is typically coarser than that of the input data, $\mathcal{T}(\mathcal{A}_{x_i})$ is up-sampled to the size of the input.

Procedurally, we first learn model $m_{\theta}^{\text{initial}}$ using original unmasked training data $\mathcal{D}^{\text{initial}}$. Then we use $m_{\theta}^{\text{initial}}$, $\mathcal{G}$ and $\mathcal{T}$ to create the masked set $\mathcal{D}^{\text{masked}}$. Finally, the fully trained predictor $m_{\theta}^\text{initial}$ is tuned using $\mathcal{D}^{\text{masked}}$ to obtain $m_{\theta}^{\text{final}}$. 
As for the masking step, any explainability approach can be applied (note that some may have more computational complexity, such as ScoreCAM~\citep{wang2020score}). We use xGradCAM~\citep{selvaraju2017grad} as it is fast and produces relatively denser heat-maps than other methods~\citep{srinivas2019full, selvaraju2017grad, wang2020score}. 

\textbf{Classification with Spurious Features in Subgroups.} In this experiment, we leverage the CelebA~\citep{liu2015deep} and the Waterbirds~\citep{sagawa2019distributionally} datasets. In the CelebA dataset, there is a high correlation between features gender=\{male, female\} and hair\_color=\{blond, dark\}, meaning that the feature gender might be used as a proxy to predict the hair\_color. In other words, an ERM would assign the label dark to male images since the majority of male images have dark hair, and there are only 1,387 (0.85\%) blond males in the training set of size 162,770. To balance the accuracy of predictions on different subgroups of data, the existing methods~\citep{sagawa2019distributionally} use subgroup information, i.e., wrong predictions of an ERM on the worst group (blond males) can be penalized during training. Some other approaches~\citep{levy2020large,nam2020learning,pezeshki2021gradient,taghanaki2021robust,liu2021just} use subgroup information during model selection (labeled validation set). However, it is impractical to recognize ``all'' subgroups in a dataset and label them. 
In Table~\ref{tab:celeba}, we show that \textit{MaskTune} achieves comparable performance to both these groups of methods \emph{without} using group information in training or in model selection. 
We also show \textit{MaskTune} significantly improves worst-group accuracy (78\% vs. 55\% classification accuracy) in comparison to methods that do not use subgroup information during training or model selection. In Figure~\ref{fig:celeba} (left), we highlight the important input features for predicting hair color on the CelebA dataste. As shown, the ERM model leverages gender features while \textit{MaskTune} forces to investigate other features as well.

The Waterbirds dataset~\citep{sagawa2019distributionally} was proposed to assess the degree to which models pick up spurious correlations in the training set. We discovered and fixed two \textit{issues} with the Waterbirds dataset: a) because the background images in the Places dataset~\citep{zhou2017places} may already contain bird images, multiple birds may appear in an image after overlaying the segmented bird images from the Caltech-UCSD Birds-200-2011 (CUB) dataset~\citep{wah2011caltech}. For example, the label of an image may be ``landbird'', but the image contains both land and water birds. We manually removed such images from the dataset. b) Because the names of the species are similar, some land birds have been mislabeled as waterbirds which we corrected. The \textit{corrected} Waterbirds dataset can be found on \textit{MaskTune}'s GitHub page. After addressing the two issues, the ERM model's worst-group accuracy increased from 60\% which is reported in~\citep{sagawa2019distributionally} to 80.8$\pm$1.3\%. We repeated the group-DRO method and ERM experiments on the corrected waterbirds dataset and reported the results in Table~\ref{tab:waterbirds}. As demonstrated, our method (without any group supervision) achieves similar accuracy to the group-DRO that benefits from full supervision. In Figure~\ref{fig:celeba} (right), we visualized feature importance before and after applying our ~\textit{MaskTune} on the modified Waterbirds dataset.

\subsection{Selective Classification} \label{sec:selective}
For selective classification task, we evaluated \textit{MaskTune} on three datasets: CIFAR-10~\citep{krizhevsky2009learning}, SVHN~\citep{netzer2011reading}, and Cats vs. Dogs~\citep{geifman2019selectivenet}, with coverage values of \{90\%, 95\%, 100\%\}. For example 90\% coverage means abstaining 10\% of the samples.

Given input image $j$ and the original (\ensuremath{m^\text{initial}_\theta}) and finetuned (\ensuremath{m^\text{final}_\theta}) models, let their respective inference-time prediction probabilities for class $i$ be $P_{ij}^\text{init} $ and $P_{ij}^\text{final}$. Then \textit{MaskTune} does not abstain and declares the class $i$ as the predicted class, iff $P_{ij}^\text{init} \cdot P_{ij}^\text{final} > \gamma $, where $\gamma$ is a threshold that allows for controlling the target coverage. 
To find the proper $\gamma$ for achieving the targeted coverage, we iterate over the validation set and find a threshold that, if applied to the validation set, would give the desired coverage (i.e., if our target coverage is 90\%, we seek for a threshold which if we apply for abstention on the validation set, we will abstain predicting 10\% of validation data). Note that the probabilities of the two models are multiplied to ensure that as the value of either probability decreases, the possibility of abstention increases.

We compare our method to the other selective classification approaches such as Softmax Response (SR)~\citep{geifman2019selectivenet}, SelectiveNet (SN)~\citep{geifman2019selectivenet}, Deep Gamblers (DG)~\citep{liu2019deep}, and One-sided Prediction (OSP)~\citep{gangrade2021selective}. Like the OSP, we first split the training data into train and validation sets. After training, we search for the abstention threshold on the validation set, then use it at the test time. 
As seen in Table~\ref{tab:selective}, \textit{MaskTune} outperforms all prior techniques on all three datasets on 95\% and 100\% coverage rates. For results on more coverage rates refer to Appendix~\ref{app:exps}.
{"value": "100", "type": "Other"} {"value": "99.99", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "9.58", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "11.07", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "10.81", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "9.74", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "99.99", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "8.96", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "95.2", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "8.74", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "94.7", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "8.34", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "8.21", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "6.98", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "94.86", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "6.54", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "90.5", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "6.52", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "89.6", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "6.45", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "90.1", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "6.14", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "90.0", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "4.67", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "89.73", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "4.74", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "100", "type": "Other"} {"value": "99.97", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "3.86", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "4.27", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "4.03", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "4.27", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "100.0", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "3.68", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "95.1", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "1.86", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "2.53", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "95.0", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "2.05", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "1.83", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "95.19", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "1.84", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "90.0", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "1.04", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "90.1", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "1.31", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "90.0", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "1.06", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "90.1", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "1.01", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "89.55", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "0.96", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "100", "type": "Other"} {"value": "100", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "5.72", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "7.36", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "6.16", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "100", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "5.93", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "99.98", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "4.83", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "95.0", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "3.46", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "95.2", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "5.1", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "4.28", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "95.1", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "2.97", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "95.01", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "2.96", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "90.0", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "2.28", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SR", "Softmax Response"], "model settings": {"xx": "yy"}} {"value": "90.2", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "3.3", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["SN", "SelectiveNet"], "model settings": {"xx": "yy"}} {"value": "90.0", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "2.5", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["DG", "Deep Gamblers"], "model settings": {"xx": "yy"}} {"value": "90.0", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "1.74", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": ["OSP", "One-sided Prediction"], "model settings": {"xx": "yy"}} {"value": "90.78", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "1.94", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}}