\begin{table}[h!]
\centering
\setlength{\tabcolsep}{3pt}
\caption{Selective clssification results on CIFAR-10, SVHN, and Cats vs. Dogs datasets for different coverage values.}
\begin{tabular}{c|c|cc}
\multirow{2}{*}{Dataset}      & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Target\\ coverage\end{tabular}} & \multicolumn{2}{c}{MaskTune} \\
                              &                                                                            & Cov.          & Error        \\ \midrule
\multirow{5}{*}{Cifar-10}     & 100\%                                                                      & 99.99$\pm$0.02    & 8.96$\pm$0.48    \\
                              & 95\%                                                                       & 94.86$\pm$0.18    & 6.54$\pm$0.39    \\
                              & 90\%                                                                       & 89.73$\pm$0.22    & 4.74$\pm$0.31    \\
                              & 85\%                                                                       & 84.46$\pm$0.07    & 3.23$\pm$0.20    \\
                              & 80\%                                                                       & 79.03$\pm$0.44    & 2.13$\pm$0.11    \\ \midrule
\multirow{5}{*}{SVHN}         & 100\%                                                                      & 100.0$\pm$0.00     & 3.68$\pm$0.16    \\
                              & 95\%                                                                       & 95.19$\pm$0.09    & 1.84$\pm$0.23    \\
                              & 90\%                                                                       & 89.55$\pm$0.26    & 0.96$\pm$0.11    \\
                              & 85\%                                                                       & 83.72$\pm$0.71    & 0.57$\pm$0.04    \\
                              & 80\%                                                                       & 77.81$\pm$0.75    & 0.45$\pm$0.05    \\ \midrule
\multirow{5}{*}{Cats vs. Dogs} & 100\%                                                                      & 99.98$\pm$0.0     & 4.83$\pm$0.17    \\
                              & 95\%                                                                       & 95.01$\pm$0.14    & 2.96$\pm$0.15    \\
                              & 90\%                                                                       & 90.78$\pm$0.16    & 1.94$\pm$0.18    \\
                              & 85\%                                                                       & 86.33$\pm$0.61    & 1.24$\pm$0.21    \\
                              & 80\%                                                                       & 81.98$\pm$0.47    & 0.89$\pm$0.20    
\end{tabular}
\label{tab:complete_selective}
\end{table}
\section{Method}
\textbf{Setup.} 
We consider the supervised learning setting with inputs $x \in \mathcal{X} \subset \mathbb{R}^d$ and corresponding labels $y \in \mathcal{Y} = \{1, \ldots, k\}$. We assume having access to samples $\mathcal{D}^0 = \{(x_i, y_i)\}_{i=1}^n$ drawn from an unknown underlying distribution $p_{data}(x,y)$.

Our goal is to learn the parameters $\theta \in \Theta$ of a prediction model $m_\theta: \mathcal{X} \rightarrow \mathcal{Y}$ that obtains low classification error w.r.t some loss function (e.g., cross entropy) $\ell: \Theta \times (\mathcal{X} \times \mathcal{Y}) \rightarrow \mathbb{R}$. Specifically, we minimize:\\

where $\mathcal{T}$ refers to a thresholding function by the threshold factor $\tau$ (i.e., $ \mathcal{T}=\mathbbm{1}_{\mathcal{A}_{x_i} \le \tau }$), and $\odot$ denotes element-wise multiplication. As the resolution of $\mathcal{A}$ is typically coarser than that of the input data, $\mathcal{T}(\mathcal{A}_{x_i})$ is up-sampled to the size of the input.

Procedurally, we first learn model $m_{\theta}^{\text{initial}}$ using original unmasked training data $\mathcal{D}^{\text{initial}}$. Then we use $m_{\theta}^{\text{initial}}$, $\mathcal{G}$ and $\mathcal{T}$ to create the masked set $\mathcal{D}^{\text{masked}}$. Finally, the fully trained predictor $m_{\theta}^\text{initial}$ is tuned using $\mathcal{D}^{\text{masked}}$ to obtain $m_{\theta}^{\text{final}}$. 
As for the masking step, any explainability approach can be applied (note that some may have more computational complexity, such as ScoreCAM~\citep{wang2020score}). We use xGradCAM~\citep{selvaraju2017grad} as it is fast and produces relatively denser heat-maps than other methods~\citep{srinivas2019full, selvaraju2017grad, wang2020score}. 

The Waterbirds dataset~\citep{sagawa2019distributionally} was proposed to assess the degree to which models pick up spurious correlations in the training set. We discovered and fixed two \textit{issues} with the Waterbirds dataset: a) because the background images in the Places dataset~\citep{zhou2017places} may already contain bird images, multiple birds may appear in an image after overlaying the segmented bird images from the Caltech-UCSD Birds-200-2011 (CUB) dataset~\citep{wah2011caltech}. For example, the label of an image may be ``landbird'', but the image contains both land and water birds. We manually removed such images from the dataset. b) Because the names of the species are similar, some land birds have been mislabeled as waterbirds which we corrected. The \textit{corrected} Waterbirds dataset can be found on \textit{MaskTune}'s GitHub page. After addressing the two issues, the ERM model's worst-group accuracy increased from 60\% which is reported in~\citep{sagawa2019distributionally} to 80.8$\pm$1.3\%. We repeated the group-DRO method and ERM experiments on the corrected waterbirds dataset and reported the results in Table~\ref{tab:waterbirds}. As demonstrated, our method (without any group supervision) achieves similar accuracy to the group-DRO that benefits from full supervision. In Figure~\ref{fig:celeba} (right), we visualized feature importance before and after applying our ~\textit{MaskTune} on the modified Waterbirds dataset.

\subsection{Selective Classification} \label{sec:selective}
For selective classification task, we evaluated \textit{MaskTune} on three datasets: CIFAR-10~\citep{krizhevsky2009learning}, SVHN~\citep{netzer2011reading}, and Cats vs. Dogs~\citep{geifman2019selectivenet}, with coverage values of \{90\%, 95\%, 100\%\}. For example 90\% coverage means abstaining 10\% of the samples.

Given input image $j$ and the original (\ensuremath{m^\text{initial}_\theta}) and finetuned (\ensuremath{m^\text{final}_\theta}) models, let their respective inference-time prediction probabilities for class $i$ be $P_{ij}^\text{init} $ and $P_{ij}^\text{final}$. Then \textit{MaskTune} does not abstain and declares the class $i$ as the predicted class, iff $P_{ij}^\text{init} \cdot P_{ij}^\text{final} > \gamma $, where $\gamma$ is a threshold that allows for controlling the target coverage. 
To find the proper $\gamma$ for achieving the targeted coverage, we iterate over the validation set and find a threshold that, if applied to the validation set, would give the desired coverage (i.e., if our target coverage is 90\%, we seek for a threshold which if we apply for abstention on the validation set, we will abstain predicting 10\% of validation data). Note that the probabilities of the two models are multiplied to ensure that as the value of either probability decreases, the possibility of abstention increases.

\section{Appendix} \label{app:exps}
\subsection{More Experimental Results}
\paragraph{Selective Classification.} In table \ref{tab:complete_selective} we show the results of the selective classification for the \textit{MaskTune} on different coverage values. We have run every experiment three times and reported the mean and std for each.
{"value": "100", "type": "Other"} {"value": "99.99", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "8.96", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "94.86", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "6.54", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "89.73", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "4.74", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "85", "type": "Other"} {"value": "84.46", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "3.23", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "80", "type": "Other"} {"value": "79.03", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "2.13", "type": "Result", "training data/set": "CIFAR-10", "test data/set": "CIFAR-10", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "100", "type": "Other"} {"value": "100.0", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "3.68", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "95.19", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "1.84", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "89.55", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "0.96", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "85", "type": "Other"} {"value": "83.72", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "0.57", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "80", "type": "Other"} {"value": "77.81", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "0.45", "type": "Result", "training data/set": "SVHN", "test data/set": "SVHN", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "100", "type": "Other"} {"value": "99.98", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "4.83", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "100"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "95", "type": "Other"} {"value": "95.01", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "2.96", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "95"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "90", "type": "Other"} {"value": "90.78", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "1.94", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "90"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "85", "type": "Other"} {"value": "86.33", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "1.24", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "85"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "80", "type": "Other"} {"value": "81.98", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Cov.", "Coverage"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "0.89", "type": "Result", "training data/set": "Cats vs. Dogs", "test data/set": "Cats vs. Dogs", "task": ["Selective classification", "image classification"], "metric": ["Err.", "Error"], "experimental settings": {"Target Coverage": "80"}, "model": "MaskTune", "model settings": {"xx": "yy"}}