\begin{table}[ht!]
\centering
\caption{Results from the Background Challenge on ImageNet-9 using ResNet-50. Our method outperforms the baselines on both Mixed-same and Only FG test sets.}
\begin{tabular}{lcccccc}
Method & Original   & Mixed-same  & Mixed-rand & Only-FG   \\ \midrule

Baseline~\citep{xiao2020noise} & 96.3   & 89.8  & 75.6 & 85.6 \\ 
CIM~\citep{taghanaki2021robust}  & \textbf{97.7}   & 89.8 & \textbf{81.1} & - \\%& 8.8  \\

SIN~\citep{sauer2021counterfactual} & 89.2   & 73.1  & 63.7 & - \\ 
INSIN~\citep{sauer2021counterfactual} & 94.7  & 85.9  & 78.5 & - \\ 

INCGN~\citep{sauer2021counterfactual} & 94.2   & 83.4  & 80.1  & - \\
 
\textit{MaskTune} (Ours) & 95.6   & \textbf{91.1}  & 78.6  & \textbf{88.1} \\ 
\end{tabular}
\label{tab:bg}
\end{table}
\begin{abstract}
A fundamental challenge of over-parameterized deep learning models is learning meaningful data representations that yield good performance on a downstream task without over-fitting spurious input features. This work proposes \textit{MaskTune}, a masking strategy that prevents over-reliance on spurious (or a limited number of) features. \textit{MaskTune} forces the trained model to explore new features during a single epoch finetuning by masking previously discovered features. \textit{MaskTune}, unlike earlier approaches for mitigating shortcut learning, does not require any supervision, such as annotating spurious features or labels for subgroup samples in a dataset. Our empirical results on biased MNIST, CelebA, Waterbirds, and ImagenNet-9L datasets show that \textit{MaskTune} is effective on tasks that often suffer from the existence of spurious correlations. Finally, we show that \textit{MaskTune} outperforms or achieves similar performance to the competing methods when applied to the selective classification (classification with rejection option) task. Code for \textit{MaskTune} is available at \url{https://github.com/aliasgharkhani/Masktune}.
\end{abstract}

We apply \textit{MaskTune} to two main tasks: a) robustness to spurious correlations, and b) selective classification. We cover four different datasets under (a) including MNIST with synthetic spurious features, CelebA and Waterbirds with spurious features in different subgroups~\citep{sagawa2019distributionally}, and the Background Challenge~\citep{xiao2020noise} which is a dataset for measuring the reliance of methods on background information for prediction. Under (b) we test \textit{MaskTune} on CIFAR-10~\citep{krizhevsky2009learning}, SVHN~\citep{netzer2011reading}, and Cats vs. Dogs~\citep{geifman2019selectivenet} datasets. On both tasks, we outperform or perform similarly to the previous complex methods using our simple technique.

\subsection{Adapting \textit{MaskTune} for Selective Classification}
Here we show how to use \textit{MaskTune} for the selective classification problem. In order to make a more reliable prediction, we ensemble the original model (\ensuremath{m^\text{initial}_\theta}) and \textit{MaskTune} (\ensuremath{m^\text{final}_\theta}) and only predict if both models agree.
As a result, if there exist two sets of features that can predict the label, our method only predicts if both agree on the label (e.g., grass and cow in Figure~\ref{fig:cow}).

\subsection{Classification with Spurious Features} \label{sec:spurious}
\textbf{MNIST with Colored Squares on the Corners:} As a warm-up, we test the ability of our method to distinguish between two MNIST digit groups (0-4 and 5-9) in the presence of spurious features. We construct a dataset such that a classifier would achieve poor accuracy by relying on spurious input features. To this end, we group digits labeled 0 to 4 into class 0 and those labeled 5 to 9 into class 1. Next, in the training set, we place 99\% and 1\% of the new class 0 and new class 1 data on a background with a small blue square on the top left corner, respectively, and keep the remaining data intact. We use two test sets during testing: the original raw MNIST test set and a biased test set. To create the biased test set we place all of the digits from the group ``5-9'' on a background with a small blue square (representing spurious features) on the top left corner and keep all of the digits from the group ``0-4'' unchanged (i.e., we don't add any squares to their background and use the original images). Figure~\ref{fig:mnist} demonstrates the performance of the ERM, RandMask, and our \textit{MaskTune} on both original and biased test sets. The RandMask method is similar to \textit{MaskTune}, but masks a \textit{randomly} chosen area of the input image. As shown in Figure~\ref{fig:mnist}, \textit{MaskTune} outperforms other methods by a large margin on both test sets. As demonstrated in Figure~\ref{fig:mnist} (right), \textit{MaskTune} forces the model to explore more features, as opposed to the ERM, which only looks into the spurious feature (the blue square). We also ran an experiment with multiple spurious features (Appendix~\ref{app:exps}) and reported results for iterative version of ~\textit{MaskTune}.

\textbf{Classification with Spurious Features in Subgroups.} In this experiment, we leverage the CelebA~\citep{liu2015deep} and the Waterbirds~\citep{sagawa2019distributionally} datasets. In the CelebA dataset, there is a high correlation between features gender=\{male, female\} and hair\_color=\{blond, dark\}, meaning that the feature gender might be used as a proxy to predict the hair\_color. In other words, an ERM would assign the label dark to male images since the majority of male images have dark hair, and there are only 1,387 (0.85\%) blond males in the training set of size 162,770. To balance the accuracy of predictions on different subgroups of data, the existing methods~\citep{sagawa2019distributionally} use subgroup information, i.e., wrong predictions of an ERM on the worst group (blond males) can be penalized during training. Some other approaches~\citep{levy2020large,nam2020learning,pezeshki2021gradient,taghanaki2021robust,liu2021just} use subgroup information during model selection (labeled validation set). However, it is impractical to recognize ``all'' subgroups in a dataset and label them. 
In Table~\ref{tab:celeba}, we show that \textit{MaskTune} achieves comparable performance to both these groups of methods \emph{without} using group information in training or in model selection. 
We also show \textit{MaskTune} significantly improves worst-group accuracy (78\% vs. 55\% classification accuracy) in comparison to methods that do not use subgroup information during training or model selection. In Figure~\ref{fig:celeba} (left), we highlight the important input features for predicting hair color on the CelebA dataste. As shown, the ERM model leverages gender features while \textit{MaskTune} forces to investigate other features as well.

The Waterbirds dataset~\citep{sagawa2019distributionally} was proposed to assess the degree to which models pick up spurious correlations in the training set. We discovered and fixed two \textit{issues} with the Waterbirds dataset: a) because the background images in the Places dataset~\citep{zhou2017places} may already contain bird images, multiple birds may appear in an image after overlaying the segmented bird images from the Caltech-UCSD Birds-200-2011 (CUB) dataset~\citep{wah2011caltech}. For example, the label of an image may be ``landbird'', but the image contains both land and water birds. We manually removed such images from the dataset. b) Because the names of the species are similar, some land birds have been mislabeled as waterbirds which we corrected. The \textit{corrected} Waterbirds dataset can be found on \textit{MaskTune}'s GitHub page. After addressing the two issues, the ERM model's worst-group accuracy increased from 60\% which is reported in~\citep{sagawa2019distributionally} to 80.8$\pm$1.3\%. We repeated the group-DRO method and ERM experiments on the corrected waterbirds dataset and reported the results in Table~\ref{tab:waterbirds}. As demonstrated, our method (without any group supervision) achieves similar accuracy to the group-DRO that benefits from full supervision. In Figure~\ref{fig:celeba} (right), we visualized feature importance before and after applying our ~\textit{MaskTune} on the modified Waterbirds dataset.

\textbf{The Background Challenge.} As a further step, we evaluated \textit{MaskTune} on the Background Challenge data~\citep{xiao2020noise} to see if the positive observations from the MNIST experiment apply to a more realistic scenario. The Background Challenge is a publicly available dataset that consists of ImageNet-9~\citep{deng2009imagenet} test sets with various levels of foreground and background signals. It is intended to assess how much deep classifiers rely on spurious features for image classification. We used two configurations to compare \textit{MaskTune}'s performance: Only FG, in which the background is completely removed, Mixed-same, in which the foreground is placed on a different background from the same class, and Mixed-rand, where the foreground is overlaid onto a random background.

As shown in Table~\ref{tab:bg}, \textit{MaskTune} outperforms the baseline ResNet-50's performance by 2.5\% on the Only-FG test set and 1.2\% on Mixed-same, showing that \textit{MaskTune} does not rely much on the background and uses both background and foreground for prediction. On the Mixed-same and Only-FG test sets, ~\textit{MaskTune} outperforms other techniques because mixing/removing the background texture/info confuses other methods. These results show that our technique helps to learn task-relevant features without depending on nuisance signal sources.
{"value": "96.3", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["ResNet-50", "Baseline"], "model settings": {"xx": "yy"}} {"value": "89.8", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["ResNet-50", "Baseline"], "model settings": {"xx": "yy"}} {"value": "75.6", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["ResNet-50", "Baseline"], "model settings": {"xx": "yy"}} {"value": "85.6", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Only-FG", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": ["ResNet-50", "Baseline"], "model settings": {"xx": "yy"}} {"value": "97.7", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "CIM", "model settings": {"xx": "yy"}} {"value": "89.8", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "CIM", "model settings": {"xx": "yy"}} {"value": "81.1", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "CIM", "model settings": {"xx": "yy"}} {"value": "89.2", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "SIN", "model settings": {"xx": "yy"}} {"value": "73.1", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "SIN", "model settings": {"xx": "yy"}} {"value": "63.7", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "SIN", "model settings": {"xx": "yy"}} {"value": "94.7", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INSIN", "model settings": {"xx": "yy"}} {"value": "85.9", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INSIN", "model settings": {"xx": "yy"}} {"value": "78.5", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INSIN", "model settings": {"xx": "yy"}} {"value": "94.2", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INCGN", "model settings": {"xx": "yy"}} {"value": "83.4", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INCGN", "model settings": {"xx": "yy"}} {"value": "80.1", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "INCGN", "model settings": {"xx": "yy"}} {"value": "95.6", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Original", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "91.1", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-same", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "78.6", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Mixed-rand", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "MaskTune", "model settings": {"xx": "yy"}} {"value": "88.1", "type": "Result", "training data/set": "ImageNet-9", "test data/set": "ImageNet-9 Only-FG", "task": ["Background Challenge", "image classification"], "metric": "Accuracy", "experimental settings": {"xx": "yy"}, "model": "MaskTune", "model settings": {"xx": "yy"}}