Implicit Counterfactual Data Augmentation for Deep Neural Networks
Machine-learning models are prone to capturing the spurious correlations
between non-causal attributes and classes, with counterfactual data
augmentation being a promising direction for breaking these spurious
associations. However, explicitly generating counterfactual data is
challenging, with the training efficiency declining. Therefore, this study
proposes an implicit counterfactual data augmentation (ICDA) method to remove
spurious correlations and make stable predictions. Specifically, first, a novel
sample-wise augmentation strategy is developed that generates semantically and
counterfactually meaningful deep features with distinct augmentation strength
for each sample. Second, we derive an easy-to-compute surrogate loss on the
augmented feature set when the number of augmented samples becomes infinite.
Third, two concrete schemes are proposed, including direct quantification and
meta-learning, to derive the key parameters for the robust loss. In addition,
ICDA is explained from a regularization aspect, with extensive experiments
indicating that our method consistently improves the generalization performance
of popular depth networks on multiple typical learning scenarios that require
out-of-distribution generalization.
Xiaoling Zhou, Ou Wu
# Implicit Counterfactual Data Augmentation

###### Abstract

Machine-learning models are prone to capturing the spurious correlations between non-causal attributes and classes, with counterfactual data augmentation being a promising direction for breaking these spurious associations. However, explicitly generating counterfactual data is challenging, with the training efficiency declining. Therefore, this study proposes an implicit counterfactual data augmentation (ICDA) method to remove spurious correlations and make stable predictions. Specifically, first, a novel sample-wise augmentation strategy is developed that generates semantically and counterfactually meaningful deep features with distinct augmentation strength for each sample. Second, we derive an easy-to-compute surrogate loss on the augmented feature set when the number of augmented samples becomes infinite. Third, two concrete schemes are proposed, including direct quantification and meta-learning, to derive the key parameters for the robust loss. In addition, ICDA is explained from a regularization aspect, with extensive experiments indicating that our method consistently improves the generalization performance of popular depth networks on multiple typical learning scenarios that require out-of-distribution generalization.

 Counterfactual, implicit augmentation, spurious correlation, meta-learning, regularization, generalization.

## I Introduction

Deep learning models are supposed to learn invariances and make stable predictions based on some right causes. However, models trained with empirical risk minimization are prone to learning spurious correlations and suffer from high generalization errors when the training and test distributions do not match [1, 2]. For example, dogs are mostly on the grass in the training set. Thus, a dog in the water can easily be misclassified as a "drake" due to its rare scene context ("water") in the "dog" class, as illustrated in Fig. 1. A promising solution for improving the models' generalization and robustness is to learn causal models [3], as if a model can concentrate more on the causal correlations but not the spurious associations between non-causal attributes and classes, stable and exact predictions are more likely.

Counterfactual augmentation has become popular for causal models because of its clear explanation and being model-agnostic. For instance, Lu et al. [4] and He et al. [5] augmented the data effectively by swapping identity pronouns in texts. Moreover, Chang et al. [6] introduced two new image generation procedures that included counterfactual and factual data augmentations to reduce spuriousness between backgrounds of images and labels, achieving higher accuracy in several challenging datasets. Mao et al. [2] utilized a novel strategy to learn robust representations that steered generative models to manufacture interventions on features caused by confounding factors. Nevertheless, the methods presented above suffer from several shortcomings. Specifically, it is not trivial to explicitly find all confounding factors, and the training efficiency will decline as excess augmented images are involved in training.

It should be mentioned that implicit data augmentation settles the inefficiency of explicit augmentation by avoiding the generation of excess samples. ISDA [7] conducts a pioneering study on implicit data augmentation, which is inspired by the observation that the deep features in a network are usually linearized. Then, it translates samples along the semantic directions in the feature space based on an assumed class-wise augmentation distribution. By deriving an upper bound on the expected cross-entropy (CE) loss, ISDA enables optimization of only the upper bound to achieve data augmentation in an efficient way. Moreover, MetaSAug [8] innovatively applies the idea of ISDA to the long-tailed recognition, which does not modify the augmentation distribution of ISDA but optimizes the covariance matrices using metadata, yielding good performance on imbalanced data. Besides, RISDA [9] constructs novel augmentation distributions of tail classes by mixing the information from relevant categories, thus more effectively enriching samples in tail categories. However, these methods adopt purely class-wise semantic augmentation strategies, and thus samples in the same class have identical augmentation distributions that are inaccurate. Fig. 1(a) illustrates samples in the same class that may be negatively influenced by different attributes (or classes), where an ideal augmentation strategy should consider these sample-wise non-causal attributes. Additionally, these methods adopt the same augmentation strength for each instance, ignoring that inappropriate distributions (e.g., imbalanced label and attribute distributions) also lead

Fig. 1: (a): Illustration for images affected by spurious correlations due to rare attributes (e.g., posture, color, and scene context). \(C_{1}\), \(C_{2}\), and \(C_{3}\) are the dog, polar bear, and drake classes, respectively. The solid line connects the sample’s ground-truth class, and the dotted line connects the class with a spurious correlation with the sample. (b): Illustration for attribute imbalance.