Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
Robustness has become an important consideration in deep learning. With the
help of explainable AI, mismatches between an explained model's decision
strategy and the user's domain knowledge (e.g. Clever Hans effects) have been
identified as a starting point for improving faulty models. However, it is less
clear what to do when the user and the explanation agree. In this paper, we
demonstrate that acceptance of explanations by the user is not a guarantee for
a machine learning model to be robust against Clever Hans effects, which may
remain undetected. Such hidden flaws of the model can nevertheless be
mitigated, and we demonstrate this by contributing a new method,
Explanation-Guided Exposure Minimization (EGEM), that preemptively prunes
variations in the ML model that have not been the subject of positive
explanation feedback. Experiments demonstrate that our approach leads to models
that strongly reduce their reliance on hidden Clever Hans strategies, and
consequently achieve higher accuracy on new data.
Lorenz Linhardt, Klaus-Robert Müller, Grégoire Montavon
# Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks

###### Abstract

Explainable AI has become a popular tool for validating machine learning models. Mismatches between the explained model's decision strategy and the user's domain knowledge (e.g. Clever Hans effects) have also been recognized as a starting point for improving faulty models. However, it is less clear what to do when the _user and the explanation agree._ In this paper, we demonstrate that acceptance of explanations by the user is not a guarantee for a ML model to function well, in particular, some Clever Hans effects may remain undetected. Such hidden flaws of the model can nevertheless be mitigated, and we demonstrate this by contributing a new method, Explanation-Guided Exposure Minimization (EGEM), that _preemptively_ prunes variations in the ML model that have not been the subject of positive explanation feedback. Experiments on natural image data demonstrate that our approach leads to models that strongly reduce their reliance on hidden Clever Hans strategies, and consequently achieve higher accuracy on new data.

keywords: Clever Hans effect, model refinement, pruning, Explainable AI, deep neural networks +
Footnote †: journal: Journal of Machine Learning

## 1 Introduction

Machine learning (ML) models such as deep neural networks have been shown to be capable of converting large datasets into highly nonlinear predictive models [44; 8; 54; 77; 86; 18]. As ML systems are increasingly being considered for high-stakes decision making, such as autonomous driving [4] or medical diagnosis [17; 90; 64; 81; 81], building them in a way that they reliably maintain their prediction accuracy on new data is crucial.

Proper data splitting and evaluation of trained models on hold-out test sets have long been recognized as an essential part of the validation process (e.g. in [11]), but unfortunately, such techniques cannot detect all flaws of a model [45; 66; 27]. Misspecified loss functions, spurious correlations, or biased datasets can potentially compromise attempts to build well-generalizing models, without altering the measured accuracy. A failure to address these more elusive flaws might lead to catastrophic failures, as has been demonstrated numerous times (e.g. [62; 26; 88; 45; 31; 90; 10; 64]), which has spurred efforts to find potential causes of such failures (e.g. [25; 45; 58; 89; 14; 84; 60]). Furthermore, in modern real-world scenarios, data is often non-i.i.d. due to intrinsic heterogeneities in the data generating process (different users, locations, sensors, etc.) [35] and plagued with spurious correlations [16]. The task of robustification against the use of spurious features, also known as the Clever Hans (CH) effect [45] or shortcut learning [27], is an especially challenging endeavor because, to the model, such CH features are indistinguishable from truly generalizing ones.

Explainable AI (XAI) [30; 71; 9; 70; 34] is a natural starting point for robustification because it places a human in the loop: Explanation techniques seek to describe a model to the human in an intelligible way, e.g. based on features that can be visualized or that have a specific meaning. Using such methods, human experts can identify hidden flaws in the model [14] and provide useful feedback for model improvement [7; 56; 83; 74; 3]. For example, if a local explanation reveals the use of spuriously correlated features [45] (Clever Hans effect), the expert may take action to correct this flawed prediction strategy. More training examples that do not exhibit the spurious correlation may be provided [60] or the model may be trained to match the users' ground-truth explanations [65; 63; 79; 83; 74; 3].

However, the perhaps more common case, where the explanation returned to the user is _correct_, i.e. it agrees with the knowledge of the human expert, is so far little explored. In this paper, we demonstrate that a model validated with classical XAI pipelines is still not guaranteed to perform well on new data. Specifically, we find that CH strategies may remain undetected, especially when the data _available_ for validation is limited or incomplete, supporting recent findings by Adebayo et al. [1]. Given the rising popularity of _foundation models_[13] (multi
purpose models made available in pretrained form by a third party), and given that such models do not always come with the full dataset used for training, this scenario is not an academic exercise but a timely concern.

To address the problem of undetected CH strategies, we propose to refine the original third-party model in a way that its overall feature exposure is reduced, subject to the constraint that explanations presented to the user remain the same. Specifically, we contribute a method, called Explanation-Guided Exposure Minimization (EGEM), that formulates an optimization problem weighting the feature exposure and explanation constraints. With mild approximations, our formulation simplifies to easy-to-implement soft-pruning rules that enable the removal or mitigation of undetected CH strategies.--Crucially, our refinement method only requires the data points whose predictions and explanations have been approved by the user. Neither prior knowledge about the spurious feature, nor data containing it is needed. Our proposal, as well as the context in which it operates are illustrated in Fig. 1.

To evaluate our approach, we simulate a number of scenarios of a user receiving a third-party model and possessing a subset of the data on which no CH strategies can be detected by classical XAI pipelines (e.g. LRP/SpRAy [6; 45]).

Results on image data demonstrate that our proposed EGEM approach (and its extension PCA-EGEM) delivers models with a much lower reliance on CH strategies, thereby achieving more stable prediction accuracy, especially when considering data with spurious features. Our approach also outperforms a number of existing and contributed baselines.

## 2 Related Work

In this section, we present related work on validating ML models that goes beyond classical validation techniques such as holdout or cross-validation [11] in order to address statistical artifacts such as domain shifts and spurious correlations. We make a distinction between methods relying on Explainable AI and users' explanatory feedback (Section 2.1), and a broader set of methods addressing domain shift and spurious correlations by statistical means (Section 2.2).

### Explainable AI and Clever Hans

Explainable AI (XAI) [29; 71; 93; 70; 53] is a major development in machine learning which has enabled insights into a broad range of black-box ML models. It has shown to be successful at explaining complex state-of-the-art neural networks classifiers [6; 62; 75; 92; 73], as well as a broader set of ML techniques such as unsupervised learning (e.g. [51; 39]). While most XAI methods generate an explanation for individual instances, solutions have been proposed to aggregate them into dataset-wide explanations that can be concisely delivered to the user [45].

Notably, XAI techniques have been successful at revealing CH features in ML models [45; 60]. Knowledge about the CH features can be used to desensitize the model to these features (e.g. via retraining [60] or layer-specific adaptations [3]). If ground-truth explanations are available (e.g. provided by a human expert), the model may be regularized to match these explanations [65; 63; 79], e.g. by minimizing the error on the explanation via gradient descent. Such adaptations to the users' expectations have also been shown to be effective in interactive settings [83; 74]. Our approach differs from these works as

Figure 1: Cartoon comparison of a naive XAI-based validation/deployment pipeline and our proposed approach incorporating an additional exposure minimization step. _Left:_ A third party trains a flawed (Clever Hans) model which exploits a spurious correlation in the data (images of the horse class have a copyright tag in the bottom-left corner). _Middle:_ The user receives the model from the third party. Because the user has limited data (in particular, no data with copyright tag), the flaw of the model cannot be easily detected with XAI methods and the model appears to be both accurate and right for the right reasons. _Right:_ Because of the undetected flaw, a naive deployment of the model is likely to result in prediction errors on new data (e.g. cats with copyright tags predicted to be horses). Our proposed approach preemptively reduces exposure to unseen features (the copyright tag), thereby avoiding these incorrect predictions.
we address the case where the available data does not contain CH features, hence making them indiscoverable by local explanations, and where the model is pretrained and thus cannot be regularized during training.

A different approach is DORA [15], which attempts to find potential CH features in a data-agnostic way and subsequently use the discovered candidate features to detect faulty decision strategies at deployment time. In contrast, we attempt to _robustify_ the network with no need for further post-processing at deployment. Furthermore, we examine the scenario where a limited amount of clean data is available, allowing us to employ conceptually different criteria besides outlierness.

### Robustness to Spurious Correlations

Our work is part of a larger body of literature concerned with _domain shift_ and how to design models robust to it. Yet, it concerns itself with a decidedly specialized and rather recent part of this area: unlearning or avoiding the use of spurious features in deep neural networks. Previous work attempting to create models that are robust against spurious correlations approached the problem from the angle of optimizing worst-group loss [22, 36, 68, 69, 80, 37, 43, 57]. This approach has shown to be effective in reducing reliance on CH features. Yet, these methods require access to samples containing the CH features and a labeling of groups in the data induced by these features. In particular, as previously pointed out by Kirichenko et al. [43], Group-DRO (distributionally robust optimization) [36], subsampling approaches [69, 37] and DFR (deep feature reweighting) [43] assume group labels on the training or validation data, and even methods that do away with these assumptions need to rely on group labels for hyper-parameter tuning [49, 22, 37]. Our setting is different from the ones above in that we assume that a pretrained model is to be robustified post hoc with limited data and that data from the groups containing the CH feature are not available at all. We believe this is a highly relevant scenario, considering the increasing prevalence of pretrained third-party models that have been trained on datasets that are unavailable or too large to fully characterize.

## 3 Explanation-Guided Exposure Minimization (EGEM)

Let us restate the scenario of interest in this paper, highlighted in Fig. 1: (1) a model provided in pretrained form by a third party, (2) a user who has limited data available to validate the third-party model and who concludes that the predictions and the associated decision strategies (as revealed by XAI) on this limited data are correct.--As argued before, in spite of the positive validation outcome, there is no guarantee that the model's decision strategy remains correct in regions of the input space not covered by the available data.

As a solution to the scenario above, we propose a preemptive model refinement approach, which we call _Explanation-Guided Exposure Minimization (EGEM)_. Technically, our approach is a particular form of knowledge distillation where the refined (or distilled) model should reproduce observed prediction strategies (i.e. predictions and explanations) of the original model on the available data. At the same time, the refined model should minimize its overall exposure to variations in the input domain so that undetected (potentially flawed) decision strategies are not incorporated into the overall decision strategy.

Let the original and refined model have the same architecture but distinct parameters \(\theta_{\text{old}}\) and \(\theta\). We denote by \(f(\mathbf{x},\theta_{\text{old}})\) and \(f(\mathbf{x},\theta)\) the predictions produced by the two models, and the explanations associated to their predictions as \(\mathcal{R}(\mathbf{x},\theta_{\text{old}})\) and \(\mathcal{R}(\mathbf{x},\theta)\) respectively. We then define the learning objective as

\[\min_{\theta}\ \mathbb{E}\left[\|\mathcal{R}(\mathbf{x},\theta)-\mathcal{R}(\mathbf{x },\theta_{\text{old}})\|^{2}+\Omega(\mathbf{x},\theta)\right] \tag{1}\]

where the expectation is computed over the available data, and where \(\Omega(\mathbf{x},\theta)\) is a function that quantifies the exposure of the model to the input variation when evaluated at the data point \(\mathbf{x}\).

Although the aforementioned formulation is general, it is not practical, because it would require optimizing a highly nonlinear and non-convex objective. Moreover, the objective depends on explanation functions which themselves may depend on multiple model evaluations, thereby making the optimization procedure intractable.

### A Practical Formulation for EGEM

To make the concept of explanation-guided exposure minimization effective, we will restrict our analysis to XAI methods that can attribute onto any layer of the model and whose produced scores have a particular structure. Specifically, we require the score assigned to a neuron \(i\) at a given layer to be decomposable in terms of the neurons \(j\) in the layer above, i.e. \(R_{i}=\sum_{j}R_{ij}\) and terms of the decomposition should have the structure

\[R_{ij}=a_{i}\underbrace{\rho(w_{ij})\,d_{j}}_{s_{ij}} \tag{2}\]

where \(a_{i}\) denotes the activation of neuron \(i\), \(w_{ij}\) is the weight connecting neuron \(i\) to neuron \(j\) in the next layer, \(\rho\) is an increasing function satisfying \(\rho(0)=0\) (e.g. the identity function), and \(d_{j}\) is a term that only indirectly depends on the parameters in the given layer and that is reasonable to approximate as constant locally. Explanation techniques that produce explanation scores with such structure include backpropagation methods such as Layerwise Relevance Propagation (LRP) [6, 55] and gradient-based techniques such as \(\text{Gradient}\times\text{Input}\) (GI) and Integrated Gradients (IG). (See Supplementary Note A for derivations.)
We now present our practical formulation of explanation-guided exposure minimization. First, it imposes explanation similarity between the original and the refined model on the messages \(R_{ij}\) at a specified layer of the network, which is not necessarily the input layer. Furthermore, it restricts the search for refined parameters to the weights of the same layer. Hence, our scheme can be interpreted as changing the parameters of a given layer so that the overall model has minimized its exposure, subject to the explanation at that layer remaining the same. Specifically, we solve:

\[\min_{w}\ \sum_{ij}\mathbb{E}\big{[}(R_{ij}(w_{ij})-R_{ij}(w_{ij}^{\text{old}})) ^{2}+\lambda(s_{ij}(w_{ij}))^{2}\big{]}, \tag{3}\]

where the expectation is taken over the available data. The first squared term constraints the explanations of the refined model to be close to that of the original model. The second squared term corresponds to the penalty used for exposure minimization. The quantity \(s_{ij}(w_{ij})\) which we use for this purpose can be interpreted as the way by which the refined model responds to the activation of neuron \(i\) through neuron \(j\), in particular, if \((s_{ij})_{j}\) becomes zero, the model becomes unresponsive to the activation of neuron \(i\). An advantage of this formulation is that it has the closed-form solution:

\[\forall_{ij}:\ w_{ij}=\frac{\mathbb{E}[a_{i}^{2}d_{j}^{2}]}{\mathbb{E}[a_{i}^ {2}d_{j}^{2}]+\lambda\mathbb{E}[d_{j}^{2}]}w_{ij}^{\text{old}} \tag{4}\]

See Supplementary Note B for a derivation. In other words, the refined model can be seen as a soft-pruned version of the original model where the pruning strength depends on how frequently and to what magnitude the input neuron is activated and how the model responds to the output neuron. If we further assume that \(a_{i}\) and \(d_{j}\) are independent (or more weakly that their squared magnitudes are decorrelated), then it can be shown that \(d_{j}\) vanishes from Equation (4). Furthermore, the refined model can be obtained by keeping the weights intact and inserting a layer directly after the activations that performs the scaling:

\[\forall_{i}:\ a_{i}\gets a_{i}c_{i} \tag{5}\]

with \(c_{i}=\mathbb{E}[a_{i}^{2}]/(\mathbb{E}[a_{i}^{2}]+\lambda)\). The pruning of the neural network architecture and the resulting loss of dependence on the CH feature are depicted in Fig. 2 (top).

The same approach can also be applied to convolutional layers. To calculate the scaling parameters \(c_{i}\), the activations of each channel are summed up along the spatial dimensions. For refinement, the pruning coefficients are then applied to all activations of the corresponding feature map (cf. Eq. 5). Such pruning strategy for convolutional layers can be derived exactly from Eq. (3) if assuming activation maps of infinite size (or circular convolutions) and stride 1. For the majority of convolution layers used in practice, Eq. (5) only derives from the objective formulation approximately.

Figure 2: _Top:_ Cartoon depicting the removal of unseen Clever Hans strategies via our proposed exposure minimization approaches (EGEM and PCA-EGEM). The refined model only retains the dependence on \(a_{2}\) (a neuron detecting the actual horse) and removes its reliance on \(a_{3}\) (a neuron responsive to spurious copyright tags). _Bottom:_ Qualitative behavior of PCA-EGEM on ML models trained on real datasets. The models produced by our approach become robust to spurious features not seen at inspection time but occurring at deployment time. (Pixel-wise explanations are computed using the zennit package [(2)].)
### Pruning in PCA Space

Within the EGEM soft-pruning strategy, each dimension of a layer is pruned individually. In practice, this only allows to eliminate undetected flawed strategies that use a set of neurons that is disjoint from the validated strategies. Because a given neuron may contribute both to detected and to undetected strategies, the standard version of EGEM may not be able to carry out the exposure minimization task optimally. To address this limitation, we propose PCA-EGEM, which inserts a virtual layer, mapping activations to the PCA space (computed from the available data) and back (cf. Fig. 2). PCA-EGEM then applies soft-pruning as in Eq. (5), but in PCA space, that is:

\[\begin{split} h_{k}&=U_{k}^{\top}(\mathbf{a}-\mathbf{ \tilde{a}})\\ h_{k}&\gets h_{k}\cdot c_{k}\\ \mathbf{a}&\leftarrow\sum_{k}U_{k}h_{k}+\mathbf{\tilde{a}} \end{split} \tag{6}\]

with \(c_{k}=\mathbb{E}[h_{k}^{2}]/(\mathbb{E}[h_{k}^{2}]+\lambda)\). Here, \(\{U_{k}\}_{k=1}^{K}\) is the basis of PCA eigenvectors and \(\mathbf{\tilde{a}}\) is the mean of the activations over the available data. The motivation for such mapping to the PCA space is that activation patterns that support observed strategies will be represented in the top PCA components. PCA-EGEM can therefore separate them better from the unobserved strategies that are likely not spanned by the top principal components.

While using PCA to find principal directions of interpretable features for GANs [28] has been proposed previously by Harkonen et al. [32] and has found application beyond that [76; 19], to our knowledge its use for the purpose of identifying a basis for exposure minimization is novel.

## 4 Experimental Evaluation

In this section, we evaluate the efficacy of the approaches introduced in Section 3 on various datasets that either naturally contain spurious correlations, giving rise to Clever Hans decision strategies, or have been modified to introduce such spurious correlations. After introducing the datasets, we demonstrate that the proposed approaches can mitigate the effect of CH behavior learned by various pretrained models. We will do this by evaluating our approaches on test datasets where the correlation of the CH feature and the true class is manipulated - i.e. a distribution shift is introduced. Additionally, we empirically explore the effect of the number of samples used for refinement and discuss the challenges of hyper-parameter selection in our setting in Sections 4.4 and 4.5. A more qualitative evaluation on the CelebA dataset [50] follows in Section 5.

### Datasets

We introduce here the datasets used to evaluate the proposed methods in Sections 4.3-4.5: a modified version of the MNIST dataset [46], the ImageNet dataset [23; 67], and the ISIC dataset [20; 85; 21]. Details on the preprocessing, and the neural networks used for each dataset can be found in the Supplemental Notes.

Modified MNISTThe original MNIST dataset [46] contains 70,000 images of hand-written digits, 10,000 of which are test data. We create a variant in which digits of the class '8' are superimposed with a small artifact in the top-left corner, with a probability of 0.7 (see Fig. 2). In order to generate a natural yet biased split of the training data that separates an artifact-free set of refinement data, we train a variational autoencoder [42] on this modified dataset and manually chose threshold along a latent dimension such that the samples affected with the artifact only fall on one side. This defines a subset of 39,942 samples from which clean refinement datasets are sampled and leaves a systematically biased subset (containing all modified '8' samples) only accessible during training. We train a small (2 convolutional and 2 fully connected layers) neural network on this dataset using binary cross-entropy loss over all ten classes on the whole training data.

ImageNetWe use the ILSVRC 2012 subset of the ImageNet dataset [23; 67] containing 1.2M natural images, each associated with one of 1000 classes for training. For evaluation, we use the 50 labeled samples per class that are contained in the validation set. Previous work has identified multiple spurious correlations potentially affecting a model's output [3]. In our experiments, we use a watermark and web-address on images of the 'carton' class and a gray frame around images of the'mountain bike' class as Clever Hans features (see Fig. 2). We vary their frequency in the test set by pasting these features on images (details in Supplementary Note C). The selected classes are evaluated in a binary classification setting against the most similar classes in terms of the output probabilities. Training set images used for refinement that do not contain the CH feature are manually selected for the 'carton' experiments and automatically for'mountain bike' experiment. For experiments on this dataset, we make use of the pretrained ResNet50 [33] (for the 'carton' class) and VGG-16 [78] (for the'mountain bike' class) networks available in pytorch1.

\begin{table}
\begin{tabular}{|l|l|l|} \hline Dataset & \multicolumn{2}{c|}{Classes (Count)} & CH \\ \hline MNIST & **8**, others (10) & 3-pixel corner \\ \hline ISIC2019 & **Melanocyte nevus**, others (8) & colored patch \\ \cline{2-3}  & **carton**, crate (2) & watermark, www \\ \cline{2-3} ImageNet & **carton**, envelope (2) & watermark, www \\ \cline{2-3}  & **carton**, packet (2) & watermark, www \\ \cline{2-3}  & **mountain bike**, bicycle-built-for-two (2) & frame \\ \hline \end{tabular}
\end{table}
Table 1: Overview of datasets, classification problems (poisoned class in **bold**, number of classes in brackets), and spurious features (CH).
_ISIC_. The ISIC 2019 dataset [20; 85; 21] consists of images containing skin lesions that are associated with one of eight medical diagnoses. The data is split into 22,797 samples for training and 2,534 for evaluation. We fine-tune a neural network based on a VGG-16 pretrained on ImageNet for this classification task using a cross-entropy loss. Some images of the class 'Melanocytic nevus' are contaminated with colored patches (see Fig. 2), which have been recognized as potential CH feature [52; 63; 12; 3]. We manually remove all contaminated images after training and use this clean dataset for refinement. Images in the test set are contaminated at the desired ratio by pasting one extracted colored patch onto other images.

### Methods

We compare several methods for the mitigation of the Clever Hans effect. The most basic baseline is the original pretrained model (Original). We evaluate both EGEM and PCA-EGEM, as well as what we will call _response-guided exposure minimization_ (RGEM), which attempts to maintain the last-layer responses, rather than the explanations, while minimizing exposure by penalizing large weights. Specifically, RGEM solves

\[\min_{\theta}\ \ \mathbb{E}[(f(\mathbf{x},\theta)-f(\mathbf{x},\theta_{\rm old}))^{2} ]+\lambda\|\theta\|^{2} \tag{7}\]

where the expectation is computed over the available data. See Supplementary Note D for more details. Furthermore, we evaluate a version of the original model fine-tuned on the refinement set (Retrain) and a version of the original model where the last layer has been replaced by weights learned via ridge regression on the refinement data (Ridge). The latter is equivalent to linear probing or DFR, which has been shown to be effective in mitigating accuracy loss due to subpopulation shifts [72] and the Clever Hans effect, when hyper-parameter selection based on worst-group accuracy optimization is possible [43]. The formulation for Ridge can also be retrieved by replacing the output of the original model, \(f(\mathbf{x},\theta_{\rm old})\), in the formulation of RGEM in Supplementary Note D with the ground-truth labels.

### Results

To evaluate robustness against spurious features we generate a fully poisoned test set by adding the CH artifact uniformly over all test images of all classes. This serves as a scenario where the correlation of the CH feature and the target class breaks. Such a distribution shift could, for example, happen in medical applications where a classifier might be trained on data in which the mode of data collection or the population characteristics of subjects are correlated with the outcome, but this correlation does not hold in the general case [64]. Note that while this poisoning scenario is an extreme case, it is not the worst case, as the class which was contaminated during training will also be modified with artifacts during testing. For refinement, 700 correctly predicted samples per class are used, oversampling images if fewer than 700 correctly predicted samples are in the available refinement data. For the modified MNIST and the ISIC dataset, we use 1000 randomly chosen test samples for each run of the evaluation, for ImageNet we use all available validation samples.

We evaluate the various models for all tasks under 0% and 100% uniform poisoning. Classification accuracy for intermediate levels of poisoning can be obtained by linear interpolation of these extremes. Figure 3 shows the obtained accuracy on those two levels of poisoning. An ideal model would obtain high accuracy with only a very small difference between clean and poisoned data. It should be invariant to the spurious feature and at most react to possible interference with other features, e.g. the spurious feature being pasted on top of a relevant part of the image, while not losing accuracy on the clean data. As expected, across all datasets increased poisoning reduces accuracy of the original model. _Importantly, this drop in accuracy cannot be detected without access to samples containing the CH feature_.

On the modified MNIST dataset, the original model loses about 30% of its clean-data accuracy when evaluated at the 100% poisoning level. All other models achieve both clean-data and 100%-poisoned accuracy levels within 4% of the original model's clean-data accuracy. While explanation-based methods lose slightly more clean-data accuracy than the other baselines, they display virtually no gap between clean-data accuracy and 100%-poisoned accuracy, making them the most predictable when no poisoned data is available.

On the more complex ISIC dataset, it can be observed that exposure to the CH feature cannot be completely removed by any of the methods. EGEM and PCA-EGEM still provide fairly robust models, with the highest poisoned-data accuracy and the smallest gap between 0%-poisoned and 100%-poisoned accuracy. PCA-EGEM retains clean-data accuracy while being the only method improving poisoned-data accuracy by more than 10 percentage points. The dataset provides a challenge for all other methods. Even though Retrain is the only method that improves clean-data accuracy in the refinement process, its poisoned-data accuracy is virtually the same as the original model's, indicating that the absence of a feature in the refinement data is not enough to remove it from the model, given only a limited amount of samples.

On the ImageNet tasks containing the 'carton' class, PCA-EGEM is the most robust refinement method. It is only outperformed in the 100% poisoned setting of the 'carton/envelope' task, where Retrain achieves the highest clean-data and poisoned accuracy. As we will see in Section 4.4, the inferior 100% poisoning accuracy of PCA-EGEM is a result of the hyper-parameter selection procedure and not fundamentally due to the pruning-based nature of the method. On the 100% poisoned setting'mountain bike' task, no refinement method is able to achieve accuracy gains over the original model. This might be due to the small magnitude of the CH effect resulting in the
clean-data loss due to refinement outweighing the robustness gain. This case also demonstrates that refinement is not beneficial in all scenarios and might not even lead to an improved 100% poisoned accuracy. Whether or not to refine should be decided based on whether the loss of clean-data accuracy can be tolerated.

Overall, this section's experiments demonstrate that the proposed refinement methods can preemptively robustify a pretrained model against Clever Hans effects, even if the latter cannot be observed from the limited available data. We could clearly establish that the attempt to robustify against CH behavior in absence of the associated artifact or knowledge thereof is not a hopeless endeavor and can be addressed with relatively simple methods. Yet, the trade-off between clean-data accuracy and poisoned-data accuracy cannot be directly observed and thus needs to be resolved heuristically. We explore this aspect in the next section.

### Hyper-parameter Selection

The hyper-parameters optimized in the experiments in this section are the number or epochs for 'Retrain' and the regularization factor \(\lambda\) for all other refinement methods. For the deep exposure-based approaches, EGEM and PCA-EGEM, we do not optimize \(\lambda\) for each layer directly, but we rather employ an approach inspired by the triangular method of Ashouri et al. [5] and earlier work [61] where pruning strength increases with the layer index, which allows us to reduce the number of parameters to optimize to one. In particular, we define thresholds \(\tau_{l}\) that denote the desired average pruning ratio per layer, where \(l\) is the layer index within the set of \(L\) layers to be refined:

\[\tau_{l}=\begin{cases}\alpha&\text{if }l=1,\\ 1&\text{if }l=L,\\ \alpha+(l-1)\times\frac{1-\alpha}{L-1}&\text{otherwise},\end{cases}\]

and optimize \(\alpha\). \(\lambda_{l}\) is then set such that the average pruning factor \(\mathbb{E}_{\mathbf{x}\in\mathbf{X}}\,\mathbb{E}_{j}\,c_{j}\) from Eq. (4) for layer \(l\) is at least \(\tau_{l}\). The search for \(\lambda_{l}\) given \(\tau_{l}\) can be easily implemented as exponential search.

Ideally, the hyper-parameters should be set such that classification loss is minimized while exposure to the spurious artifact is negligible. While classification loss on clean data can be readily approximated by evaluating the loss function on the refinement data, exposure to the spurious artifact is a more elusive quantity and cannot be measured without a priori knowledge of the spurious artifact. In previous work (e.g. [36; 68; 69; 22; 80; 37; 43; 57]) it is assumed that for each class a set of samples with and without the spurious artifact is given and in most cases that the _worst-group-accuracy_ can be directly optimized or at least used for hyper-parameter selection, circumventing this problem. Since in our problem setting access to samples with the artifact is not given, this metric for parameter selection is not available and we need to establish a heuristic approach. Assuming that the classification loss on clean data can be approximated accurately, one option is to pick the strongest refinement hyper-parameter (i.e. highest number of epochs, largest \(\lambda\) or smallest \(\alpha\)) from a pre-defined set (see Supplementary Note F) for which the validation accuracy after refinement is at least as high as the one achieved by the original model.

As it is possible that strong refinement also impairs the use of generalizing features, there may be a trade-off between clean-data accuracy and robustness to spurious features. That optimizing overall clean-data accuracy is generally not the best approach to optimizing overall accuracy is highlighted by the fact that other works optimize _worst-group-accuracy_, as mentioned above. We explore the accuracy trade-off in Fig. 4 by introducing a'slack' parameter \(s\) to the hyper-parameter selection for PCA-EGEM. We refer to Supplementary Note G for the results of all methods. The refinement hyper-parameter is then chosen as the strongest regularization, given that the validation accuracy is at most \(s\%\) smaller than the one achieved by the original model. The idea is that minimizing loss of

Figure 3: The accuracy for 0% (lighter shade) and uniform 100% (darker shade) poisoning with the spurious feature. The last four bars for each method refer to the binary tasks constructed from the ImageNet dataset. Solid lines show average 100%-poisoned accuracy and dashed lines show average clean-data accuracy over all datasets. The results shown are the mean accuracy and standard deviation obtained using 700 refinement samples per class on the respective test sets over five runs.
classification accuracy on the refinement data prevents removing too much exposure to useful features, yet, allowing for some slack counteracts the tendency to choose trivial least-refinement solutions.

We suspect that in the simple case of the modified MNIST dataset, the model only learned few important high-level features and that the CH feature is close to being disentangled in some layer of the network. This scenario is a natural fit for pruning methods which could simply remove the outgoing connections of the node corresponding to the CH feature. Stronger refinement risks pruning useful features as well, which is an effect that can be observed in Fig. 4. For most datasets, we can observe that the accuracy curves first converge to or maintain a minimal 0%-100% poisoning gap. In this regime PCA-EGEM prunes unused or CH features. After crossing a certain level of slack, both accuracy values deteriorate as features necessary for correct classifications are being pruned as well.

The results previously presented in Fig. 3 are the outcomes for \(s=5\%\). This is a heuristic and it can be seen from Fig. 4 that different values of slack may be beneficial to increase robustness, depending on the dataset. We also show in Supplementary Notes G and H that PCA-EGEM provides the most robust refinement over a large range of slack values. As slack cannot be optimized w.r.t. the true deployment-time accuracy, we propose to set \(s\) between 1% and 5% as a rule of thumb.

In principle, another hyper-parameter is the choice of layers to refine. Knowledge of the type of Clever Hans could potentially guide this choice [3; 47] as the layer in which a concept is best represented may differ across concepts [41]. Since we do not assume such knowledge in our experiments, we simply refine the activations after every ResNet50 or VGG-16 block for the parts of the models that are derived from those architectures and additionally after every ReLU following a fully connected or convolutional layer that is not contained in a ResNet or VGG block. For 'Retrain' we fine-tune the whole network and RGEM and Ridge are restricted to the last layer.

### The Effect of the Sample Size

As the number of instances available for refinement is limited, a natural question is what impact the number of samples has on the efficacy of refinement and if refining with too few instances can be detrimental. In this section, we repeat the experiment from Section 4.3 for refinement datasets containing 25, 50, 200, 500, and 700 instances per class for 0% and 100% uniform poisoning. Slack is again set to 5%. If for some classes fewer correctly classified instances are available, these are over-sampled to achieve the desired number.

The effect of varying sample size is shown in Figure 5. It can be seen that especially in the low-sample regime, the positive effect of refinement is modulated by the number of instances. See Supplementary Note H for all other methods. While refinement with a small sample size appears in most cases to be remarkably effective for increasing 100%-poisoned accuracy, clean-data accuracy tends to suffer as the sample does not cover all of the features necessary to generalize, some of which are thus pruned away. For this reason, a larger refinement sample is in most cases beneficial, in particular for preserving clean-data accuracy. Yet, there are two cases that stand out as breaking this rule: The modified MNIST dataset and the 'carton/envelope' task. In both cases, the gap between 0% and 100%-poisoned accuracy is close to constant, suggesting that the drop in accuracy stems from a loss of generalizing features rather than a loss of robustness, as could be induced e.g. by samples contaminated by CH features. Considering the effect of slack, displayed in Fig. 4, we can also see that those two scenarios are also the cases for which 5% slack is not optimal. We hypothesize that here, the negative effect of increasing sample size stems from the interrelation between sample size and refinement strength. In particular, for EGEM and PCA-EGEM, using fewer instances generally means less coverage of the feature space, which leads to more zero or near-zero coefficients in the pruning procedure (cf. Eq. 5). Hence, for EGEM and PCA-EGEM, larger sample sizes potentially lead to _weaker_ refinement which can be similar in effect to

Figure 4: Accuracy under variations of the slack parameter. Higher slack means higher refinement-data loss is accepted when selecting the refinement hyper-parameter. The dotted line indicates clean-data accuracy whereas the solid line indicates 100% poisoned data accuracy. Mean and standard deviation are computed over 5 runs.
a decrease in slack.

Since clean-data accuracy can be evaluated on held-out data, we can observe that applying PCA-EGEM results in fairly predictable 100%-poisoning performance across a wide range of sample sizes, i.e. the spread between clean-data and poisoned-data accuracy is small, as is demonstrated by the relatively small shaded area in Fig. 5.

## 5 Use Case on CelebA: Reducing Bias

In this section, we will take a closer look at the effect of applying PCA-EGEM to a model trained on the CelebA dataset [50]. In contrast to the previous experiments, we do not evaluate based on a specific known CH feature, but rather conduct the analysis in an exploratory manner, uncovering subpopulations for which a learned CH behavior is leading to biased classifications. In practice, such an analysis could be done in hindsight, e.g. when PCA-EGEM has been applied before deployment, and its effect is later evaluated on new samples collected during deployment.

The CelebA dataset contains 202,599 portrait images of celebrities, each associated with 40 binary attributes. The existence of spurious correlations in the CelebA dataset has been documented previously [69; 91; 40; 68] and it can be seen in Supplementary Note C.2 that the attributes in the training set are correlated to various degrees. We train a convolutional neural network (details in Supplementary Note E) on the 'train' split of the CelebA dataset using cross-entropy loss on a 'blonde hair'-vs-not classification task. The training data is stratified and we achieve a binary test accuracy of 93%, which is comparable to accuracy reported in other works, e.g. Sagawa et al. [68]. We regard this classifier as a model given to the user by a third party.

In the following, we will assume a scenario where the user seeks to use the third-party classifier to retrieve blond people from a set of images available during deployment. They wish this retrieval process to be accurate and not biased against subgroups in the population. In order to analyze the impact of applying PCA-EGEM on such retrieval task, we simulate a validation set where the user has a limited subset of 'clean' examples, specifically, 200 examples of both classes, that are correctly predicted by the model and whose explanation highlight the actual blond hair as determined by LRP scores falling dominantly within the area of the image where the hair is located (see Supplementary Note F). These explanations (considered by the user to be all valid) are then fed to PCA-EGEM in order to produce a model that is more robust to potential unobserved Clever Hans effects. As for the previous experiments, we use 5% slack, which translates here to \(\alpha=0.01\).

### PCA-EGEM Reduces Exposure to Shirt Collars

After the model is deployed, the analysis of the decision strategy (of the original and refined model) can be reexamined in light of the new data now available. Fig. 6 shows explanations for some retrieved images, specifically, evidence for them being predicted to be blond. We can

Figure 5: Effect of the number of instances used for refinement. The dotted line indicates clean-data accuracy whereas the solid line indicates 100%-poisoned data accuracy. Mean and standard deviation are computed over 5 runs.

Figure 6: Test set images that exhibit strong changes in the detection of blond hair and corresponding LRP explanations, before and after refinement. Red indicates positive and blue negative contribution to the detection of blond hair. Shirt collars and similar features appear to inhibit the prediction of blond hair in the original model but less so in the refined one.
observe that pixels displaying hair are considered to be relevant and remain so after refinement.

In contrast, one can identify a significant change of strategy before and after refinement in the lower part of the image: The original model appears to make heavy use of shirt and suit collars as a feature inhibiting the detection of blond hair, whereas such inhibiting effect is much milder in the refined model. This observation suggests that PCA-EGEM has effectively mitigated a previously unobserved Clever Hans strategy present in the original model, and as a result, effectively aided the retrieval of images with collars on them.

### PCA-EGEM Balances Recall Across Subgroups

We will now analyze the implication of the Clever Hans effect reduction by PCA-EGEM on specific subgroups, specifically, whether certain subgroups benefit from the model refinement in terms of recalling members with the attribute 'blond'.

To this end, we randomly sample for every attribute in the dataset, a subset of 5000 images from the test data that only contains samples exhibiting this attribute. If fewer images are available for some attribute, we use all of the available samples. We evaluate the classifier with and without the application of PCA-EGEM on each of these subgroups.

Figure 7 shows recall scores for each subgroup before and after application of PCA-EGEM. We observe a substantial increase of recall on low-recall subgroups, such as 'Wearing_Necktie', 'Goatee', and 'Male'. Most high-recall group see only minuscule negative effects. Overall, while having almost no effect on the dataset-wide recall, we can observe that the application of PCA-EGEM rebalances recall in favor of under-recalled subsets. Our investigation thus demonstrates that a model bias responsible for under-detecting blond hair in these subgroups has been mitigated by applying PCA-EGEM, and consequently leads to a set of retrieved images that is more representative of the different subgroups and more diverse.

It is of theoretical interest to ask whether such rebalancing effect would generalize to other scenarios. An argument is that the underrepresentation of certain subgroups in the retrieved set is mainly caused by subgroups with low prevalence of the class of interest being actively suppressed by the model in order to optimize its accuracy. In practice, such suppression can be achieved by identifying features specific to the subgroup and, although causally unrelated to the task, making these features contribute negatively to the output score. Our PCA-EGEM technique, by removing such task-irrelevant Clever Hans features, redresses the decision function in favor of these low-prevalence subgroups, thereby leading to a more balanced set of retrieved instances.

Two outliers to the overall rebalancing effect can however be noted in Fig.7: 'Wearing_Hat', and 'Blurry'. Interestingly, these are two subgroups in which the feature of interest (the hair) is occluded or made less visible. In other words, in these two subgroups, only weakly correlated features are available for detection, and their removal by PCA-EGEM consequently reduces the recall. An underlying assumption behind the rebalancing effect is therefore that the true features are detectable in the input image without resorting to weakly or spuriously correlated features.

Overall, we have demonstrated in our CelebA use case, that PCA-EGEM can be useful beyond raising accuracy on disadvantageous test-set distributions. Specifically, we have shown that our PCA-EGEM approach enables the retrieval of a more diverse set of positive instances from a large heterogeneous dataset.

## 6 Open Questions

We could demonstrate the efficacy of the proposed methods for mitigation of Clever Hans effects in Sections 4.3 and 5, however, it can also be observed that 1) a complete removal of the model's response to the spurious (CH) feature is usually not achieved, and 2) classification accuracy on clean data may suffer. We suggest that there are multiple reasons for these undesired effects.

Firstly, in deep neural networks, CH features are generally not neatly disentangled from generalizing features. This means that either entangled well-generalizing features might suffer from pruning, reducing clean-data accuracy, or CH features might not be pruned due to being entangled with a feature present in the clean dataset. The latter would inhibit robustification against the CH feature. While the PCA-EGEM extension we have proposed achieves some basic form of disentanglement, more refined disentanglement methods based, for example, based on finding independent components, could be considered in future work.

A second open question is posed by the fact that the number of examples for which one collects explanatory feedback is limited. Thus, not all generalizing features may be present in the refinement data, thereby leading to these features being pruned away. Methods to more extensively draw from the user's explanatory feedback (e.g. rendering explanations to the user in a more intuitive way so that more examples can be inspected in a more precise manner, or presenting to the user in priority examples considered to be more informative) should be the focus of further investigation.

As pointed out in previous work [40], removing spurious (CH) features can hurt performance on data where the spurious correlation holds. Thus, un-biasing methods such as the refinement approaches introduced in this paper should be used under the assumption that they may hurt classification accuracy on biased data.

We also point out that our technique relies on the refinement set not containing CH features. This may naturally be satisfied in many cases where samples for refinement originate from a different data source than the
training data. Generally, our methods rely on post-hoc attribution methods to validate that the given samples are clean. While there have been concerns on the effectiveness of attribution methods in some scenarios [1], these concerns could be addressed in future work by moving beyond pixel-wise attribution for explanation validation e.g. using concept-based or counterfactual explanations [82; 87; 24].

## 7 Conclusion

Sensitivity to distribution shifts, such as the ones induced by spurious correlations (so-called Clever Hans effects) has long been an Achilles heel of machine learning approaches, such as deep learning. The problem becomes even more pronounced with the increasing adoption of foundation models for which the training data may not be public and is thus closed to scrutiny. Explanation techniques have the potential to uncover such deficiencies by putting a human in the loop [48; 45; 56; 71]. Previous work in XAI has mainly focused on improving explanations or fixing flaws in the model that have been identified by the user from such explanations. In contrast, we have considered the under-explored case where the human and the explanation _agree_ but where there are possibly unobserved spurious features that the model is sensitive to. While recent work has shown that XAI-based validation techniques may fail to detect some of these Clever Hans strategies employed by a model [1], we have argued that one can nevertheless still reduce the exposure of a model to some of these hidden strategies and demonstrated this via our contributed Explanation-Guided Exposure Minimization approach.

Our approach, while formulated as an optimization problem, reduces to simple pruning rules applied in intermediate layers, thereby making our method easily applicable, without retraining, to complex deep neural network models such as those used in computer vision. Our method was capable of systematically improving prediction performance on a variety of complex classification problems, outperforming existing and contributed baselines.

Concluding this paper, we would like to emphasize the novelty of our approach, which constitutes an early attempt to leverage correct explanations for producing refined ML models and attempts to tackle the realistic scenario where Clever Hans features are not accessible. We believe that in future work, the utility derived from explanations via refinement can still be expanded, e.g. by letting the user specify what is correct and what is incorrect in an explanation so that the two components can be treated separately, or by identifying sets of examples to present to the user that are the most useful to achieve model refinement, for example, by ensuring that they cover the feature space adequately or by active learning schemes.

## 8 Acknowledgements

This work was partly funded by the German Ministry for Education and Research (under refs 01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18056A, 01IS18025A and 01IS18037A), the German Research Foundation (DFG) as Math+: Berlin Mathematics Research Center (EXC 2046/1, project-ID: 390685689). Furthermore KRM was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program, Korea University and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug-and-Play Explanation and Verification of Explanation). We thank Pattarawat Chormai and Christopher Anders for the valuable discussion and the extracted watermark artifact.

## References

* [1] J. Adebayo, M. Muelly, H. Abelson, and B. Kim (2022) Post hoc Explanations may be Ineffective for Detecting Unknown Spurious Correlation. In Proceedings of the 10th International Conference on Learning Representations, Cited by: SS1, SS2.1, SS2.2, SS2.
* [2] C. J. Anders, D. Neumann, W. Samek, K. Muller, and S. Lapuschkin. Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zemmit, CoReLay, and ViRelay. _arXiv preprint_, 2106.13200, 2021.
* [3] C. J. Anders, L. Weber, D. Neumann, W. Samek, K.-R. Muller, and S. Lapuschkin. Finding and removing Clever Hans: Using explanation methods to debug and improve deep models. _Information Fusion_, 77:261-295, 2022.
* [4] S. Aradi. Survey of deep reinforcement learning for motion planning of autonomous vehicles. _IEEE Transactions on Intelligent Transportation Systems_, 23(2):740-759, 2022.
* [5] A. H. Ashouri, T. S. Abdelrahman, and A. Dos Remedios. Retraining-Free Methods for Fast on-the-Fly Pruning of Convolutional Neural Networks. _Neurocomputing_, 370(C):56-69, 2019.
* [6] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Muller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. _PLoS ONE_, 10(7):e0130140, 2015.
* [7] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Muller. How to Explain Individual Classification Decisions. _Journal of Machine Learning Research_, 11:1803-1831, 2010.
* [8] D. Bahdanau, K. Cho, and Y. Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In _Proceedings of the 3rd International Conference on Learning Representations_, 2015.
* [9] A. Barredo Arrieta, N. Diaz-Rodriguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera. Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. _Information Fusion_, 58:82-115, 2020.
* [10] A. Binder, M. Bockmayr, M. Hagele, S. Wienert, D. Heim, K. Hellweg, M. Ishii, A. Stenzinger, A. Hocke, C. Denkert, et al. Morphological and molecular breast cancer profiling through explainable machine learning. _Nature Machine Intelligence_, 3(4):355-366, 2021.
* [11] C. Bishop. _Pattern Recognition and Machine Learning_. Springer, 2006.
* [12] A. Bissoto, E. Valle, and S. Avila. Debiasing Skin L-Aision Datasets and Models? Not So Fast. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_, 2020.
* [13] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, et al. On the Opportunities and Risks of Foundation Models. _arXiv preprint_, 2108.07258, 2021.
* [14] S. Booth, Y. Zhou, A. Shah, and J. Shah. Bayes-TrEx: a Bayesian Sampling Approach to Model Transparency by Example. In _Proceedings of the 35th AAAI Conference on Artificial Intelligence_, 2020.
* [15] K. Bykov, M. Deb, D. Grinwald, K.-R. Muller, and M. M. C. Hohne. DORA: Exploring outlier representations in Deep Neural Networks. _arXiv preprint_, 2020.04530, 2022.
* [16] C. S. Calude and G. Longo. The deluge of spurious correlations in big data. _Foundations of Science_, 22(3):595-612, Mar. 2016.
* [17] D. Capper, D. T. W. Jones, M. Sill, V. Hovestadt, D. Schrimpf, et al. DNA methylation-based classification of central nervous system tumours. _Nature_, 555(7697):469-474, 2018.
* [18] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119, pages 1597-1607, 2020.
* [19] P. Chormai, J. Herrmann, K.-R. Muller, and G. Montavon. Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces. _arXiv preprint_, 2212.14855, dec 2022.
* [20] N. C. F. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Lioypris, N. Mishra, H. Kittler, and A. Halpern. Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC). _arXiv preprint_, 1710.05006, 2018.
* [21] M. Combalia, N. C. F. Codella, V. Rotemberg, B. Helba, V. Giaplasma, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpern, S. Puig, and J. Malvehy. BCN2000: Dermoscopic Lesions in the Wild. _arXiv preprint_, 1908.02288, 2019.
* [22] E. Creager, J.-H. Jacobsen, and R. Zemel. Environment Inference for Invariant Learning. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139, pages 2189-2200, 2021.
* [23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 248-255, 2009.
* [24] A.-K. Dombrowski, J. E. Gerken, K.-R. Muller, and P. Kessel. Diffeomorphic Counterfactuals with Generative Models. _arXiv preprint_, 2206.05075, jun 2022.
* [25] D. J. Fremont, X. Yue, T. Dreossi, A. L. Sangiovanni-Vincentelli, S. Ghosh, and S. A. Seshia. Scenic: A language for scenario specification and scene generation. In _Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI)_, pages 63-78. Association for Computing Machinery, 2019.
* [26] T. Gebru and J. Buolamwini. Gender Shades: Intersection Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_, volume 81, pages 77-91, 2018.
* [27] R. Geirhos, J. H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann. Shortcut learning in deep neural networks. _Nature Machine Intelligence_, (211):665-673, 2020.
* [28] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In _Advances in Neural Information Processing Systems_, volume 27, pages 2672-2680, 2014.
* [29] D. Gunning and D. W. Aha. Darpa's explainable artificial intelligence (XAI) program. _AI Mag._, 40(2):44-58, 2019.
* [30] D. Gunning, M. Steffik, J. Choi, T. Miller, S. Stumpf, and G.-Z. Yang. XAI-Explainable artificial intelligence. _Science Robotics_, 4(37):eaay7120, 2019.
* [31] M. Hagele, P. Segeerer, S. Lapuschkin, M. Bockmayr, W. Samek, F. Klauschen, K.-R. Muller, and A. Binder. Resolving challenges in deep learning-based analyses of histopathological images using explanation methods. _Scientific reports_, 10:6423, 2020.
* [32] E. Harkonen, A. Hertzmann, J. Lehtinen, and S. Paris. GANSpace: Discovering Interpretable GAN Controls. In _Advances in Neural Information Processing Systems_, volume 33, pages 9841-9850, 2020.
* [33] K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 770-778. IEEE, jun 2016.
* Beyond Explainable AI
- International Workshop, Held in Conjunction with ICML 2020_, volume 13200 of _Lecture Notes in Computer Science_. Springer, 2022.
* [35] K. Hsieh, A. Phanishayee, O. Mutlu, and P. B. Gibbons. The non-lid data quangime of decentralized machine learning. In _ICML_, volume 119 of _Proceedings of Machine Learning Research_, pages 4387-4398. PMLR, 2020.
* [36] W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In _Proceedings of the 35th International Conference on Machine Learning_, volume 80, pages 2029-2037, 2018.
* [37] B. Y. Idrissi, M. Arjovsky, M. Pezeshki, and D. Lopez-Paz. Simple data balancing achieves competitive worst-group-accuracy. In _Proceedings of the First Conference on Causal Learning and Reasoning_, pages 177:336-351, 2022.
* [38] P. Jurmeister, S. Gloss, R. Roller, M. Leitheiser, S. Schmid, L. H. Mochmann, et al. DNA methylation-based classification of sinonasal tumors. _Nature Communications_, 13(1):7148, 2022.
* [39] J. Kauffmann, M. Esders, L. Ruff, G. Montavon, W. Samek, and K.-R. Muller. From Clustering to Cluster Explanations via Neural Networks. _IEEE Transactions on Neural Networks and Learning Systems_, pages 1-15, 2022.
* [40] F. Khani and P. Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In _Proceedings of the ACM Conference on Fairness, Accountability, and Transparency_, pages 196-205. ACM, 2021.
* [41] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, and R. Sayres. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In _Proceedings of the 35th International Conference on Machine Learning_, volume 80, pages 2668-2677, 2018.
* [42] D. P. Kingma and M. Welling. Auto-Encoding Variational Bayes. _arXiv preprint_, 1312.6114, 2014.
* [43] P. Kirchenko, P. Izmailov, and A. G. Wilson. Last Layer Refraining is Sufficient for Robustness to Spurious Correlations. _arXiv preprint_, 2204.02937, 2022.
* [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In _Proceedings of the 25th International Conference on Advances in Neural Information Processing Systems_, pages 1097-1105, 2012.
* [45] S. Lapuschkin, S. Waldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Muller. Unmasking Clever Hans predictors and assessing what machines really learn. _Nature Communications_, 10(1):1096, 2019.
* [46] Y. LeCun and C. Cortes. MNIST handwritten digit database. [http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/), 1998.
* [47] Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts. _arXiv preprint_, 2210.11466, oct 2022.
* [48] Z. C. Lipton. The Mythos of Model Interpretability. _Communications of the ACM_, 61(10):35-43, 2016.
* [49] E. Z. Liu, B. Haghgoo, A. S. Chen, A. Raghunathan, P. W. Koh, S. Sagawa, P. Liang, and C. Finn. Just Train Twice: Improving Group Robustness without Training Group Information. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139, pages 6781-6792, 2021.
* [50] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In _Proceedings of International Conference on Computer Vision (ICCV)_, 2015.
* [51] P. Liznerski, L. Ruff, R. A. Vandermeulen, B. J. Franks, M. Kloft, and K.-R. Muller. Explainable Deep One-Class Classification. _arXiv preprint_, 2007.01760, 2020.
* [52] N. K. Mishra and M. E. Celebi. An Overview of Melanoma Detection in Dermoscopy Images Using Image Processing and Machine Learning. _arXiv preprint_, 1601.07843, 2016.
* [53] B. Mittelstadt, C. Russell, and S. Wachter. Explaining Explanations in AI. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_, pages 279-288. ACM, 2019.
* [54] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. _Nature_, 518(7540):529-533, feb 2015.
* [55] G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R. Muller. Layer-Wise Relevance Propagation: An Overview. In _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_, pages 193-209. Springer International Publishing, 2019.
* [56] W. J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Definitions, methods, and applications in interpretable machine learning. _Proceedings of the National Academy of Sciences of the United States of America_, 116(44):22071-22080, 2019.
* [57] J. Nam, J. Kim, J. Lee, and J. Shin. Spread Spurious Attribute: Improving Worst-group Accuracy with Spurious Attribute Estimation. _arXiv preprint_, 2204.02070, 2022.
* [58] A. Odena and I. Goodfellow. TensorFlowFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing. In _Proceedings of the 36th International Conference on Machine Learning_, pages 8603-8613. International Machine Learning Society (IMLS), 2019.
* [59] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In _Advances in Neural Information Processing Systems 32_, pages 8024-8035, 2019.
* [60] G. Plumb, M. T. Ribeiro, and A. Talwalkar. Finding and Fixing Spurious Patterns with Explanations. _arXiv preprint_, 2106.02112, jun 2021.
* [61] A. Polyak and L. Wolf. Channel-level acceleration of deep face representations. _IEEE Access_, 3:2163-2175, oct 2015.
* [62] M. T. Ribeiro, S. Singh, and C. Guestrin. "Why should I trust you?" Explaining the predictions of any classifier. In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_, pages 1135-1144, 2016.
* [63] L. Rieger, C. Singh, W. J. Murdoch, and B. Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In _Proceedings of the 37th International Conference on Machine Learning_, pages 8116-8126, 2019.
* [64] M. Roberts, D. Driggs, M. Thorpe, J. Gilbey, M. Yeung, et al. Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans. _Nature Machine Intelligence_, 3(3):199-217, 2021.
* [65] A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_, pages 2662-2670, 2017.
* [66] C. Rudin. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. _Nature Machine Intelligence_, 1(5):206-215, 2019.
* [67] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. _International Journal of Computer Vision (IJCV)_, 115(3):211-252, 2015.
* [68] S. Sagawa, P. W. Koh, T. B. Hashimoto, and P. Liang. Distributionally Robust Neural Networks. In _Proceedings of the International Conference on Learning Representations_, 2020.
* [69] S. Sagawa, A. Raghunathan, P. W. Koh, and P. Liang. An investigation of why overparameterization exacerbates spurious correlations. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119, pages 8346-8356, 2020.
* [70] W. Samek, G. Montavon, S. Lapuschkin, C. J. Anders, and K.-R. Muller. Explaining deep neural networks and beyond: A review of methods and applications. _Proceedings of the IEEE_, 109(3):247-278, 2021.
* [71] W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.-R. Muller, editors. _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_, volume 11700 of _Lecture Notes in Computer Science_. Springer International Publishing, Cham, 2019.
* [72] S. Santurkar, D. Tsipras, and A. Madry. BREEDS: benchmarks for subpopulation shift. _arXiv preprint_, 2008.04859, 2020.
* [73] T. Schnake, O. Eberle, J. Lederer, S. Nakajima, K. T. Schutt, K.-R. Muller, and G. Montavon. Higher-Order Explanations of Graph Neural Networks via Relevant Walks. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 44(11):7581-7596, 2022.
* [74] P. Schramowski, W. Stammer, S. Teso, A. Brugger, F. Herbert, X. Shao, H. G. Luigs, A. K. Mahlein, and K. Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. _Nature Machine Intelligence_, 2(8):476-486, 2020.
* [75] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. _Int. J. Comput. Vis._, 128(2):336-359, 2020.
* [76] Y. Shen and B. Zhou. Closed-form factorization of latent semantics in gans. In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1532-1540, 2021.
* [77] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sire, et al. Mastering the game of Go with deep neural networks and tree search. _Nature_, 529(7587):484-489, jan 2016.
* [78] K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In _Proceedings of the 3rd International Conference on Learning Representations_, 2015.
* [79] B. Simpson, F. Dutil, Y. Bengio, and J. P. Cohen. GradMask: Reduce Overfitting by Regularizing Saliency. _arXiv preprint_, 1904.07478, 2019.
* [80] N. S. Sohoni, M. Sanjabi, N. Ballas, A. Grover, S. Nie, H. Firooz, and C. Re. BARACK: Partially Supervised Group Robustness With Guarantees. _arXiv preprint_, 2201.00072, 2021.
* [81] E. Sorantin, M. G. Grasser, A. Hemmelmayr, S. Tschauner, F. Hrzic, V. Weiss, J. Laeckova, and A. Holzinger. The augmented radiologist: artificial intelligence in the practice of radiology. _Pediatric Radiology_, 52(11):2074-2086, 2022.
* [82] I. Stepin, J. M. Alonso, A. Catala, and M. Pereira-Farina. A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence, 2021.
* [83] S. Teso and K. Kresting. Explanatory interactive machine learning. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_, page 239-245, 2019.
* [84] Y. Tian, Z. Zhong, V. Ordonez, G. Kaiser, and B. Ray. Testing DNN image classifiers for confusion & bias errors. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_, pages 1122-1134, 2020.
* [85] P. Tschandl, C. Rosendahl, and H. Kittler. The HAM10000 Dataset: A Large Collection of Multi-Source Dermatoscopic Images of Common Pigmented Skin Lesions. _Scientific Data_, 5:18016, 2018.
* [86] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is All you Need. In _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc., 2017.
* [87] S. Verma, J. Dickerson, and K. Hines. Counterfactual Explanations for Machine Learning: Challenges Revisited. _arXiv preprint_, 2106.07756, jun 2021.
* [88] J. K. Winkler, C. Fink, F. Toberer, A. Enk, T. Deinlein, R. Hofmann-Wellenhof, L. Thomas, A. Lallas, A. Blum, W. Stolz, and H. A. Haenske. Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition. _JAMA Dermatology_, 155(10):1135, 2019.
* [89] W. Wu, H. Xu, S. Zhong, M. R. Lyu, and I. King. Deep Validation: Toward Detecting Real-World Corner Cases for Deep Neural Networks. In _Proceedings of the 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks_, pages 125-137, 2019.
* [90] L. Wynants, B. Van Calster, G. S. Collins, R. D. Riley, G. Heinze, et al. Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. _The BMJ_, 369(8242):m1328, 2020.
* ECCV 2020 Workshops_, pages 506-523, 2020.
* [92] Q. Zhang, X. Wang, R. Cao, Y. N. Wu, F. Shi, and S.-C. Zhu. Extraction of an explanatory graph to interpret a CNN. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 43(11):3863-3877, Nov. 2021.
* [93] Y. Zhang, P. Tino, A. Leonardis, and K. Tang. A Survey on Neural Network Interpretability. _IEEE Transactions on Emerging Topics in Computational Intelligence_, 5(5):726-742, 2021.
# Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks

(Supplementary Material)

Lorenz Linhardt

 Klaus-Robert Muller

 Gregoire Montavon

###### Abstract

In this section we want to show that, assuming a common neural network with neurons of the type

\[z_{j} =\sum_{i}a_{i}w_{ij}+b_{j}\] \[a_{j} =g(z_{j}), \tag{1}\]

attribution scores associated to the explanation techniques \(\operatorname{Gradient}\times\operatorname{Input}\), Integrated Gradients [8], and layer-wise relevance propagation (LRP) [1; 6] can be decomposed and written in the form:

\[R_{i} =\sum_{j}R_{ij}\] \[R_{ij} =a_{i}\rho(w_{ij})d_{j} \tag{2}\]

where \(\rho:\mathbb{R}\rightarrow\mathbb{R}\) is some function, and \(d_{j}\) is a term that only indirectly depends on the activation \(a_{i}\) and the weight \(w_{ij}\).

### Gradient \(\times\operatorname{Input}\)

Denoting by \(y\) the output of the neural network that we would like to explain, we can write the scores obtained by \(\operatorname{Gradient}\times\operatorname{Input}\) w.r.t. any layer with activations \(\{a_{i}\}_{i=1}^{N}\) as:

\[R_{i} =a_{i}\frac{\partial y}{\partial a_{i}}. \tag{3}\]

Using the chain rule, the equation can be further developed as:

\[=a_{i}\sum_{j}\frac{\partial y}{\partial z_{j}}\frac{\partial z_{j}}{ \partial a_{i}}=\sum_{j}\underbrace{a_{i}\underbrace{\frac{\partial z_{j}}{ \partial a_{i}}\underbrace{\frac{\partial y}{\partial z_{j}}}}_{R_{ij}}}_{R_{ ij}} \tag{4}\]

form which we can identify the desired structure of Eq. (2).

### _Integrated Gradients_

For the Integrated Gradients formulation, we consider as integration path a segment from the origin to the data point (i.e. the map \(t\to t\cdot\mathbf{a}\) with \(t\in[0,1]\)):

\[R_{i}=\int\frac{\partial y}{\partial a_{i}}\frac{\partial a_{i}}{\partial t}dt \tag{5}\]

Using the chain rule, the equation can be further developed as:

\[=\int\sum_{j}\frac{\partial y}{\partial z_{j}}\frac{\partial z_{j}}{\partial a _{i}}\frac{\partial a_{i}}{\partial t}dt \tag{6}\]
\[=\sum_{j}\int\frac{\partial y}{\partial z_{j}}\frac{\partial z_{j}}{ \partial a_{i}}\frac{\partial a_{i}}{\partial t}dt \tag{7}\] \[=\sum_{j}\underbrace{\Big{(}\int\frac{\partial y}{\partial z_{j}} dt\Big{)}}_{R_{ij}}w_{ij}a_{i} \tag{8}\]

From which we can again identify the structure of Eq. (2).

### Layer-wise Relevance Propagation

Starting from a generic LRP rule that admits standard LRP rules such as LRP-0, LRP-\(\gamma\) and LRP-\(\epsilon\) as special cases, specifically:

\[R_{i}=\sum_{j}\frac{a_{i}\rho(w_{ij})}{\sum_{i^{\prime}}a_{i^{ \prime}}\rho(w_{i^{\prime}j})+\epsilon}\cdot R_{j} \tag{9}\]

with \(\rho(t)=t+\gamma\max(0,t)\) with \(\gamma\) nonnegative, we get after some slight reorderings the equation

\[=\sum_{j}\underbrace{a_{i}\rho(w_{ij})\underbrace{\frac{R_{j}}{ \sum_{i^{\prime}}a_{i^{\prime}}\rho(w_{i^{\prime}j})+\epsilon}}_{d_{j}}}_{R_{ ij}} \tag{10}\]

which has the structure of Eq. (2).

## Appendix B Derivation of EGEM

In this section we derive a closed form solution for the EGEM method, which we stated in Section 3 of the main paper as

\[\forall_{ij}:\ w_{ij}=\frac{\mathbb{E}[a_{i}^{2}d_{j}^{2}]}{ \mathbb{E}[a_{i}^{2}d_{j}^{2}]+\lambda\mathbb{E}[d_{j}^{2}]}w_{ij}^{\text{old}} \tag{11}\]

with the purpose of solving the objective

\[\min_{w}\ \sum_{ij}\mathbb{E}\big{[}(R_{ij}(w_{ij})-R_{ij}(w_{ij}^{ \text{old}}))^{2}+\lambda(s_{ij}(w_{ij}))^{2}\big{]} \tag{12}\]

where

\[R_{ij}(w_{ij})=a_{i}\,\rho(w_{ij})\,d_{j} \tag{13}\]

and

\[s_{ij}(w_{ij})=\rho(w_{ij})\,d_{j}. \tag{14}\]

Substituting these last two terms into the objective, we get:

\[\min_{w}\ \sum_{ij}\mathbb{E}\big{[}\big{(}a_{i}\,\rho(w_{ij})\,d_{j}- a_{i}\,\rho(w_{ij}^{\text{old}})\,d_{j}\big{)}^{2}+\lambda\cdot\big{(}\rho(w_{ ij})\,d_{j}\big{)}^{2}\big{]} \tag{15}\]
We observe that each term of the sum depends on its own parameter \(w_{ij}\). Hence, each term can be minimized separately. Consider one such term and compute its gradient:

\[E_{ij}(w_{ij}) =\mathbb{E}\big{[}\big{(}a_{i}\,\rho(w_{ij})\,d_{j}-a_{i}\,\rho(w_{ ij}^{\text{old}})\,d_{j}\big{)}^{2}+\lambda\cdot(\rho(w_{ij})\,d_{j})^{2}\big{]} \tag{16}\] \[\nabla E_{ij}(w_{ij}) =\mathbb{E}\big{[}2\big{(}a_{i}\,\rho(w_{ij})\,d_{j}-a_{i}\,\rho( w_{ij}^{\text{old}})\,d_{j}\big{)}\cdot a_{i}\,\rho^{\prime}(\theta_{ij})\,d_{j}+2 \lambda\cdot\rho(\theta_{ij})\,\rho^{\prime}(\theta_{ij})\,d_{j}^{2}\big{]} \tag{17}\]

We now find where the gradient is zero. Our derivation uses the fact that \(\rho(w_{ij})\) and its derivative do not depend on the data and can therefore be taken out of the expectation:

\[\mathbb{E}\big{[}2\big{(}a_{i}\,\rho(w_{ij})\,d_{j}-a_{i}\,\rho( w_{ij}^{\text{old}})\,d_{j}\big{)}\cdot a_{i}\,\rho^{\prime}(\theta_{ij})\,d_{j}+2 \lambda\cdot\rho(\theta_{ij})\,\rho^{\prime}(\theta_{ij})\,d_{j}^{2}\big{]} \stackrel{{!}}{{=}}0 \tag{18}\] \[\mathbb{E}\big{[}\big{(}a_{i}\,\rho(w_{ij})\,d_{j}-a_{i}\,\rho( w_{ij}^{\text{old}})\,d_{j}\big{)}\cdot a_{i}\,d_{j}+\lambda\cdot\rho(\theta_{ij}) \,d_{j}^{2}\big{]} \stackrel{{!}}{{=}}0\] (19) \[\rho(w_{ij})\,\mathbb{E}(a_{i}^{2}d_{j}^{2})-\rho(w_{ij}^{\text{ old}})\,\mathbb{E}(a_{i}^{2}d_{j}^{2})+\lambda\rho(w_{ij})\,\mathbb{E}[d_{j}^{2}] \stackrel{{!}}{{=}}0\] (20) \[\rho(w_{ij})\,\mathbb{E}(a_{i}^{2}d_{j}^{2})+\lambda\cdot\rho(w_ {ij})\mathbb{E}[d_{j}^{2}] \stackrel{{!}}{{=}}\rho(w_{ij}^{\text{old}})\, \mathbb{E}[a_{i}^{2}d_{j}^{2}]\] (21) \[\rho(w_{ij}) \stackrel{{!}}{{=}}\frac{\mathbb{E}[a_{i}^{2}d_{j}^{2} ]}{\mathbb{E}[a_{i}^{2}d_{j}^{2}]+\lambda\cdot\mathbb{E}[d_{j}^{2}]}\cdot\rho (w_{ij}^{\text{old}}) \tag{22}\]

Furthermore, the equation above also implies

\[w_{ij}\stackrel{{!}}{{=}}\frac{\mathbb{E}[a_{i}^{2}d_{j}^{2}]}{ \mathbb{E}[a_{i}^{2}d_{j}^{2}]+\lambda\cdot\mathbb{E}[d_{j}^{2}]}\cdot w_{ij} ^{\text{old}} \tag{23}\]

for the choices of function \(\rho\) encountered in Supplementary Note A.

## Supplementary Note C Data

### Supplementary Note C.1. Modified MNIST

The modified MNIST is a variant of the original MNIST dataset [4] where a small 3-pixel artifact is pasted onto the top left corner of 70% of the images of the digit '8'. In order to generate a somewhat natural split of the data where one part is free of the artifact, we train a variational autoencoder [3] on the artifact-modified dataset to model the underlying distribution. Then, we manually select a single dimension and threshold along this dimension to separate the data into two partitions, such that both partitions contain all digits, but the modified samples only fall onto one side. This partitioning defines the pool of clean samples from which to draw the refinement data.

### Supplementary Note C.2. Correlation Structure in CelebA

In this section we present the correlation structure of the attributes in the training partition of the CelebA dataset [5], which is essential when reasoning about Clever Hans effects since they are based on spurious correlations in the training data. Fig. 1 displays the respective correlation matrix. Notably, 'Male' and 'Blond_Hair' are negatively correlated, while 'Male' and 'Wearing_Necktie', 'Goatee', and 'Sideburns' are positively correlated, which could lead 'Male'-related visual features to be used as inhibitory signal by models trained to detect blond hair on this dataset.

## Supplementary Note D. Response-Guided Exposure Minimization

Consider the problem of learning a low-complexity model that reproduces the output of some original model on the validated data. Let \(f(\mathbf{x},\mathbf{w})\) and \(f(\mathbf{x},\mathbf{w}_{\text{old}})\) be the output of the student and teacher models respectively, parameterized by weights \(\mathbf{w}\) and \(\mathbf{w}_{\text{old}}\). We can formulate the objective as the optimization problem:

\[\min_{\mathbf{w}}\quad\mathbb{E}[(f(\mathbf{x},\mathbf{w})-f(\mathbf{x},\mathbf{w}_{\text{old}}))^ {2}]+\lambda\|\mathbf{w}\|^{2} \tag{24}\]
model output on the refinement data. The second term with regularization parameter \(\lambda\) penalizes overall model exposure, e.g. forcing the model to not be too complex. For the linear case, where the teacher model and student model are given by \(f(\mathbf{x},\mathbf{w}_{\text{old}})=\mathbf{w}_{\text{old}}{}^{\top}\mathbf{x}\) and \(f(\mathbf{x},\mathbf{w})=\mathbf{w}^{\top}\mathbf{x}\) respectively, we get the closed form solution:

\[\mathbf{w}=(\Sigma+\lambda I)^{-1}\Sigma\mathbf{w}_{\text{old}} \tag{25}\]

where \(\Sigma=\mathbb{E}[\mathbf{x}\mathbf{x}^{\top}]\). This equation resembles the ridge regression solution; the difference being that the cross-covariance between data and targets \(\mathbb{E}[\mathbf{x}\mathbf{y}]\) in the original model is replaced by the term \(\Sigma\mathbf{w}_{\text{old}}\). This term performs a realignment of the pretrained model's weights along the validation data. This realignment with the refinement data, desensitizes the model to directions in feature space that are not expressed in the available data and that the user could not verify, giving some level of immunity against a possible CH effect in the classifier. For neural networks with a final linear projection layer, response-guided exposure minimization (RGEM) can be applied to this last layer.

We can rewrite Eq. (25) as ridge regression on the predictions \(f(\mathbf{x},\mathbf{w}_{\text{old}})\):

\[\mathbf{w}=(\mathbf{X}^{T}\mathbf{X}+\lambda I)^{-1}\mathbf{X}^{T}f(\mathbf{X},\mathbf{w}_{\text{old}}) \tag{26}\]

where \(f(\mathbf{X},\mathbf{w}_{\text{old}})\) is the vector of outputs of the original model on the refinement data.

Figure 1: Correlation matrix of attributes in the CelebA training data.
## Supplementary Note E Models: Training Procedure

In this section, we provide the details of the training procedure relevant for reproducing the models used in our experiments in Section 4 of the main paper.

For the experiments on ImageNet and the ISIC dataset, we use the pretrained VGG-16 network, as provided in the pytorch library [7]. For the ISIC dataset, only keep the first two output nodes and fine-tune the network for 10 epochs with learning rate 0.0001 and batch size 64. For the experiments on MNIST, we train a neural network specified in Table 1 for 5 epochs with learning rate 0.001 and batch size 128. For the experiments on the CelebA dataset, we train a neural network specified in Table 2 for 10 epochs with a learning rate of 0.000025 and a batch size of 100. For all training procedures, the Adam optimizer [2] is used.

## Supplementary Note F User Verification and Experimental Details

In the following we provide details of the simulated user verification procedure and of the generation of the poisoned data sets, where the relation between the class and the spurious features is modified.

### Supplementary Note F.1 Simulation of User-Verification

Generally, we would consider samples to be user-verified if they are correctly predicted by the model and if a human has inspected and agrees with the corresponding explanation.

For the experiments in Section 4 of the main paper we assumed that all correctly classified clean samples are user-verified and thus just removed samples containing the artifact from the refinement data. For the experiments related to the class _mountain bike_, we only retained samples not containing a frame. This is easily automated by checking multiple pixels along the border for agreement with the gray value of the artifact. As it is challenging to automatically filter out images containing the _carton_-related watermarks, we did so manually for the corresponding experiments and permitted any correctly classified image without the spurious feature to be used for refinement. For the ISIC dataset, we manually removed samples containing the colored patches from the data after training. For the experiments on CelebA in Section 5 of the main paper, as the task was more open and no specific Clever Hans feature was targeted, we simulated user-verification by checking for agreement of an LRP-based explanation [1; 6] of the model's decision with an expected location of hair features. To this end we defined a mask, seen in Fig. 2, which roughly corresponds to where we expect features that a user would use to judge a celebrity's hair color to lie. We considered samples to be user-verified if at least 75% of the absolute LRP relevance for the correct class lies within this mask.

\begin{table}
\begin{tabular}{|c|l|c|c|c|} \hline ID & Type & Channels & Kernel & Stride \\ \hline
1 & Conv. (ReLU) & 8 & 3 & 1 \\
2 & Max-pool & - & 2 & - \\
3 & Conv. (ReLU) & 16 & 5 & 1 \\
4 & Max-pool & - & 2 & - \\
5 & FC (ReLU) & 200 & - & - \\
6 & FC (Identity) & 10 & - & - \\ \hline \end{tabular}
\end{table}
Table 1: Neural network architecture used for the experiments using the MNIST dataset.

\begin{table}
\begin{tabular}{|c|l|c|c|c|} \hline ID & Type & Channels & Kernel & Stride \\ \hline
1-11 & 3 VGG-16 blocks & - & - & - \\
12 & Conv. (ReLU) & 128 & 3 & 1 \\
13 & Adaptive max-pool & - & - & - \\
14 & FC (ReLU) & 512 & - & - \\
15 & FC (Identity) & 2 & - & - \\ \hline \end{tabular}
\end{table}
Table 2: Neural network architecture used for the experiments using the ISIC dataset.
### Task Selection and Preprocessing

We will first describe the choice of classes the ImageNet experiments are based on, and then provide further information about the way the individual test samples for the different poisoning scenarios have been manipulated.

The classes to be discriminated were selected based on their similarity to the target classes for which artifacts were identified. Similarity was measured by the mean absolute difference of the soft-max output (see Table 3). The selection was done using all training samples of the respective target class. We adopted this selection scheme because natural spurious signals cannot be expected to have a strong enough effect to facilitate significant confusion between classes that are only weakly related.

In the following we will describe the process of manipulating individual data points to shift the distribution of spurious artifacts and create the poisoned dataset for evaluation. The poisoning level denotes with what probability any data point in the clean evaluation dataset is manipulated.

For the class _mountain bike_, we resize the original image to fit into the frame-shaped artifact. The modified image consists of the frame and the shrunken image. We prefer the resizing approach over simply pasting the frame onto the image, as it avoids to remove potential class evidence near the border of the image. Furthermore, we removed the one mountain-bike image from the test set that exhibits the frame artifact.

For the _carton_ class, we paste the watermark and URL occurring on images of this class to the center and bottom right, respectively, of the image to be modified. In particular, we manually recreated the URL artifact to look like the ones found in the dataset, but without any additional background. Similarly, the watermark artifact we used was cleaned of its background and pasted with transparency.

Due to the small number of evaluation samples in the ImageNet dataset (50 per class, i.e. the official validation set), we apply data augmentation in order to get robuster estimates. We do this by adding a horizontally flipped version of each samples to the test set.

### Refinement Hyper-parameters

In this section we provide the hyper-parameter space we searched over for each refinement method, as well as some additional technical details.

Figure 2: Example of an image from the CelebA dataset. The highlighted area is the mask within which relevance must be concentrated to be accepted as correctly explained.

\begin{table}
\begin{tabular}{|c|l|l|c|} \hline ID & Class Name & Target & Distance \\ \hline
519 & crate & carton & 0.685 \\
549 & envelope & carton & 0.691 \\
692 & packet & carton & 0.695 \\
444 & bicycle-built-for-two & mountain bike & 0.834 \\ \hline \end{tabular}
\end{table}
Table 3: Mean absolute distance to the target class, calculated from the soft-max outputs of a pretrained ResNet50 network.
Retraining.For the Retrain baseline, all layer of the pretrained models are fine-tuned on the available refinement data using the Adam optimizer [2] without further regularization. We use 20% of the refinement data (rounded up) as validation set and optimize over the number of epochs \(N_{e}\in\{1,5,10,20,30,50,100\}\) for all datasets. The learning rates are set as follows. MNIST: \(1\cdot 10^{-3}\), ImageNet(ResNet50): \(5\cdot 10^{-6}\), ImageNet(VGG-16): \(5\cdot 10^{-5}\), ISIC: \(1\cdot 10^{-7}\). We fine-tuned the networks with frozen batch-norm parameters, and gradients clipped to \(10^{-3}\).

Rgem and Ridge.We optimize over the regularization parameter \(\lambda\in\{0.0001,\,0.001,\,0.01,\,1,\,10,\,100,\,1000,\,1000\}\). The difference between the baselines is that Ridge regresses the true labels \(y\) of the refinement samples whereas REGEM regresses the outputs \(\hat{y}\) of the original model.

Egem and PCA-Egem.We optimize over the refinement parameter \(\alpha\in\{0.00001,\,0.0001,\,0.001,\,0.01,\,0.1,\,0.2,\,0.3,\,0.4,\,0.5, \,0.6,\,0.7,\,0.8,\,0.9,\,1\}\). The activations to be refined are the ones immediately after every ResNet50 or VGG-16 block for the parts of the models that are derived from those architectures and additionally after every ReLU activation following a fully connected or convolutional layer outside of such blocks.

## Supplementary Note G. Varying Slack

In this section we report the 0%-poisoning and 100%-poisoning results for varying the value of the slack variable in the hyper-parameter selection for all models in Fig. 3. Low slack means that we require the clean-data accuracy of the refined model to be close to the unrefined model, where larger slack allows for increasingly stronger deviations from this base accuracy. Larger slack implies stronger refinement. Additionally, we show the progression of the average performance with increasing slack in Fig. 4. The 5% slack level chosen in the main body of the paper can be taken as a rule of thumb, which appears to lead to effective refinement without sacrificing too much clean-data accuracy in most cases. Yet, it can be seen that the optimal level depends on the method and the dataset. For example, at 5% slack PCA-EGEM loses about 4% accuracy on clean and poisoned MNIST data, whereas it would be able to perfectly refine the model without loss of accuracy at the 0% level.

It can also be observed in Fig. 3 that the optimal slack value on the various ImageNet tasks is not uniform, which is surprising, since all tasks containing the _'_carton' class are poisoned with the same artifact. We assume that this effect arises from the other class losing critical features at different refinement levels. Fig. 3 demonstrates that PCA-EGEM leads to the most effective refinement over a large number of slack values - the advantage over other methods being even more apparent in settings with lower slack values than the 5% presented in the main body of the paper.

## Supplementary Note H. Varying Samples

In this section we report the 0%-poisoning and 100%-poisoning results for varying the value of the number of available samples for refinement and hyper-parameter selection for all models in Fig. 5.

## Supplementary Note I. Additional Experiments on CelebA

In this section we report additional results for the analysis of PCA-EGEM on the CelebA dataset [5].

### Supplementary Note I.1. Precision and Recall

We here present the precision and recall - with and without application of PCA-EGEM - for all subgroups in the CelebA test data that are induced by the attributes provided in the dataset. In Fig. 6 it can be seen that for a majority of subgroups effect are negligible, but for a minority of the groups the application of PCA-EGEM results in large changes, in particular on recall. The effect on precision is comparably small and in most cases negative, as the strongest CH features appear to be inhibitive. Interestingly, for the
subgroups induced by the attributes 'Wearing_Hat', and 'Blurry', where we see a reduction in recall, we also see increased precision, which may be explained by the classifier relying on spurious features when the hair is not clearly visible.

_Supplementary Note I.2. Validation of the Collar Clever Hans Feature on CelebA_

To corroborate our observation that shirt collars are a CH feature, we created a counterfactual dataset where the bottom section of the images is occluded by an image of a wall (see Fig. 7) and repeated our analysis with occlusion instead of refinement. Indeed, we can observe that occlusion of the bottom section of the images improves recall of the 'Wearing_Necktie', 'Sideburns', 'Goatee', and 'Male' groups. Furthermore, the recall of the 'Wearing_Hat' group is reduced - this is most likely due to the fact that for some samples the blond part of the hair would only be visible in the now-occluded part of the image, as it is covered by a hat in the upper parts of the image. Overall, these results supports our observation of collars being inhibitive CH features.
